2024-03-24T09:01:34.843 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T09:01:34.843 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041 --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/orig.config.yaml 2024-03-24T09:01:34.894 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T09:01:34.977 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041 branch: wip-batrick-testing-20240323.003144-squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: pdonnell@redhat.com first_in_suite: false job_id: '7619041' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 21b3bd672163092b4e542f772797fcba02d3d4de subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYTgwwOu+qTUs1gsSOAj/87LIqr9KvLYwkWMvOtbWNrIEn9HddFLJ3lhalWnfLbi2fNxScUOEgQ9dHvquEaMDg= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIdp7nXb8gXZr4iEpfroPoTTDcs4HToxi81HhJcAuMTP4kHdGtHX7tb18XCJAiWD04HaKCbfS44q0CZi6tlEvbw= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+NgiQJlqPmpDDDfH0nKgQEVRF7PjFKCsnhiFi0ocLjsAmtiy4I9SWepamQiP71ZN73t1ataxNAOtjyjOEH42c= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T09:01:34.978 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T09:01:34.978 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T09:01:34.978 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T09:01:34.980 INFO:teuthology.task.internal:Checking packages... 2024-03-24T09:01:35.000 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T09:01:35.000 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T09:01:35.000 INFO:teuthology.packaging:ref: None 2024-03-24T09:01:35.000 INFO:teuthology.packaging:tag: None 2024-03-24T09:01:35.000 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T09:01:35.000 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:01:35.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T09:01:35.219 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T09:01:35.220 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T09:01:35.227 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T09:01:35.227 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T09:01:35.337 INFO:teuthology.task.internal:Saving configuration 2024-03-24T09:01:35.356 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T09:01:35.360 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T09:01:35.383 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 08:56:43.707467', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYTgwwOu+qTUs1gsSOAj/87LIqr9KvLYwkWMvOtbWNrIEn9HddFLJ3lhalWnfLbi2fNxScUOEgQ9dHvquEaMDg='} 2024-03-24T09:01:35.405 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 08:56:43.703876', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIdp7nXb8gXZr4iEpfroPoTTDcs4HToxi81HhJcAuMTP4kHdGtHX7tb18XCJAiWD04HaKCbfS44q0CZi6tlEvbw='} 2024-03-24T09:01:35.422 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 08:56:43.705931', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+NgiQJlqPmpDDDfH0nKgQEVRF7PjFKCsnhiFi0ocLjsAmtiy4I9SWepamQiP71ZN73t1ataxNAOtjyjOEH42c='} 2024-03-24T09:01:35.423 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T09:01:35.429 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-24T09:01:35.429 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-24T09:01:35.429 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-24T09:01:35.429 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T09:01:35.549 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f05336a2160>, signals=[15]) 2024-03-24T09:01:35.549 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T09:01:35.555 INFO:teuthology.task.internal:Opening connections... 2024-03-24T09:01:35.555 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:01:35.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:01:35.649 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-24T09:01:35.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:01:35.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T09:01:35.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:01:35.821 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T09:01:35.827 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-24T09:01:35.845 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-24T09:01:35.845 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-24T09:01:35.901 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T09:01:35.902 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-24T09:01:35.902 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T09:01:35.902 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T09:01:35.902 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T09:01:35.902 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-24T09:01:35.923 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-24T09:01:35.941 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-24T09:01:35.941 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-24T09:01:35.997 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-24T09:01:35.997 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T09:01:35.998 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T09:01:35.998 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-24T09:01:36.023 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-03-24T09:01:36.041 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-03-24T09:01:36.042 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T09:01:36.099 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-03-24T09:01:36.100 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-03-24T09:01:36.100 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T09:01:36.100 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-03-24T09:01:36.100 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T09:01:36.100 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T09:01:36.100 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T09:01:36.100 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-03-24T09:01:36.123 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T09:01:36.133 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T09:01:36.153 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T09:01:36.154 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-24T09:01:36.156 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-24T09:01:36.158 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-03-24T09:01:36.174 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T09:01:36.217 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T09:01:36.217 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-24T09:01:36.220 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-24T09:01:36.222 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-03-24T09:01:36.311 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T09:01:36.375 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T09:01:36.389 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T09:01:36.389 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T09:01:36.389 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T09:01:36.389 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T09:01:36.390 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T09:01:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:01:36.406 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-24T09:01:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:01:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:01:36.408 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-24T09:01:36.408 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-24T09:01:36.465 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-361.el9.x86_64 2024-03-24T09:01:36.466 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.14.0-361.el9.x86_64 2024-03-24T09:01:36.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2024-03-24T09:01:36.467 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-361.el9.x86_64 2024-03-24T09:01:36.467 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-361.el9.x86_64 2024-03-24T09:01:36.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-03-24T09:01:36.468 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-24T09:01:36.469 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-24T09:01:36.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-24T09:01:38.540 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T09:01:38.564 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T09:01:38.653 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T09:01:39.981 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-24T09:01:40.049 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-24T09:01:40.228 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-24T09:01:41.196 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-24T09:01:42.446 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.0 MB 00:02 2024-03-24T09:01:44.020 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-24T09:01:45.672 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 1.5 MB/s | 8.0 MB 00:05 2024-03-24T09:01:47.623 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-24T09:01:48.154 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-24T09:01:49.747 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-24T09:01:50.548 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 30 MB/s | 21 MB 00:00 2024-03-24T09:01:51.088 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-24T09:01:55.030 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-24T09:01:55.523 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-24T09:01:55.622 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.1 MB 00:04 2024-03-24T09:01:56.506 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-24T09:01:57.423 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-24T09:01:57.621 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T09:01:57.622 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T09:01:57.715 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-24T09:01:57.725 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-24T09:01:57.726 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-24T09:01:57.726 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-24T09:01:57.726 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-24T09:01:57.726 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-24T09:01:57.726 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T09:01:57.726 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-24T09:01:57.727 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:01:57.728 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 76 M 2024-03-24T09:01:57.728 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-24T09:01:57.728 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-24T09:01:58.266 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 168 kB/s | 44 kB 00:00 2024-03-24T09:01:58.390 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-24T09:01:58.658 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-24T09:01:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-24T09:01:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 17 MB/s | 27 MB 00:01 2024-03-24T09:01:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:01:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-24T09:02:00.582 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-24T09:02:00.946 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-24T09:02:00.946 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-24T09:02:01.929 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-24T09:02:01.929 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-24T09:02:02.617 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-24T09:02:03.265 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-03-24T09:02:04.387 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-24T09:02:04.715 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T09:02:04.715 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T09:02:04.815 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T09:02:04.826 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T09:02:04.827 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T09:02:04.827 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:04.827 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-24T09:02:04.827 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-03-24T09:02:04.827 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-03-24T09:02:04.827 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:04.828 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 76 M 2024-03-24T09:02:04.828 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 123 M 2024-03-24T09:02:04.828 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-24T09:02:04.985 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:02:05.066 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:05.101 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 192 kB/s | 44 kB 00:00 2024-03-24T09:02:05.382 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T09:02:05.383 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T09:02:05.476 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T09:02:05.488 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 76 M 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 123 M 2024-03-24T09:02:05.489 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-24T09:02:05.560 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-24T09:02:05.891 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 332 kB/s | 44 kB 00:00 2024-03-24T09:02:06.094 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-24T09:02:06.935 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 15 MB/s | 27 MB 00:01 2024-03-24T09:02:06.938 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:02:06.938 INFO:teuthology.orchestra.run.smithi143.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-24T09:02:07.358 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 9.3 MB/s | 15 MB 00:01 2024-03-24T09:02:07.604 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-24T09:02:07.817 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:07.953 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:07.996 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-24T09:02:07.997 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-24T09:02:08.750 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 9.6 MB/s | 27 MB 00:02 2024-03-24T09:02:08.982 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-24T09:02:08.983 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-24T09:02:09.950 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 8.0 MB/s | 33 MB 00:04 2024-03-24T09:02:09.950 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:02:09.950 INFO:teuthology.orchestra.run.smithi117.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-24T09:02:10.611 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-24T09:02:10.969 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-24T09:02:10.969 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-24T09:02:11.480 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-24T09:02:11.969 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-24T09:02:11.970 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-24T09:02:12.068 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:02:12.154 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:12.652 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:12.686 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:14.417 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-24T09:02:14.870 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:15.011 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:15.026 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:02:15.108 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:17.477 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:17.848 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:18.030 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:19.684 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:19.728 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:22.709 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:22.810 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:24.460 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:27.691 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:46.833 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:46.833 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T09:02:46.833 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:02:46.877 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:47.227 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:47.227 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:02:47.227 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:47.227 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:47.652 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:02:47.653 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-24T09:02:47.957 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:02:48.074 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:48.075 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:02:48.075 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T09:02:48.075 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T09:02:48.075 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T09:02:48.075 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T09:02:48.075 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T09:02:48.075 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:02:48.154 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:48.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:02:48.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T09:02:48.189 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:02:48.190 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T09:02:48.191 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T09:02:48.191 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-430.el9.x86_64 2024-03-24T09:02:48.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T09:02:48.826 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-24T09:02:50.233 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T09:02:50.277 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-24T09:02:50.280 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-24T09:02:50.295 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.zT0hRbcG3k 2024-03-24T09:02:50.296 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zT0hRbcG3k 2024-03-24T09:02:50.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.zT0hRbcG3k 2024-03-24T09:02:50.505 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.zT0hRbcG3k is 6KB 2024-03-24T09:02:50.555 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.zT0hRbcG3k 2024-03-24T09:02:50.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T09:02:50.643 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T09:02:50.644 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T09:02:50.644 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T09:02:50.644 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T09:02:50.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T09:02:50.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-24T09:02:54.023 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:54.023 INFO:teuthology.orchestra.run.smithi143.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T09:02:54.023 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:54.068 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:54.456 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:54.456 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:02:54.456 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:54.457 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:54.874 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-24T09:02:55.199 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:02:55.338 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:55.338 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:02:55.338 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T09:02:55.338 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T09:02:55.338 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T09:02:55.338 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-24T09:02:55.338 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T09:02:55.339 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:02:55.418 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:55.419 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:02:55.419 DEBUG:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.06 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 68.el9 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Group : Unspecified 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 8264254 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T09:02:55.453 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : builder@centos.org 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2024-03-24T09:02:55.454 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:02:55.455 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T09:02:55.455 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T09:02:55.455 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 5.14.0-430.el9.x86_64 2024-03-24T09:02:55.456 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T09:02:56.072 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-03-24T09:02:56.904 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:56.904 INFO:teuthology.orchestra.run.smithi117.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T09:02:56.904 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:56.949 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:57.328 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:57.328 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T09:02:57.329 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T09:02:57.329 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T09:02:57.490 INFO:teuthology.orchestra.run.smithi143.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T09:02:57.523 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-03-24T09:02:57.525 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-03-24T09:02:57.540 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.Z2HNVCfJBw 2024-03-24T09:02:57.540 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z2HNVCfJBw 2024-03-24T09:02:57.609 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.Z2HNVCfJBw 2024-03-24T09:02:57.747 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.Z2HNVCfJBw is 6KB 2024-03-24T09:02:57.748 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:57.749 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-24T09:02:57.766 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.Z2HNVCfJBw 2024-03-24T09:02:57.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T09:02:57.853 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T09:02:57.853 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T09:02:57.853 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T09:02:57.853 INFO:teuthology.orchestra.run.smithi143.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T09:02:57.854 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T09:02:58.011 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-03-24T09:02:58.070 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:02:58.206 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:58.207 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:02:58.207 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T09:02:58.207 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T09:02:58.207 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T09:02:58.207 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T09:02:58.207 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T09:02:58.207 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T09:02:58.285 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T09:02:58.285 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T09:02:58.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T09:02:58.319 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:02:58.320 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T09:02:58.321 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T09:02:58.321 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-430.el9.x86_64 2024-03-24T09:02:58.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T09:02:58.913 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-03-24T09:03:00.345 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T09:03:00.378 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-03-24T09:03:00.380 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-24T09:03:00.399 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.F76eyE14aM 2024-03-24T09:03:00.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.F76eyE14aM 2024-03-24T09:03:00.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.F76eyE14aM 2024-03-24T09:03:00.602 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.F76eyE14aM is 6KB 2024-03-24T09:03:00.620 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.F76eyE14aM 2024-03-24T09:03:00.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T09:03:00.702 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T09:03:00.702 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T09:03:00.702 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T09:03:00.702 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T09:03:00.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T09:03:00.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-03-24T09:03:20.782 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T09:03:20.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:03:20.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:03:28.020 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-24T09:03:28.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T09:03:28.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:03:30.867 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T09:03:30.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T09:03:30.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:03:39.295 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-24T09:03:48.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:03:48.298 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:03:51.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-24T09:04:03.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:04:03.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:04:21.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-24T09:04:28.031 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T09:04:30.873 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T09:04:36.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:04:36.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:04:37.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T09:04:37.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:04:37.231 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-24T09:04:37.610 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:04:37.610 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T09:04:37.610 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-24T09:04:37.624 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-430.el9.x86_64 2024-03-24T09:04:37.624 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T09:04:37.624 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T09:04:37.624 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T09:04:38.478 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-24T09:04:38.625 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T09:04:38.625 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T09:04:38.701 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-03-24T09:04:38.745 DEBUG:teuthology.parallel:result is None 2024-03-24T09:04:39.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T09:04:39.021 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-430.el9.x86_64"... 2024-03-24T09:04:39.021 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-03-24T09:04:39.036 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-430.el9.x86_64 2024-03-24T09:04:39.036 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T09:04:39.036 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T09:04:39.036 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T09:04:39.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T09:04:39.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:04:40.036 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-24T09:04:40.037 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T09:04:40.114 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-03-24T09:04:40.165 DEBUG:teuthology.parallel:result is None 2024-03-24T09:04:41.358 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-24T09:04:41.862 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T09:04:41.862 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T09:04:41.862 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-24T09:04:41.877 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-430.el9.x86_64 2024-03-24T09:04:41.877 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T09:04:41.877 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T09:04:41.877 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T09:04:42.878 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T09:04:42.878 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T09:04:42.961 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-03-24T09:04:42.995 DEBUG:teuthology.parallel:result is None 2024-03-24T09:04:42.995 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T09:04:43.003 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T09:04:43.003 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T09:04:43.005 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T09:04:43.008 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T09:04:43.023 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T09:04:43.029 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T09:04:43.063 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T09:04:43.063 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T09:04:43.066 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T09:04:43.068 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T09:04:43.114 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T09:04:43.120 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T09:04:43.120 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T09:04:43.142 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T09:04:43.149 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T09:04:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:04:43.188 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:04:43.188 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:04:43.192 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:04:43.200 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:04:43.205 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T09:04:43.207 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T09:04:43.213 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T09:04:43.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T09:04:43.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T09:04:43.244 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T09:04:43.271 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T09:04:43.279 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T09:04:43.280 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T09:04:43.304 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T09:04:43.311 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T09:04:43.325 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:04:43.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:04:43.461 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:04:43.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:04:43.590 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:04:43.590 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T09:04:43.660 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:04:43.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:04:43.764 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:04:43.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:04:43.846 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:04:43.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T09:04:43.913 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:04:43.958 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T09:04:44.010 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:04:44.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T09:04:44.126 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:04:44.127 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T09:04:44.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-24T09:04:44.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-24T09:04:44.194 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-03-24T09:04:44.236 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:04:44.246 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:04:44.289 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:04:44.656 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T09:04:44.665 INFO:teuthology.task.internal:Starting timer... 2024-03-24T09:04:44.665 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T09:04:44.717 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T09:04:44.748 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-24T09:04:44.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-24T09:04:44.819 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-24T09:04:44.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-24T09:04:44.882 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-24T09:04:44.884 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-03-24T09:04:44.974 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-03-24T09:04:44.982 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T09:04:44.982 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-24T09:04:45.004 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-24T09:04:45.005 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-24T09:04:45.035 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-24T09:04:45.035 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-03-24T09:04:45.068 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-03-24T09:04:45.068 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi143.front.sepia.ceph.com': 'permissive'} 2024-03-24T09:04:45.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:04:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:04:45.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:04:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:04:45.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:04:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:04:45.171 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T09:04:45.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-24T09:04:45.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-24T09:04:45.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-03-24T09:04:45.395 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T09:04:45.405 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T09:04:45.422 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T09:04:45.423 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi117.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T09:08:58.614 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2024-03-24T09:08:58.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:08:58.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:08:58.704 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-24T09:08:58.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-24T09:08:58.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T09:08:58.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:08:58.877 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-24T09:08:58.952 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-24T09:08:58.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T09:08:58.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T09:08:59.045 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-03-24T09:08:59.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-03-24T09:08:59.117 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T09:08:59.129 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T09:08:59.129 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T09:08:59.130 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:08:59.132 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T09:08:59.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:08:59.134 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T09:08:59.134 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:08:59.161 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T09:08:59.162 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T09:08:59.178 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T09:08:59.178 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T09:08:59.190 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T09:08:59.207 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T09:08:59.207 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-24T09:08:59.208 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-24T09:08:59.220 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-24T09:08:59.221 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-24T09:08:59.236 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-03-24T09:08:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T09:08:59.237 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T09:08:59.250 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-03-24T09:08:59.253 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T09:08:59.254 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T09:08:59.266 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T09:08:59.282 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T09:08:59.304 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:08:59.307 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:08:59.307 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-24T09:08:59.307 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.307 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.307 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.307 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.308 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:08:59.311 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:08:59.312 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-24T09:08:59.312 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.312 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.312 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.312 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.343 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:08:59.346 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:08:59.346 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-24T09:08:59.346 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.346 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.347 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.347 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:08:59.347 INFO:teuthology.run_tasks:Running task install... 2024-03-24T09:08:59.355 DEBUG:teuthology.task.install:project ceph 2024-03-24T09:08:59.355 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T09:08:59.355 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T09:08:59.355 INFO:teuthology.task.install:Using flavor: default 2024-03-24T09:08:59.360 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T09:08:59.361 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T09:08:59.361 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T09:08:59.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:08:59.363 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T09:08:59.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:08:59.365 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T09:08:59.365 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:08:59.519 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T09:08:59.519 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T09:08:59.530 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T09:08:59.530 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T09:08:59.586 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T09:08:59.586 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T09:08:59.634 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T09:08:59.634 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:08:59.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T09:08:59.639 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T09:08:59.639 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:08:59.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T09:08:59.661 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T09:08:59.662 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T09:08:59.668 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T09:08:59.668 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T09:08:59.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T09:08:59.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T09:08:59.760 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T09:08:59.760 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:08:59.760 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T09:08:59.789 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T09:08:59.790 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T09:08:59.809 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T09:08:59.816 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T09:08:59.858 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T09:08:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-24T09:08:59.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-24T09:08:59.884 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-24T09:08:59.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-24T09:08:59.939 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T09:09:00.006 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-03-24T09:09:00.007 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-03-24T09:09:00.127 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-24T09:09:00.129 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-24T09:09:00.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T09:09:00.157 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T09:09:00.257 INFO:teuthology.orchestra.run.smithi143.stdout:74 files removed 2024-03-24T09:09:00.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T09:09:00.867 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-03-24T09:09:00.913 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-03-24T09:09:01.060 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 175 kB/s | 83 kB 00:00 2024-03-24T09:09:01.280 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-24T09:09:01.309 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-24T09:09:01.486 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-24T09:09:01.631 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-24T09:09:01.677 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-24T09:09:01.903 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-24T09:09:02.058 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T09:09:02.087 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T09:09:02.313 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T09:09:03.451 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-24T09:09:03.487 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-24T09:09:03.814 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-24T09:09:04.258 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-24T09:09:04.528 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-24T09:09:04.865 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-24T09:09:05.959 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-24T09:09:06.896 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-24T09:09:08.730 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-03-24T09:09:09.844 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-24T09:09:11.122 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-24T09:09:11.453 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-24T09:09:12.589 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-24T09:09:12.604 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-24T09:09:12.730 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 16 kB 00:00 2024-03-24T09:09:13.882 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-24T09:09:14.506 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-24T09:09:15.810 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-24T09:09:17.554 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-24T09:09:18.740 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-24T09:09:19.942 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T09:09:19.943 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T09:09:19.957 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T09:09:19.958 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T09:09:19.959 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T09:09:19.960 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T09:09:19.962 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T09:09:19.965 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T09:09:19.966 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T09:09:20.045 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T09:09:20.064 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T09:09:20.065 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T09:09:20.066 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T09:09:20.067 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T09:09:20.068 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T09:09:20.069 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T09:09:20.070 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T09:09:20.071 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T09:09:20.072 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T09:09:20.073 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T09:09:20.074 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T09:09:20.075 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T09:09:20.076 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:Install 150 Packages 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 24 Packages 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 224 M 2024-03-24T09:09:20.077 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-24T09:09:20.808 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-24T09:09:20.855 INFO:teuthology.orchestra.run.smithi116.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-24T09:09:21.064 INFO:teuthology.orchestra.run.smithi116.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 4.2 MB/s | 888 kB 00:00 2024-03-24T09:09:21.120 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T09:09:21.121 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T09:09:21.131 INFO:teuthology.orchestra.run.smithi116.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 2.1 MB/s | 145 kB 00:00 2024-03-24T09:09:21.135 INFO:teuthology.orchestra.run.smithi117.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T09:09:21.136 INFO:teuthology.orchestra.run.smithi117.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T09:09:21.137 INFO:teuthology.orchestra.run.smithi117.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T09:09:21.138 INFO:teuthology.orchestra.run.smithi117.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T09:09:21.141 INFO:teuthology.orchestra.run.smithi117.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T09:09:21.143 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T09:09:21.144 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T09:09:21.223 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-24T09:09:21.235 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-24T09:09:21.235 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-24T09:09:21.235 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-24T09:09:21.235 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T09:09:21.236 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T09:09:21.237 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T09:09:21.238 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T09:09:21.239 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T09:09:21.240 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T09:09:21.241 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T09:09:21.242 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T09:09:21.243 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T09:09:21.244 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T09:09:21.245 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T09:09:21.246 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T09:09:21.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T09:09:21.248 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T09:09:21.249 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T09:09:21.250 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T09:09:21.251 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T09:09:21.252 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout:Install 150 Packages 2024-03-24T09:09:21.253 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 24 Packages 2024-03-24T09:09:21.254 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:09:21.254 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 224 M 2024-03-24T09:09:21.254 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-24T09:09:21.456 INFO:teuthology.orchestra.run.smithi116.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 7.2 MB/s | 2.4 MB 00:00 2024-03-24T09:09:21.690 INFO:teuthology.orchestra.run.smithi116.stdout:(5/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-03-24T09:09:21.773 INFO:teuthology.orchestra.run.smithi116.stdout:(6/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 4.9 MB/s | 5.3 MB 00:01 2024-03-24T09:09:21.941 INFO:teuthology.orchestra.run.smithi117.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-24T09:09:22.374 INFO:teuthology.orchestra.run.smithi116.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-03-24T09:09:22.599 INFO:teuthology.orchestra.run.smithi116.stdout:(8/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 11 MB/s | 21 MB 00:01 2024-03-24T09:09:22.675 INFO:teuthology.orchestra.run.smithi116.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 335 kB/s | 25 kB 00:00 2024-03-24T09:09:22.774 INFO:teuthology.orchestra.run.smithi117.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.0 MB/s | 888 kB 00:00 2024-03-24T09:09:22.958 INFO:teuthology.orchestra.run.smithi117.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 789 kB/s | 145 kB 00:00 2024-03-24T09:09:23.354 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T09:09:23.355 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T09:09:23.369 INFO:teuthology.orchestra.run.smithi143.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T09:09:23.370 INFO:teuthology.orchestra.run.smithi143.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T09:09:23.371 INFO:teuthology.orchestra.run.smithi143.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T09:09:23.372 INFO:teuthology.orchestra.run.smithi143.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T09:09:23.375 INFO:teuthology.orchestra.run.smithi143.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T09:09:23.377 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T09:09:23.378 INFO:teuthology.orchestra.run.smithi143.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T09:09:23.458 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-03-24T09:09:23.474 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T09:09:23.475 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T09:09:23.476 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T09:09:23.477 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T09:09:23.478 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T09:09:23.479 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T09:09:23.480 INFO:teuthology.orchestra.run.smithi143.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T09:09:23.481 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T09:09:23.482 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T09:09:23.483 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T09:09:23.484 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T09:09:23.485 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T09:09:23.486 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T09:09:23.487 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T09:09:23.488 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T09:09:23.489 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T09:09:23.490 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T09:09:23.491 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T09:09:23.492 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T09:09:23.493 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:09:23.494 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-03-24T09:09:23.494 INFO:teuthology.orchestra.run.smithi143.stdout:============================================================================================ 2024-03-24T09:09:23.494 INFO:teuthology.orchestra.run.smithi143.stdout:Install 150 Packages 2024-03-24T09:09:23.494 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 24 Packages 2024-03-24T09:09:23.494 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:09:23.495 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 224 M 2024-03-24T09:09:23.495 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-03-24T09:09:23.642 INFO:teuthology.orchestra.run.smithi116.stdout:(10/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 7.7 MB/s | 9.8 MB 00:01 2024-03-24T09:09:23.733 INFO:teuthology.orchestra.run.smithi116.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 2.3 MB/s | 219 kB 00:00 2024-03-24T09:09:23.809 INFO:teuthology.orchestra.run.smithi116.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 360 kB/s | 27 kB 00:00 2024-03-24T09:09:23.884 INFO:teuthology.orchestra.run.smithi116.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 424 kB/s | 32 kB 00:00 2024-03-24T09:09:24.309 INFO:teuthology.orchestra.run.smithi116.stdout:(14/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 6.6 MB/s | 17 MB 00:02 2024-03-24T09:09:24.334 INFO:teuthology.orchestra.run.smithi116.stdout:(15/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 1.6 MB/s | 744 kB 00:00 2024-03-24T09:09:24.350 INFO:teuthology.orchestra.run.smithi117.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 1.7 MB/s | 2.4 MB 00:01 2024-03-24T09:09:24.377 INFO:teuthology.orchestra.run.smithi116.stdout:(16/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 2.4 MB/s | 163 kB 00:00 2024-03-24T09:09:24.401 INFO:teuthology.orchestra.run.smithi116.stdout:(17/174): librados-devel-19.0.0-1855.g21b3bd67. 1.9 MB/s | 127 kB 00:00 2024-03-24T09:09:24.534 INFO:teuthology.orchestra.run.smithi143.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-03-24T09:09:24.552 INFO:teuthology.orchestra.run.smithi116.stdout:(18/174): libradosstriper1-19.0.0-1855.g21b3bd6 2.8 MB/s | 499 kB 00:00 2024-03-24T09:09:24.652 INFO:teuthology.orchestra.run.smithi116.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 452 kB/s | 45 kB 00:00 2024-03-24T09:09:24.761 INFO:teuthology.orchestra.run.smithi116.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.2 MB/s | 131 kB 00:00 2024-03-24T09:09:24.878 INFO:teuthology.orchestra.run.smithi116.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.4 MB/s | 161 kB 00:00 2024-03-24T09:09:24.891 INFO:teuthology.orchestra.run.smithi117.stdout:(5/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-03-24T09:09:24.986 INFO:teuthology.orchestra.run.smithi116.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.9 MB/s | 320 kB 00:00 2024-03-24T09:09:25.103 INFO:teuthology.orchestra.run.smithi116.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.5 MB/s | 298 kB 00:00 2024-03-24T09:09:25.212 INFO:teuthology.orchestra.run.smithi116.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 917 kB/s | 99 kB 00:00 2024-03-24T09:09:25.312 INFO:teuthology.orchestra.run.smithi116.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 852 kB/s | 85 kB 00:00 2024-03-24T09:09:25.387 INFO:teuthology.orchestra.run.smithi116.stdout:(26/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 5.2 MB/s | 5.1 MB 00:00 2024-03-24T09:09:25.479 INFO:teuthology.orchestra.run.smithi116.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-24T09:09:25.571 INFO:teuthology.orchestra.run.smithi116.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 344 kB/s | 31 kB 00:00 2024-03-24T09:09:25.663 INFO:teuthology.orchestra.run.smithi116.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.5 MB/s | 138 kB 00:00 2024-03-24T09:09:25.926 INFO:teuthology.orchestra.run.smithi143.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 638 kB/s | 888 kB 00:01 2024-03-24T09:09:26.030 INFO:teuthology.orchestra.run.smithi116.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 4.3 MB/s | 3.1 MB 00:00 2024-03-24T09:09:26.089 INFO:teuthology.orchestra.run.smithi116.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 4.2 MB/s | 1.8 MB 00:00 2024-03-24T09:09:26.118 INFO:teuthology.orchestra.run.smithi143.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 755 kB/s | 145 kB 00:00 2024-03-24T09:09:26.172 INFO:teuthology.orchestra.run.smithi116.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.9 MB/s | 248 kB 00:00 2024-03-24T09:09:26.248 INFO:teuthology.orchestra.run.smithi116.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 657 kB/s | 49 kB 00:00 2024-03-24T09:09:26.283 INFO:teuthology.orchestra.run.smithi117.stdout:(6/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 3.4 MB/s | 4.7 MB 00:01 2024-03-24T09:09:26.608 INFO:teuthology.orchestra.run.smithi117.stdout:(7/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 1.1 MB/s | 5.3 MB 00:05 2024-03-24T09:09:26.609 INFO:teuthology.orchestra.run.smithi143.stdout:(4/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 2.3 MB/s | 5.3 MB 00:02 2024-03-24T09:09:26.657 INFO:teuthology.orchestra.run.smithi116.stdout:(34/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 12 MB/s | 47 MB 00:03 2024-03-24T09:09:26.681 INFO:teuthology.orchestra.run.smithi116.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1855.g2 35 kB/s | 15 kB 00:00 2024-03-24T09:09:26.807 INFO:teuthology.orchestra.run.smithi116.stdout:(36/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 1.7 MB/s | 264 kB 00:00 2024-03-24T09:09:26.849 INFO:teuthology.orchestra.run.smithi116.stdout:(37/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-24T09:09:26.916 INFO:teuthology.orchestra.run.smithi116.stdout:(38/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 167 kB/s | 18 kB 00:00 2024-03-24T09:09:26.943 INFO:teuthology.orchestra.run.smithi143.stdout:(5/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 4.5 MB/s | 1.5 MB 00:00 2024-03-24T09:09:27.016 INFO:teuthology.orchestra.run.smithi116.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-24T09:09:27.042 INFO:teuthology.orchestra.run.smithi116.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.3 MB/s | 920 kB 00:00 2024-03-24T09:09:27.067 INFO:teuthology.orchestra.run.smithi116.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-24T09:09:27.101 INFO:teuthology.orchestra.run.smithi116.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-24T09:09:27.126 INFO:teuthology.orchestra.run.smithi116.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 545 kB/s | 32 kB 00:00 2024-03-24T09:09:27.193 INFO:teuthology.orchestra.run.smithi116.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-24T09:09:27.218 INFO:teuthology.orchestra.run.smithi116.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 191 kB/s | 17 kB 00:00 2024-03-24T09:09:27.243 INFO:teuthology.orchestra.run.smithi116.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-24T09:09:27.269 INFO:teuthology.orchestra.run.smithi116.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-24T09:09:27.294 INFO:teuthology.orchestra.run.smithi116.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-24T09:09:27.319 INFO:teuthology.orchestra.run.smithi116.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T09:09:27.344 INFO:teuthology.orchestra.run.smithi116.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-24T09:09:27.370 INFO:teuthology.orchestra.run.smithi116.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 937 kB/s | 46 kB 00:00 2024-03-24T09:09:27.403 INFO:teuthology.orchestra.run.smithi116.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-24T09:09:27.429 INFO:teuthology.orchestra.run.smithi116.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-24T09:09:27.454 INFO:teuthology.orchestra.run.smithi116.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-24T09:09:27.468 INFO:teuthology.orchestra.run.smithi143.stdout:(6/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 1.7 MB/s | 2.4 MB 00:01 2024-03-24T09:09:27.479 INFO:teuthology.orchestra.run.smithi116.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-24T09:09:27.580 INFO:teuthology.orchestra.run.smithi116.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-24T09:09:27.622 INFO:teuthology.orchestra.run.smithi116.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-24T09:09:27.647 INFO:teuthology.orchestra.run.smithi116.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-24T09:09:27.672 INFO:teuthology.orchestra.run.smithi116.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-24T09:09:27.789 INFO:teuthology.orchestra.run.smithi116.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-24T09:09:27.801 INFO:teuthology.orchestra.run.smithi143.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 5.5 MB/s | 4.7 MB 00:00 2024-03-24T09:09:27.823 INFO:teuthology.orchestra.run.smithi116.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-24T09:09:27.848 INFO:teuthology.orchestra.run.smithi116.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T09:09:27.915 INFO:teuthology.orchestra.run.smithi116.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 379 kB/s | 24 kB 00:00 2024-03-24T09:09:27.990 INFO:teuthology.orchestra.run.smithi116.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-24T09:09:28.024 INFO:teuthology.orchestra.run.smithi116.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 167 kB/s | 18 kB 00:00 2024-03-24T09:09:28.050 INFO:teuthology.orchestra.run.smithi116.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-24T09:09:28.075 INFO:teuthology.orchestra.run.smithi116.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 634 kB/s | 32 kB 00:00 2024-03-24T09:09:28.167 INFO:teuthology.orchestra.run.smithi116.stdout:(68/174): ceph-mgr-diskprediction-local-19.0.0- 3.5 MB/s | 7.4 MB 00:02 2024-03-24T09:09:28.234 INFO:teuthology.orchestra.run.smithi116.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-24T09:09:28.259 INFO:teuthology.orchestra.run.smithi116.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-24T09:09:28.284 INFO:teuthology.orchestra.run.smithi116.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 174 kB 00:00 2024-03-24T09:09:28.310 INFO:teuthology.orchestra.run.smithi116.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T09:09:28.326 INFO:teuthology.orchestra.run.smithi116.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 618 kB/s | 41 kB 00:00 2024-03-24T09:09:28.352 INFO:teuthology.orchestra.run.smithi116.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-24T09:09:28.377 INFO:teuthology.orchestra.run.smithi116.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T09:09:28.402 INFO:teuthology.orchestra.run.smithi116.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-24T09:09:28.418 INFO:teuthology.orchestra.run.smithi143.stdout:(8/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 4.9 MB/s | 21 MB 00:04 2024-03-24T09:09:28.427 INFO:teuthology.orchestra.run.smithi116.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 634 kB/s | 47 kB 00:00 2024-03-24T09:09:28.453 INFO:teuthology.orchestra.run.smithi116.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2024-03-24T09:09:28.468 INFO:teuthology.orchestra.run.smithi143.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 507 kB/s | 25 kB 00:00 2024-03-24T09:09:28.586 INFO:teuthology.orchestra.run.smithi116.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 243 kB/s | 39 kB 00:00 2024-03-24T09:09:28.736 INFO:teuthology.orchestra.run.smithi116.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-24T09:09:28.762 INFO:teuthology.orchestra.run.smithi116.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 134 kB/s | 41 kB 00:00 2024-03-24T09:09:28.787 INFO:teuthology.orchestra.run.smithi116.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-24T09:09:28.812 INFO:teuthology.orchestra.run.smithi116.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T09:09:28.837 INFO:teuthology.orchestra.run.smithi116.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-24T09:09:28.862 INFO:teuthology.orchestra.run.smithi116.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-24T09:09:28.888 INFO:teuthology.orchestra.run.smithi116.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T09:09:28.913 INFO:teuthology.orchestra.run.smithi116.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-24T09:09:28.947 INFO:teuthology.orchestra.run.smithi116.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-24T09:09:28.972 INFO:teuthology.orchestra.run.smithi116.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-24T09:09:28.997 INFO:teuthology.orchestra.run.smithi116.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-24T09:09:29.039 INFO:teuthology.orchestra.run.smithi116.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-24T09:09:29.264 INFO:teuthology.orchestra.run.smithi116.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 703 kB/s | 187 kB 00:00 2024-03-24T09:09:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 732 kB/s | 195 kB 00:00 2024-03-24T09:09:29.319 INFO:teuthology.orchestra.run.smithi143.stdout:(10/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 6.4 MB/s | 9.8 MB 00:01 2024-03-24T09:09:29.340 INFO:teuthology.orchestra.run.smithi116.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-24T09:09:29.365 INFO:teuthology.orchestra.run.smithi116.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-24T09:09:29.399 INFO:teuthology.orchestra.run.smithi116.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 432 kB/s | 25 kB 00:00 2024-03-24T09:09:29.410 INFO:teuthology.orchestra.run.smithi143.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 2.3 MB/s | 219 kB 00:00 2024-03-24T09:09:29.424 INFO:teuthology.orchestra.run.smithi116.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 183 kB/s | 21 kB 00:00 2024-03-24T09:09:29.486 INFO:teuthology.orchestra.run.smithi143.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 360 kB/s | 27 kB 00:00 2024-03-24T09:09:29.491 INFO:teuthology.orchestra.run.smithi116.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-24T09:09:29.516 INFO:teuthology.orchestra.run.smithi116.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-24T09:09:29.561 INFO:teuthology.orchestra.run.smithi143.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 423 kB/s | 32 kB 00:00 2024-03-24T09:09:29.567 INFO:teuthology.orchestra.run.smithi116.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 697 kB/s | 35 kB 00:00 2024-03-24T09:09:29.592 INFO:teuthology.orchestra.run.smithi116.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-24T09:09:29.684 INFO:teuthology.orchestra.run.smithi116.stdout:(102/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-03-24T09:09:29.709 INFO:teuthology.orchestra.run.smithi116.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 313 kB/s | 44 kB 00:00 2024-03-24T09:09:29.734 INFO:teuthology.orchestra.run.smithi116.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 659 kB/s | 94 kB 00:00 2024-03-24T09:09:29.801 INFO:teuthology.orchestra.run.smithi116.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-03-24T09:09:29.828 INFO:teuthology.orchestra.run.smithi143.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 2.7 MB/s | 744 kB 00:00 2024-03-24T09:09:29.903 INFO:teuthology.orchestra.run.smithi143.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 2.1 MB/s | 163 kB 00:00 2024-03-24T09:09:29.910 INFO:teuthology.orchestra.run.smithi116.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-03-24T09:09:29.978 INFO:teuthology.orchestra.run.smithi143.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 1.7 MB/s | 127 kB 00:00 2024-03-24T09:09:29.999 INFO:teuthology.orchestra.run.smithi117.stdout:(8/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x86 4.5 MB/s | 17 MB 00:03 2024-03-24T09:09:30.018 INFO:teuthology.orchestra.run.smithi116.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-24T09:09:30.044 INFO:teuthology.orchestra.run.smithi116.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-03-24T09:09:30.069 INFO:teuthology.orchestra.run.smithi116.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 122 kB/s | 19 kB 00:00 2024-03-24T09:09:30.099 INFO:teuthology.orchestra.run.smithi117.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 252 kB/s | 25 kB 00:00 2024-03-24T09:09:30.103 INFO:teuthology.orchestra.run.smithi116.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 444 kB/s | 26 kB 00:00 2024-03-24T09:09:30.120 INFO:teuthology.orchestra.run.smithi143.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 3.4 MB/s | 499 kB 00:00 2024-03-24T09:09:30.161 INFO:teuthology.orchestra.run.smithi116.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-24T09:09:30.187 INFO:teuthology.orchestra.run.smithi116.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 414 kB/s | 49 kB 00:00 2024-03-24T09:09:30.262 INFO:teuthology.orchestra.run.smithi116.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-03-24T09:09:30.305 INFO:teuthology.orchestra.run.smithi116.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-24T09:09:30.337 INFO:teuthology.orchestra.run.smithi116.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-24T09:09:30.363 INFO:teuthology.orchestra.run.smithi116.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 756 kB/s | 43 kB 00:00 2024-03-24T09:09:30.404 INFO:teuthology.orchestra.run.smithi143.stdout:(18/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 5.7 MB/s | 17 MB 00:02 2024-03-24T09:09:30.405 INFO:teuthology.orchestra.run.smithi116.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-03-24T09:09:30.438 INFO:teuthology.orchestra.run.smithi116.stdout:(118/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-24T09:09:30.464 INFO:teuthology.orchestra.run.smithi116.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-24T09:09:30.497 INFO:teuthology.orchestra.run.smithi116.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.8 MB/s | 172 kB 00:00 2024-03-24T09:09:30.504 INFO:teuthology.orchestra.run.smithi143.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 452 kB/s | 45 kB 00:00 2024-03-24T09:09:30.522 INFO:teuthology.orchestra.run.smithi116.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-24T09:09:30.548 INFO:teuthology.orchestra.run.smithi116.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-24T09:09:30.573 INFO:teuthology.orchestra.run.smithi116.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 467 kB/s | 35 kB 00:00 2024-03-24T09:09:30.607 INFO:teuthology.orchestra.run.smithi116.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-24T09:09:30.613 INFO:teuthology.orchestra.run.smithi143.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.2 MB/s | 131 kB 00:00 2024-03-24T09:09:30.641 INFO:teuthology.orchestra.run.smithi116.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-24T09:09:30.674 INFO:teuthology.orchestra.run.smithi116.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-03-24T09:09:30.700 INFO:teuthology.orchestra.run.smithi116.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-24T09:09:30.713 INFO:teuthology.orchestra.run.smithi143.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 1.6 MB/s | 161 kB 00:00 2024-03-24T09:09:30.725 INFO:teuthology.orchestra.run.smithi116.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-24T09:09:30.775 INFO:teuthology.orchestra.run.smithi116.stdout:(129/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-03-24T09:09:30.792 INFO:teuthology.orchestra.run.smithi116.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-24T09:09:30.809 INFO:teuthology.orchestra.run.smithi116.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-24T09:09:30.821 INFO:teuthology.orchestra.run.smithi143.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 2.9 MB/s | 320 kB 00:00 2024-03-24T09:09:30.859 INFO:teuthology.orchestra.run.smithi116.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-03-24T09:09:30.876 INFO:teuthology.orchestra.run.smithi116.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 691 kB/s | 46 kB 00:00 2024-03-24T09:09:30.901 INFO:teuthology.orchestra.run.smithi116.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 855 kB/s | 79 kB 00:00 2024-03-24T09:09:30.927 INFO:teuthology.orchestra.run.smithi116.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 860 kB/s | 58 kB 00:00 2024-03-24T09:09:30.930 INFO:teuthology.orchestra.run.smithi143.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.7 MB/s | 298 kB 00:00 2024-03-24T09:09:30.969 INFO:teuthology.orchestra.run.smithi116.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-03-24T09:09:30.994 INFO:teuthology.orchestra.run.smithi116.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-24T09:09:31.019 INFO:teuthology.orchestra.run.smithi116.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 977 kB/s | 90 kB 00:00 2024-03-24T09:09:31.031 INFO:teuthology.orchestra.run.smithi143.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 991 kB/s | 99 kB 00:00 2024-03-24T09:09:31.094 INFO:teuthology.orchestra.run.smithi116.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-03-24T09:09:31.136 INFO:teuthology.orchestra.run.smithi116.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 264 kB/s | 31 kB 00:00 2024-03-24T09:09:31.164 INFO:teuthology.orchestra.run.smithi143.stdout:(25/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 4.9 MB/s | 5.1 MB 00:01 2024-03-24T09:09:31.170 INFO:teuthology.orchestra.run.smithi116.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-03-24T09:09:31.175 INFO:teuthology.orchestra.run.smithi117.stdout:(10/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 2.1 MB/s | 9.8 MB 00:04 2024-03-24T09:09:31.189 INFO:teuthology.orchestra.run.smithi143.stdout:(26/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 538 kB/s | 85 kB 00:00 2024-03-24T09:09:31.195 INFO:teuthology.orchestra.run.smithi116.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-03-24T09:09:31.221 INFO:teuthology.orchestra.run.smithi116.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 469 kB/s | 59 kB 00:00 2024-03-24T09:09:31.246 INFO:teuthology.orchestra.run.smithi116.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-24T09:09:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-24T09:09:31.298 INFO:teuthology.orchestra.run.smithi143.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.5 MB/s | 170 kB 00:00 2024-03-24T09:09:31.304 INFO:teuthology.orchestra.run.smithi116.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-24T09:09:31.308 INFO:teuthology.orchestra.run.smithi117.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.6 MB/s | 219 kB 00:00 2024-03-24T09:09:31.330 INFO:teuthology.orchestra.run.smithi116.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-24T09:09:31.355 INFO:teuthology.orchestra.run.smithi116.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-24T09:09:31.375 INFO:teuthology.orchestra.run.smithi117.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 407 kB/s | 27 kB 00:00 2024-03-24T09:09:31.381 INFO:teuthology.orchestra.run.smithi143.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 377 kB/s | 31 kB 00:00 2024-03-24T09:09:31.389 INFO:teuthology.orchestra.run.smithi116.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-24T09:09:31.425 INFO:teuthology.orchestra.run.smithi117.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 637 kB/s | 32 kB 00:00 2024-03-24T09:09:31.439 INFO:teuthology.orchestra.run.smithi116.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-24T09:09:31.490 INFO:teuthology.orchestra.run.smithi143.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.2 MB/s | 138 kB 00:00 2024-03-24T09:09:31.614 INFO:teuthology.orchestra.run.smithi116.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.6 MB/s | 321 kB 00:00 2024-03-24T09:09:31.757 INFO:teuthology.orchestra.run.smithi143.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 5.2 MB/s | 3.1 MB 00:00 2024-03-24T09:09:31.792 INFO:teuthology.orchestra.run.smithi117.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 2.0 MB/s | 744 kB 00:00 2024-03-24T09:09:31.798 INFO:teuthology.orchestra.run.smithi143.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 5.8 MB/s | 1.8 MB 00:00 2024-03-24T09:09:31.823 INFO:teuthology.orchestra.run.smithi116.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-24T09:09:31.915 INFO:teuthology.orchestra.run.smithi116.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 20 MB/s | 1.8 MB 00:00 2024-03-24T09:09:31.916 INFO:teuthology.orchestra.run.smithi143.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.1 MB/s | 248 kB 00:00 2024-03-24T09:09:31.943 INFO:teuthology.orchestra.run.smithi117.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.1 MB/s | 163 kB 00:00 2024-03-24T09:09:31.973 INFO:teuthology.orchestra.run.smithi116.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 721 kB/s | 42 kB 00:00 2024-03-24T09:09:31.999 INFO:teuthology.orchestra.run.smithi143.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 592 kB/s | 49 kB 00:00 2024-03-24T09:09:32.040 INFO:teuthology.orchestra.run.smithi116.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 3.6 MB/s | 248 kB 00:00 2024-03-24T09:09:32.043 INFO:teuthology.orchestra.run.smithi117.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 1.2 MB/s | 127 kB 00:00 2024-03-24T09:09:32.091 INFO:teuthology.orchestra.run.smithi143.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 167 kB/s | 15 kB 00:00 2024-03-24T09:09:32.107 INFO:teuthology.orchestra.run.smithi116.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-24T09:09:32.166 INFO:teuthology.orchestra.run.smithi116.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 625 kB/s | 36 kB 00:00 2024-03-24T09:09:32.200 INFO:teuthology.orchestra.run.smithi143.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.4 MB/s | 264 kB 00:00 2024-03-24T09:09:32.225 INFO:teuthology.orchestra.run.smithi116.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-24T09:09:32.291 INFO:teuthology.orchestra.run.smithi116.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.8 MB/s | 256 kB 00:00 2024-03-24T09:09:32.350 INFO:teuthology.orchestra.run.smithi116.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-03-24T09:09:32.352 INFO:teuthology.orchestra.run.smithi117.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 1.6 MB/s | 499 kB 00:00 2024-03-24T09:09:32.409 INFO:teuthology.orchestra.run.smithi116.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-24T09:09:32.467 INFO:teuthology.orchestra.run.smithi116.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-24T09:09:32.526 INFO:teuthology.orchestra.run.smithi116.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-24T09:09:32.550 INFO:teuthology.orchestra.run.smithi143.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 2.1 MB/s | 748 kB 00:00 2024-03-24T09:09:32.584 INFO:teuthology.orchestra.run.smithi116.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-24T09:09:32.642 INFO:teuthology.orchestra.run.smithi143.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 199 kB/s | 18 kB 00:00 2024-03-24T09:09:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 492 kB/s | 29 kB 00:00 2024-03-24T09:09:32.735 INFO:teuthology.orchestra.run.smithi116.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2024-03-24T09:09:32.835 INFO:teuthology.orchestra.run.smithi116.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2024-03-24T09:09:32.936 INFO:teuthology.orchestra.run.smithi116.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.3 MB 00:00 2024-03-24T09:09:32.960 INFO:teuthology.orchestra.run.smithi117.stdout:(18/174): ceph-common-19.0.0-1855.g21b3bd67.el9 1.8 MB/s | 21 MB 00:11 2024-03-24T09:09:33.003 INFO:teuthology.orchestra.run.smithi116.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-03-24T09:09:33.010 INFO:teuthology.orchestra.run.smithi117.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 910 kB/s | 45 kB 00:00 2024-03-24T09:09:33.053 INFO:teuthology.orchestra.run.smithi116.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 625 kB/s | 31 kB 00:00 2024-03-24T09:09:33.076 INFO:teuthology.orchestra.run.smithi143.stdout:(38/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 10 MB/s | 47 MB 00:04 2024-03-24T09:09:33.085 INFO:teuthology.orchestra.run.smithi117.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.7 MB/s | 131 kB 00:00 2024-03-24T09:09:33.136 INFO:teuthology.orchestra.run.smithi117.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 3.2 MB/s | 161 kB 00:00 2024-03-24T09:09:33.136 INFO:teuthology.orchestra.run.smithi116.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-03-24T09:09:33.201 INFO:teuthology.orchestra.run.smithi143.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-03-24T09:09:33.211 INFO:teuthology.orchestra.run.smithi117.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 4.2 MB/s | 320 kB 00:00 2024-03-24T09:09:33.220 INFO:teuthology.orchestra.run.smithi116.stdout:(172/174): librados2-19.0.0-1855.g21b3bd67.el9. 1.8 MB/s | 3.4 MB 00:01 2024-03-24T09:09:33.245 INFO:teuthology.orchestra.run.smithi116.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 684 kB/s | 74 kB 00:00 2024-03-24T09:09:33.294 INFO:teuthology.orchestra.run.smithi117.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.5 MB/s | 298 kB 00:00 2024-03-24T09:09:33.318 INFO:teuthology.orchestra.run.smithi143.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 980 kB/s | 662 kB 00:00 2024-03-24T09:09:33.343 INFO:teuthology.orchestra.run.smithi143.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 921 kB/s | 106 kB 00:00 2024-03-24T09:09:33.353 INFO:teuthology.orchestra.run.smithi117.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.7 MB/s | 99 kB 00:00 2024-03-24T09:09:33.368 INFO:teuthology.orchestra.run.smithi143.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-24T09:09:33.393 INFO:teuthology.orchestra.run.smithi143.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 307 kB 00:00 2024-03-24T09:09:33.410 INFO:teuthology.orchestra.run.smithi143.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 476 kB/s | 32 kB 00:00 2024-03-24T09:09:33.411 INFO:teuthology.orchestra.run.smithi117.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-24T09:09:33.427 INFO:teuthology.orchestra.run.smithi143.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 513 kB/s | 17 kB 00:00 2024-03-24T09:09:33.502 INFO:teuthology.orchestra.run.smithi143.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 27 MB/s | 3.0 MB 00:00 2024-03-24T09:09:33.504 INFO:teuthology.orchestra.run.smithi117.stdout:(26/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 4.5 MB/s | 5.1 MB 00:01 2024-03-24T09:09:33.512 INFO:teuthology.orchestra.run.smithi116.stdout:(174/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 1.5 MB/s | 3.1 MB 00:02 2024-03-24T09:09:33.518 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:09:33.518 INFO:teuthology.orchestra.run.smithi116.stdout:Total 17 MB/s | 224 MB 00:13 2024-03-24T09:09:33.519 INFO:teuthology.orchestra.run.smithi143.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 92 kB/s | 10 kB 00:00 2024-03-24T09:09:33.545 INFO:teuthology.orchestra.run.smithi143.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-24T09:09:33.545 INFO:teuthology.orchestra.run.smithi117.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 4.0 MB/s | 170 kB 00:00 2024-03-24T09:09:33.561 INFO:teuthology.orchestra.run.smithi143.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 758 kB/s | 44 kB 00:00 2024-03-24T09:09:33.579 INFO:teuthology.orchestra.run.smithi117.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 946 kB/s | 31 kB 00:00 2024-03-24T09:09:33.595 INFO:teuthology.orchestra.run.smithi143.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-24T09:09:33.629 INFO:teuthology.orchestra.run.smithi143.stdout:(51/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-24T09:09:33.629 INFO:teuthology.orchestra.run.smithi117.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 2.7 MB/s | 138 kB 00:00 2024-03-24T09:09:33.645 INFO:teuthology.orchestra.run.smithi143.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 553 kB/s | 46 kB 00:00 2024-03-24T09:09:33.671 INFO:teuthology.orchestra.run.smithi143.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-24T09:09:33.696 INFO:teuthology.orchestra.run.smithi143.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T09:09:33.721 INFO:teuthology.orchestra.run.smithi143.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-24T09:09:33.746 INFO:teuthology.orchestra.run.smithi143.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 454 kB/s | 34 kB 00:00 2024-03-24T09:09:33.796 INFO:teuthology.orchestra.run.smithi117.stdout:(30/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 8.0 MB/s | 3.1 MB 00:00 2024-03-24T09:09:33.847 INFO:teuthology.orchestra.run.smithi143.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-24T09:09:33.921 INFO:teuthology.orchestra.run.smithi117.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 6.1 MB/s | 1.8 MB 00:00 2024-03-24T09:09:33.939 INFO:teuthology.orchestra.run.smithi143.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-24T09:09:33.972 INFO:teuthology.orchestra.run.smithi143.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 3.5 MB/s | 811 kB 00:00 2024-03-24T09:09:33.997 INFO:teuthology.orchestra.run.smithi143.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 905 kB/s | 134 kB 00:00 2024-03-24T09:09:34.000 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-24T09:09:34.005 INFO:teuthology.orchestra.run.smithi117.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.9 MB/s | 248 kB 00:00 2024-03-24T09:09:34.063 INFO:teuthology.orchestra.run.smithi117.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 845 kB/s | 49 kB 00:00 2024-03-24T09:09:34.098 INFO:teuthology.orchestra.run.smithi143.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-24T09:09:34.122 INFO:teuthology.orchestra.run.smithi117.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 263 kB/s | 15 kB 00:00 2024-03-24T09:09:34.123 INFO:teuthology.orchestra.run.smithi143.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 634 kB/s | 78 kB 00:00 2024-03-24T09:09:34.156 INFO:teuthology.orchestra.run.smithi143.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-24T09:09:34.231 INFO:teuthology.orchestra.run.smithi117.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.4 MB/s | 264 kB 00:00 2024-03-24T09:09:34.240 INFO:teuthology.orchestra.run.smithi143.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-24T09:09:34.266 INFO:teuthology.orchestra.run.smithi143.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 177 kB/s | 24 kB 00:00 2024-03-24T09:09:34.277 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-24T09:09:34.277 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-24T09:09:34.290 INFO:teuthology.orchestra.run.smithi143.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-24T09:09:34.316 INFO:teuthology.orchestra.run.smithi143.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T09:09:34.333 INFO:teuthology.orchestra.run.smithi143.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 761 kB/s | 32 kB 00:00 2024-03-24T09:09:34.356 INFO:teuthology.orchestra.run.smithi117.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 5.8 MB/s | 748 kB 00:00 2024-03-24T09:09:34.406 INFO:teuthology.orchestra.run.smithi117.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 366 kB/s | 18 kB 00:00 2024-03-24T09:09:34.408 INFO:teuthology.orchestra.run.smithi143.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 4.8 MB 00:00 2024-03-24T09:09:34.433 INFO:teuthology.orchestra.run.smithi143.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2024-03-24T09:09:34.459 INFO:teuthology.orchestra.run.smithi143.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 174 kB 00:00 2024-03-24T09:09:34.500 INFO:teuthology.orchestra.run.smithi143.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 753 kB/s | 69 kB 00:00 2024-03-24T09:09:34.517 INFO:teuthology.orchestra.run.smithi143.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 708 kB/s | 41 kB 00:00 2024-03-24T09:09:34.542 INFO:teuthology.orchestra.run.smithi143.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-24T09:09:34.548 INFO:teuthology.orchestra.run.smithi117.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-03-24T09:09:34.568 INFO:teuthology.orchestra.run.smithi143.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-24T09:09:34.585 INFO:teuthology.orchestra.run.smithi143.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-24T09:09:34.610 INFO:teuthology.orchestra.run.smithi143.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-24T09:09:34.627 INFO:teuthology.orchestra.run.smithi143.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 733 kB/s | 43 kB 00:00 2024-03-24T09:09:34.641 INFO:teuthology.orchestra.run.smithi117.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 8.8 MB/s | 7.4 MB 00:00 2024-03-24T09:09:34.682 INFO:teuthology.orchestra.run.smithi117.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.7 MB/s | 920 kB 00:00 2024-03-24T09:09:34.708 INFO:teuthology.orchestra.run.smithi117.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T09:09:34.733 INFO:teuthology.orchestra.run.smithi117.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T09:09:34.758 INFO:teuthology.orchestra.run.smithi117.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-24T09:09:34.827 INFO:teuthology.orchestra.run.smithi143.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 196 kB/s | 39 kB 00:00 2024-03-24T09:09:34.844 INFO:teuthology.orchestra.run.smithi143.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 191 kB/s | 41 kB 00:00 2024-03-24T09:09:34.858 INFO:teuthology.orchestra.run.smithi117.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-24T09:09:34.883 INFO:teuthology.orchestra.run.smithi117.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 139 kB/s | 17 kB 00:00 2024-03-24T09:09:34.917 INFO:teuthology.orchestra.run.smithi117.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 304 kB/s | 10 kB 00:00 2024-03-24T09:09:34.942 INFO:teuthology.orchestra.run.smithi117.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-24T09:09:34.968 INFO:teuthology.orchestra.run.smithi117.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-24T09:09:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T09:09:35.002 INFO:teuthology.orchestra.run.smithi143.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-24T09:09:35.018 INFO:teuthology.orchestra.run.smithi117.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-24T09:09:35.028 INFO:teuthology.orchestra.run.smithi143.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-24T09:09:35.043 INFO:teuthology.orchestra.run.smithi117.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 943 kB/s | 46 kB 00:00 2024-03-24T09:09:35.045 INFO:teuthology.orchestra.run.smithi143.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 284 kB/s | 56 kB 00:00 2024-03-24T09:09:35.061 INFO:teuthology.orchestra.run.smithi143.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T09:09:35.069 INFO:teuthology.orchestra.run.smithi117.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T09:09:35.078 INFO:teuthology.orchestra.run.smithi143.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T09:09:35.094 INFO:teuthology.orchestra.run.smithi117.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T09:09:35.103 INFO:teuthology.orchestra.run.smithi143.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-24T09:09:35.119 INFO:teuthology.orchestra.run.smithi117.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-24T09:09:35.128 INFO:teuthology.orchestra.run.smithi143.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-24T09:09:35.145 INFO:teuthology.orchestra.run.smithi117.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2024-03-24T09:09:35.154 INFO:teuthology.orchestra.run.smithi143.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T09:09:35.188 INFO:teuthology.orchestra.run.smithi143.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-24T09:09:35.213 INFO:teuthology.orchestra.run.smithi143.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-03-24T09:09:35.238 INFO:teuthology.orchestra.run.smithi143.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-24T09:09:35.253 INFO:teuthology.orchestra.run.smithi117.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-03-24T09:09:35.295 INFO:teuthology.orchestra.run.smithi117.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-24T09:09:35.329 INFO:teuthology.orchestra.run.smithi117.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-24T09:09:35.354 INFO:teuthology.orchestra.run.smithi117.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T09:09:35.471 INFO:teuthology.orchestra.run.smithi117.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-24T09:09:35.513 INFO:teuthology.orchestra.run.smithi117.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-24T09:09:35.530 INFO:teuthology.orchestra.run.smithi117.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T09:09:35.663 INFO:teuthology.orchestra.run.smithi117.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-24T09:09:35.689 INFO:teuthology.orchestra.run.smithi117.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 155 kB/s | 24 kB 00:00 2024-03-24T09:09:35.723 INFO:teuthology.orchestra.run.smithi117.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 308 kB/s | 18 kB 00:00 2024-03-24T09:09:35.748 INFO:teuthology.orchestra.run.smithi117.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-24T09:09:35.841 INFO:teuthology.orchestra.run.smithi117.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 268 kB/s | 32 kB 00:00 2024-03-24T09:09:35.965 INFO:teuthology.orchestra.run.smithi117.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-24T09:09:35.990 INFO:teuthology.orchestra.run.smithi117.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-24T09:09:36.016 INFO:teuthology.orchestra.run.smithi117.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-24T09:09:36.041 INFO:teuthology.orchestra.run.smithi117.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T09:09:36.066 INFO:teuthology.orchestra.run.smithi117.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 824 kB/s | 41 kB 00:00 2024-03-24T09:09:36.088 INFO:teuthology.orchestra.run.smithi143.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 214 kB/s | 187 kB 00:00 2024-03-24T09:09:36.091 INFO:teuthology.orchestra.run.smithi117.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-24T09:09:36.117 INFO:teuthology.orchestra.run.smithi117.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T09:09:36.142 INFO:teuthology.orchestra.run.smithi117.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T09:09:36.146 INFO:teuthology.orchestra.run.smithi143.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 215 kB/s | 195 kB 00:00 2024-03-24T09:09:36.167 INFO:teuthology.orchestra.run.smithi117.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-24T09:09:36.209 INFO:teuthology.orchestra.run.smithi117.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 639 kB/s | 43 kB 00:00 2024-03-24T09:09:36.222 INFO:teuthology.orchestra.run.smithi143.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 289 kB/s | 21 kB 00:00 2024-03-24T09:09:36.272 INFO:teuthology.orchestra.run.smithi143.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-03-24T09:09:36.333 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-24T09:09:36.334 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-24T09:09:36.385 INFO:teuthology.orchestra.run.smithi117.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 224 kB/s | 39 kB 00:00 2024-03-24T09:09:36.497 INFO:teuthology.orchestra.run.smithi143.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-03-24T09:09:36.535 INFO:teuthology.orchestra.run.smithi117.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-24T09:09:36.539 INFO:teuthology.orchestra.run.smithi143.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 5.8 MB/s | 245 kB 00:00 2024-03-24T09:09:36.560 INFO:teuthology.orchestra.run.smithi117.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 234 kB/s | 41 kB 00:00 2024-03-24T09:09:36.581 INFO:teuthology.orchestra.run.smithi143.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-03-24T09:09:36.585 INFO:teuthology.orchestra.run.smithi117.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T09:09:36.611 INFO:teuthology.orchestra.run.smithi117.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T09:09:36.623 INFO:teuthology.orchestra.run.smithi143.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 216 kB 00:00 2024-03-24T09:09:36.636 INFO:teuthology.orchestra.run.smithi117.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T09:09:36.661 INFO:teuthology.orchestra.run.smithi117.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-24T09:09:36.665 INFO:teuthology.orchestra.run.smithi143.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 841 kB/s | 35 kB 00:00 2024-03-24T09:09:36.686 INFO:teuthology.orchestra.run.smithi117.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T09:09:36.707 INFO:teuthology.orchestra.run.smithi143.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-24T09:09:36.712 INFO:teuthology.orchestra.run.smithi117.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T09:09:36.745 INFO:teuthology.orchestra.run.smithi117.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-24T09:09:36.749 INFO:teuthology.orchestra.run.smithi143.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-24T09:09:36.762 INFO:teuthology.orchestra.run.smithi117.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-24T09:09:36.779 INFO:teuthology.orchestra.run.smithi117.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 3.6 MB/s | 124 kB 00:00 2024-03-24T09:09:36.805 INFO:teuthology.orchestra.run.smithi117.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-24T09:09:36.858 INFO:teuthology.orchestra.run.smithi143.stdout:(103/174): python3-requests-2.25.1-8.el9.noarch 162 kB/s | 125 kB 00:00 2024-03-24T09:09:36.908 INFO:teuthology.orchestra.run.smithi143.stdout:(104/174): libgfortran-11.4.1-3.el9.x86_64.rpm 474 kB/s | 803 kB 00:01 2024-03-24T09:09:36.966 INFO:teuthology.orchestra.run.smithi143.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-24T09:09:36.980 INFO:teuthology.orchestra.run.smithi117.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 936 kB/s | 187 kB 00:00 2024-03-24T09:09:37.063 INFO:teuthology.orchestra.run.smithi117.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-24T09:09:37.122 INFO:teuthology.orchestra.run.smithi117.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-03-24T09:09:37.156 INFO:teuthology.orchestra.run.smithi117.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-24T09:09:37.281 INFO:teuthology.orchestra.run.smithi117.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 135 kB/s | 21 kB 00:00 2024-03-24T09:09:37.298 INFO:teuthology.orchestra.run.smithi117.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 179 kB/s | 25 kB 00:00 2024-03-24T09:09:37.334 INFO:teuthology.orchestra.run.smithi143.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-03-24T09:09:37.367 INFO:teuthology.orchestra.run.smithi143.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 825 kB/s | 309 kB 00:00 2024-03-24T09:09:37.409 INFO:teuthology.orchestra.run.smithi143.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 260 kB/s | 19 kB 00:00 2024-03-24T09:09:37.457 INFO:teuthology.orchestra.run.smithi117.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-24T09:09:37.467 INFO:teuthology.orchestra.run.smithi143.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-03-24T09:09:37.523 INFO:teuthology.orchestra.run.smithi117.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-03-24T09:09:37.526 INFO:teuthology.orchestra.run.smithi143.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 836 kB/s | 49 kB 00:00 2024-03-24T09:09:37.591 INFO:teuthology.orchestra.run.smithi117.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 3.2 MB/s | 216 kB 00:00 2024-03-24T09:09:37.593 INFO:teuthology.orchestra.run.smithi143.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-24T09:09:37.641 INFO:teuthology.orchestra.run.smithi117.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 697 kB/s | 35 kB 00:00 2024-03-24T09:09:37.668 INFO:teuthology.orchestra.run.smithi143.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-24T09:09:37.693 INFO:teuthology.orchestra.run.smithi143.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-24T09:09:37.733 INFO:teuthology.orchestra.run.smithi117.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 9.2 MB/s | 4.1 MB 00:00 2024-03-24T09:09:37.744 INFO:teuthology.orchestra.run.smithi143.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-24T09:09:37.750 INFO:teuthology.orchestra.run.smithi117.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 410 kB/s | 44 kB 00:00 2024-03-24T09:09:37.792 INFO:teuthology.orchestra.run.smithi117.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-24T09:09:37.811 INFO:teuthology.orchestra.run.smithi143.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-24T09:09:37.836 INFO:teuthology.orchestra.run.smithi143.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-03-24T09:09:37.880 INFO:teuthology.orchestra.run.smithi117.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 171 kB/s | 22 kB 00:00 2024-03-24T09:09:37.886 INFO:teuthology.orchestra.run.smithi143.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-24T09:09:37.911 INFO:teuthology.orchestra.run.smithi143.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-03-24T09:09:37.970 INFO:teuthology.orchestra.run.smithi143.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-24T09:09:37.995 INFO:teuthology.orchestra.run.smithi143.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-24T09:09:38.034 INFO:teuthology.orchestra.run.smithi117.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-03-24T09:09:38.054 INFO:teuthology.orchestra.run.smithi143.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-03-24T09:09:38.079 INFO:teuthology.orchestra.run.smithi143.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-03-24T09:09:38.138 INFO:teuthology.orchestra.run.smithi143.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-24T09:09:38.188 INFO:teuthology.orchestra.run.smithi143.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-03-24T09:09:38.221 INFO:teuthology.orchestra.run.smithi143.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-24T09:09:38.264 INFO:teuthology.orchestra.run.smithi143.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-24T09:09:38.289 INFO:teuthology.orchestra.run.smithi143.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-03-24T09:09:38.331 INFO:teuthology.orchestra.run.smithi143.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-03-24T09:09:38.356 INFO:teuthology.orchestra.run.smithi143.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 291 kB/s | 19 kB 00:00 2024-03-24T09:09:38.359 INFO:teuthology.orchestra.run.smithi117.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 951 kB/s | 309 kB 00:00 2024-03-24T09:09:38.384 INFO:teuthology.orchestra.run.smithi117.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-24T09:09:38.390 INFO:teuthology.orchestra.run.smithi143.stdout:(130/174): protobuf-compiler-3.14.0-13.el9.x86_ 564 kB/s | 863 kB 00:01 2024-03-24T09:09:38.410 INFO:teuthology.orchestra.run.smithi117.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 386 kB/s | 19 kB 00:00 2024-03-24T09:09:38.415 INFO:teuthology.orchestra.run.smithi143.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-03-24T09:09:38.460 INFO:teuthology.orchestra.run.smithi117.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 519 kB/s | 26 kB 00:00 2024-03-24T09:09:38.490 INFO:teuthology.orchestra.run.smithi143.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-03-24T09:09:38.510 INFO:teuthology.orchestra.run.smithi117.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 976 kB/s | 49 kB 00:00 2024-03-24T09:09:38.516 INFO:teuthology.orchestra.run.smithi143.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 784 kB/s | 79 kB 00:00 2024-03-24T09:09:38.566 INFO:teuthology.orchestra.run.smithi143.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-03-24T09:09:38.569 INFO:teuthology.orchestra.run.smithi117.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-24T09:09:38.599 INFO:teuthology.orchestra.run.smithi143.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-24T09:09:38.636 INFO:teuthology.orchestra.run.smithi117.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-24T09:09:38.641 INFO:teuthology.orchestra.run.smithi143.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-24T09:09:38.666 INFO:teuthology.orchestra.run.smithi143.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-24T09:09:38.692 INFO:teuthology.orchestra.run.smithi143.stdout:(138/174): python3-logutils-0.3.5-21.el9.noarch 153 kB/s | 46 kB 00:00 2024-03-24T09:09:38.717 INFO:teuthology.orchestra.run.smithi143.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-24T09:09:38.742 INFO:teuthology.orchestra.run.smithi143.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-24T09:09:38.769 INFO:teuthology.orchestra.run.smithi117.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 838 kB 00:00 2024-03-24T09:09:38.792 INFO:teuthology.orchestra.run.smithi143.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 787 kB/s | 59 kB 00:00 2024-03-24T09:09:38.818 INFO:teuthology.orchestra.run.smithi143.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 477 kB/s | 36 kB 00:00 2024-03-24T09:09:38.860 INFO:teuthology.orchestra.run.smithi143.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-03-24T09:09:38.895 INFO:teuthology.orchestra.run.smithi117.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-03-24T09:09:38.953 INFO:teuthology.orchestra.run.smithi117.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 745 kB/s | 43 kB 00:00 2024-03-24T09:09:38.959 INFO:teuthology.orchestra.run.smithi143.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 307 kB/s | 51 kB 00:00 2024-03-24T09:09:39.002 INFO:teuthology.orchestra.run.smithi143.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 632 kB/s | 90 kB 00:00 2024-03-24T09:09:39.012 INFO:teuthology.orchestra.run.smithi117.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 553 kB/s | 32 kB 00:00 2024-03-24T09:09:39.027 INFO:teuthology.orchestra.run.smithi143.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-03-24T09:09:39.061 INFO:teuthology.orchestra.run.smithi143.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-03-24T09:09:39.070 INFO:teuthology.orchestra.run.smithi117.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-24T09:09:39.086 INFO:teuthology.orchestra.run.smithi143.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-24T09:09:39.111 INFO:teuthology.orchestra.run.smithi143.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-24T09:09:39.146 INFO:teuthology.orchestra.run.smithi117.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-24T09:09:39.170 INFO:teuthology.orchestra.run.smithi143.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-24T09:09:39.220 INFO:teuthology.orchestra.run.smithi143.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-24T09:09:39.246 INFO:teuthology.orchestra.run.smithi117.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-03-24T09:09:39.271 INFO:teuthology.orchestra.run.smithi117.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-24T09:09:39.322 INFO:teuthology.orchestra.run.smithi117.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-24T09:09:39.347 INFO:teuthology.orchestra.run.smithi117.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 469 kB/s | 35 kB 00:00 2024-03-24T09:09:39.389 INFO:teuthology.orchestra.run.smithi117.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-24T09:09:39.481 INFO:teuthology.orchestra.run.smithi117.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-03-24T09:09:39.556 INFO:teuthology.orchestra.run.smithi117.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-03-24T09:09:39.590 INFO:teuthology.orchestra.run.smithi117.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-03-24T09:09:39.615 INFO:teuthology.orchestra.run.smithi117.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-03-24T09:09:39.649 INFO:teuthology.orchestra.run.smithi117.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 395 kB/s | 23 kB 00:00 2024-03-24T09:09:39.666 INFO:teuthology.orchestra.run.smithi117.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-24T09:09:39.699 INFO:teuthology.orchestra.run.smithi117.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-03-24T09:09:39.749 INFO:teuthology.orchestra.run.smithi117.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 924 kB/s | 46 kB 00:00 2024-03-24T09:09:39.825 INFO:teuthology.orchestra.run.smithi117.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-24T09:09:39.850 INFO:teuthology.orchestra.run.smithi117.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-03-24T09:09:39.875 INFO:teuthology.orchestra.run.smithi117.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-24T09:09:39.917 INFO:teuthology.orchestra.run.smithi117.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-24T09:09:39.934 INFO:teuthology.orchestra.run.smithi117.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 281 kB/s | 16 kB 00:00 2024-03-24T09:09:39.976 INFO:teuthology.orchestra.run.smithi117.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-24T09:09:39.987 INFO:teuthology.orchestra.run.smithi143.stdout:(152/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 3.6 MB/s | 3.1 MB 00:00 2024-03-24T09:09:39.993 INFO:teuthology.orchestra.run.smithi117.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-24T09:09:40.027 INFO:teuthology.orchestra.run.smithi117.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-03-24T09:09:40.060 INFO:teuthology.orchestra.run.smithi117.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-24T09:09:40.077 INFO:teuthology.orchestra.run.smithi117.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-24T09:09:40.111 INFO:teuthology.orchestra.run.smithi117.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-03-24T09:09:40.128 INFO:teuthology.orchestra.run.smithi117.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-24T09:09:40.178 INFO:teuthology.orchestra.run.smithi117.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-24T09:09:40.195 INFO:teuthology.orchestra.run.smithi143.stdout:(153/174): librados2-19.0.0-1855.g21b3bd67.el9. 3.1 MB/s | 3.4 MB 00:01 2024-03-24T09:09:40.203 INFO:teuthology.orchestra.run.smithi117.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-24T09:09:40.254 INFO:teuthology.orchestra.run.smithi117.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-03-24T09:09:40.279 INFO:teuthology.orchestra.run.smithi117.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-24T09:09:40.320 INFO:teuthology.orchestra.run.smithi117.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-24T09:09:40.454 INFO:teuthology.orchestra.run.smithi117.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-24T09:09:40.812 INFO:teuthology.orchestra.run.smithi143.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 68 kB/s | 42 kB 00:00 2024-03-24T09:09:41.421 INFO:teuthology.orchestra.run.smithi117.stdout:(150/174): ceph-test-19.0.0-1855.g21b3bd67.el9. 4.2 MB/s | 47 MB 00:11 2024-03-24T09:09:41.480 INFO:teuthology.orchestra.run.smithi117.stdout:(151/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 3.1 MB/s | 3.1 MB 00:01 2024-03-24T09:09:41.505 INFO:teuthology.orchestra.run.smithi117.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.8 MB/s | 321 kB 00:00 2024-03-24T09:09:41.556 INFO:teuthology.orchestra.run.smithi117.stdout:(153/174): librados2-19.0.0-1855.g21b3bd67.el9. 2.7 MB/s | 3.4 MB 00:01 2024-03-24T09:09:41.673 INFO:teuthology.orchestra.run.smithi117.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 361 kB/s | 42 kB 00:00 2024-03-24T09:09:41.771 INFO:teuthology.orchestra.run.smithi143.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 259 kB/s | 248 kB 00:00 2024-03-24T09:09:41.823 INFO:teuthology.orchestra.run.smithi117.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.6 MB/s | 248 kB 00:00 2024-03-24T09:09:41.881 INFO:teuthology.orchestra.run.smithi117.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-24T09:09:41.923 INFO:teuthology.orchestra.run.smithi117.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.4 MB/s | 1.8 MB 00:00 2024-03-24T09:09:41.948 INFO:teuthology.orchestra.run.smithi117.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 546 kB/s | 36 kB 00:00 2024-03-24T09:09:41.982 INFO:teuthology.orchestra.run.smithi117.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-24T09:09:42.024 INFO:teuthology.orchestra.run.smithi117.stdout:(160/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.2 MB/s | 2.3 MB 00:00 2024-03-24T09:09:42.049 INFO:teuthology.orchestra.run.smithi117.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 344 kB/s | 23 kB 00:00 2024-03-24T09:09:42.074 INFO:teuthology.orchestra.run.smithi117.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.0 MB/s | 256 kB 00:00 2024-03-24T09:09:42.100 INFO:teuthology.orchestra.run.smithi117.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-24T09:09:42.125 INFO:teuthology.orchestra.run.smithi117.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-24T09:09:42.150 INFO:teuthology.orchestra.run.smithi117.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-24T09:09:42.175 INFO:teuthology.orchestra.run.smithi117.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 845 kB/s | 64 kB 00:00 2024-03-24T09:09:42.201 INFO:teuthology.orchestra.run.smithi117.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 380 kB/s | 29 kB 00:00 2024-03-24T09:09:42.318 INFO:teuthology.orchestra.run.smithi117.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.3 MB/s | 1.2 MB 00:00 2024-03-24T09:09:42.410 INFO:teuthology.orchestra.run.smithi117.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-03-24T09:09:42.460 INFO:teuthology.orchestra.run.smithi117.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 624 kB/s | 31 kB 00:00 2024-03-24T09:09:42.488 INFO:teuthology.orchestra.run.smithi143.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 100 kB/s | 71 kB 00:00 2024-03-24T09:09:42.544 INFO:teuthology.orchestra.run.smithi117.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.9 MB/s | 2.2 MB 00:00 2024-03-24T09:09:42.586 INFO:teuthology.orchestra.run.smithi117.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-24T09:09:42.627 INFO:teuthology.orchestra.run.smithi117.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.4 MB/s | 2.3 MB 00:00 2024-03-24T09:09:42.677 INFO:teuthology.orchestra.run.smithi117.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 557 kB/s | 74 kB 00:00 2024-03-24T09:09:42.681 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:09:42.682 INFO:teuthology.orchestra.run.smithi117.stdout:Total 10 MB/s | 224 MB 00:21 2024-03-24T09:09:42.704 INFO:teuthology.orchestra.run.smithi143.stdout:(157/174): NetworkManager-1.47.2-1.el9.x86_64.r 678 kB/s | 2.3 MB 00:03 2024-03-24T09:09:42.896 INFO:teuthology.orchestra.run.smithi143.stdout:(158/174): NetworkManager-libnm-1.47.2-1.el9.x8 645 kB/s | 1.8 MB 00:02 2024-03-24T09:09:43.105 INFO:teuthology.orchestra.run.smithi143.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 59 kB/s | 36 kB 00:00 2024-03-24T09:09:43.134 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-24T09:09:43.248 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-24T09:09:43.337 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T09:09:43.407 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-24T09:09:43.407 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-24T09:09:43.497 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T09:09:43.613 INFO:teuthology.orchestra.run.smithi143.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 212 kB/s | 192 kB 00:00 2024-03-24T09:09:43.645 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T09:09:43.657 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T09:09:43.680 INFO:teuthology.orchestra.run.smithi143.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-03-24T09:09:43.705 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T09:09:43.769 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T09:09:43.816 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T09:09:43.830 INFO:teuthology.orchestra.run.smithi143.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 274 kB/s | 256 kB 00:00 2024-03-24T09:09:43.841 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T09:09:43.913 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T09:09:44.240 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T09:09:44.300 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:09:44.339 INFO:teuthology.orchestra.run.smithi143.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 149 kB/s | 108 kB 00:00 2024-03-24T09:09:44.422 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:09:44.451 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:09:44.523 INFO:teuthology.orchestra.run.smithi143.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 184 kB/s | 155 kB 00:00 2024-03-24T09:09:44.534 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:09:44.569 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T09:09:44.612 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T09:09:44.623 INFO:teuthology.orchestra.run.smithi143.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 171 kB/s | 136 kB 00:00 2024-03-24T09:09:44.780 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T09:09:44.827 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T09:09:44.856 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:09:44.942 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:09:44.974 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T09:09:45.025 INFO:teuthology.orchestra.run.smithi143.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 93 kB/s | 64 kB 00:00 2024-03-24T09:09:45.038 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T09:09:45.095 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T09:09:45.125 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T09:09:45.141 INFO:teuthology.orchestra.run.smithi143.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 46 kB/s | 29 kB 00:00 2024-03-24T09:09:45.141 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T09:09:45.327 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T09:09:45.358 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T09:09:45.398 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T09:09:45.407 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-24T09:09:45.407 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-24T09:09:45.656 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T09:09:45.908 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T09:09:45.969 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T09:09:46.095 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:09:46.145 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:09:46.351 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:09:46.470 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T09:09:46.562 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T09:09:46.591 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T09:09:46.618 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T09:09:46.662 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T09:09:46.715 INFO:teuthology.orchestra.run.smithi143.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 592 kB/s | 1.2 MB 00:02 2024-03-24T09:09:46.736 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T09:09:46.790 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T09:09:46.836 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T09:09:46.908 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T09:09:46.944 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T09:09:46.987 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T09:09:47.020 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T09:09:47.050 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T09:09:47.119 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T09:09:47.155 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T09:09:47.198 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T09:09:47.227 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T09:09:47.321 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T09:09:47.366 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T09:09:47.440 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T09:09:47.581 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T09:09:47.635 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T09:09:47.899 INFO:teuthology.orchestra.run.smithi143.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 393 kB/s | 465 kB 00:01 2024-03-24T09:09:48.154 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T09:09:48.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T09:09:48.316 INFO:teuthology.orchestra.run.smithi143.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 671 kB/s | 2.2 MB 00:03 2024-03-24T09:09:48.325 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T09:09:48.367 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T09:09:48.376 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T09:09:48.408 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T09:09:48.448 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T09:09:48.485 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T09:09:48.516 INFO:teuthology.orchestra.run.smithi143.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 50 kB/s | 31 kB 00:00 2024-03-24T09:09:48.516 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T09:09:48.616 INFO:teuthology.orchestra.run.smithi143.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 745 kB/s | 74 kB 00:00 2024-03-24T09:09:48.658 INFO:teuthology.orchestra.run.smithi143.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 665 kB/s | 2.3 MB 00:03 2024-03-24T09:09:49.005 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T09:09:49.182 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T09:09:49.284 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T09:09:49.337 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T09:09:49.468 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T09:09:49.531 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T09:09:49.586 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T09:09:49.658 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T09:09:49.690 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T09:09:49.727 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T09:09:49.773 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T09:09:49.818 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T09:09:49.860 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T09:09:49.903 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T09:09:50.034 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T09:09:50.075 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T09:09:50.105 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T09:09:50.146 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T09:09:50.242 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T09:09:50.250 INFO:teuthology.orchestra.run.smithi143.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 568 kB/s | 1.1 MB 00:01 2024-03-24T09:09:50.255 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:09:50.255 INFO:teuthology.orchestra.run.smithi143.stdout:Total 8.4 MB/s | 224 MB 00:26 2024-03-24T09:09:50.288 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T09:09:50.329 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T09:09:50.354 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T09:09:50.379 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T09:09:50.461 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T09:09:50.586 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T09:09:50.685 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T09:09:50.719 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-03-24T09:09:50.729 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T09:09:50.777 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T09:09:50.853 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T09:09:50.993 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-03-24T09:09:50.993 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-03-24T09:09:51.039 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T09:09:51.082 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T09:09:51.212 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T09:09:51.261 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T09:09:51.304 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T09:09:51.335 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T09:09:51.363 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T09:09:51.397 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T09:09:51.436 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T09:09:51.468 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T09:09:51.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T09:09:51.574 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T09:09:51.608 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T09:09:51.641 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T09:09:51.691 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T09:09:51.734 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T09:09:51.822 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T09:09:51.913 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T09:09:51.956 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T09:09:51.989 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T09:09:52.036 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T09:09:52.132 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T09:09:52.190 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T09:09:52.539 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-24T09:09:52.623 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T09:09:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T09:09:52.720 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T09:09:52.730 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T09:09:52.778 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T09:09:52.833 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T09:09:52.865 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T09:09:52.899 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T09:09:52.908 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T09:09:52.953 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T09:09:52.961 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T09:09:52.995 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T09:09:52.999 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-03-24T09:09:52.999 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-03-24T09:09:53.043 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T09:09:53.122 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T09:09:53.163 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T09:09:53.286 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T09:09:53.344 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:09:53.349 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T09:09:53.385 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T09:09:53.448 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:09:53.464 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T09:09:53.479 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:09:53.552 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:09:53.591 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T09:09:53.637 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T09:09:53.818 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T09:09:53.864 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T09:09:53.892 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:09:53.978 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:09:54.001 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T09:09:54.058 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T09:09:54.101 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T09:09:54.105 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T09:09:54.128 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T09:09:54.143 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T09:09:54.215 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T09:09:54.303 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T09:09:54.334 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T09:09:54.366 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T09:09:54.557 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T09:09:54.695 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T09:09:54.739 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T09:09:54.846 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:09:54.898 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:09:55.117 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:09:55.224 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T09:09:55.323 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T09:09:55.353 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T09:09:55.380 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T09:09:55.415 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T09:09:55.426 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T09:09:55.480 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T09:09:55.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T09:09:55.534 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T09:09:55.542 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T09:09:55.573 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T09:09:55.581 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T09:09:55.608 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T09:09:55.636 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T09:09:55.672 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T09:09:55.715 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T09:09:55.749 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T09:09:55.776 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T09:09:55.846 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T09:09:55.889 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T09:09:55.921 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T09:09:55.933 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T09:09:55.955 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T09:09:55.961 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T09:09:56.030 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T09:09:56.045 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T09:09:56.076 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T09:09:56.077 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T09:09:56.113 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T09:09:56.150 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T09:09:56.284 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T09:09:56.345 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T09:09:56.572 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T09:09:56.600 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T09:09:56.670 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T09:09:56.693 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T09:09:56.863 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T09:09:56.940 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T09:09:57.029 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T09:09:57.077 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T09:09:57.085 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T09:09:57.125 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T09:09:57.167 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T09:09:57.204 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T09:09:57.251 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T09:09:57.563 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T09:09:57.764 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T09:09:57.886 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T09:09:57.981 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T09:09:58.110 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T09:09:58.158 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T09:09:58.213 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T09:09:58.286 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T09:09:58.318 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T09:09:58.354 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T09:09:58.400 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T09:09:58.436 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T09:09:58.479 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T09:09:58.522 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T09:09:58.654 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T09:09:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T09:09:58.732 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T09:09:58.774 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T09:09:58.867 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T09:09:58.907 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T09:09:58.939 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T09:09:58.965 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T09:09:58.980 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:09:58.989 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T09:09:59.015 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:09:59.067 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T09:09:59.198 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T09:09:59.305 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T09:09:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T09:09:59.404 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T09:09:59.468 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T09:09:59.554 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:09:59.593 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T09:09:59.652 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T09:09:59.687 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T09:09:59.687 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T09:09:59.688 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T09:09:59.688 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:09:59.701 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T09:09:59.727 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T09:09:59.829 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T09:09:59.867 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T09:09:59.898 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T09:09:59.929 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T09:09:59.966 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T09:10:00.008 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T09:10:00.048 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T09:10:00.079 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T09:10:00.109 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T09:10:00.184 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T09:10:00.203 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-03-24T09:10:00.219 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T09:10:00.253 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T09:10:00.286 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T09:10:00.287 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T09:10:00.320 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T09:10:00.321 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T09:10:00.359 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T09:10:00.369 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T09:10:00.409 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T09:10:00.416 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T09:10:00.469 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T09:10:00.491 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T09:10:00.502 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T09:10:00.534 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T09:10:00.536 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T09:10:00.566 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T09:10:00.583 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T09:10:00.613 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T09:10:00.710 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T09:10:00.767 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T09:10:00.914 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T09:10:00.987 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:10:01.091 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:10:01.121 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:10:01.212 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:10:01.248 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T09:10:01.289 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T09:10:01.469 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T09:10:01.493 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T09:10:01.523 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T09:10:01.539 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T09:10:01.551 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:10:01.573 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T09:10:01.622 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T09:10:01.653 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:10:01.677 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T09:10:01.708 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T09:10:01.742 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-24T09:10:01.750 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T09:10:01.781 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T09:10:01.803 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T09:10:01.830 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T09:10:01.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T09:10:01.940 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T09:10:01.985 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T09:10:02.017 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T09:10:02.025 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T09:10:02.064 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T09:10:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T09:10:02.375 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T09:10:02.438 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T09:10:02.548 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:10:02.611 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:10:02.667 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T09:10:02.778 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T09:10:02.849 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T09:10:02.957 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T09:10:03.052 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T09:10:03.082 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T09:10:03.110 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T09:10:03.153 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T09:10:03.227 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T09:10:03.280 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T09:10:03.319 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T09:10:03.383 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T09:10:03.418 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T09:10:03.470 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T09:10:03.503 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T09:10:03.530 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T09:10:03.599 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T09:10:03.635 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-24T09:10:03.678 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-24T09:10:03.707 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-24T09:10:03.776 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T09:10:03.821 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T09:10:03.894 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T09:10:04.008 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T09:10:04.028 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T09:10:04.082 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T09:10:04.098 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T09:10:04.129 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T09:10:04.161 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T09:10:04.188 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T09:10:04.497 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T09:10:04.526 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T09:10:04.605 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T09:10:04.617 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T09:10:04.648 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T09:10:04.684 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T09:10:04.687 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T09:10:04.774 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T09:10:04.833 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T09:10:04.841 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T09:10:04.881 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T09:10:04.930 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T09:10:04.968 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-24T09:10:04.998 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T09:10:05.136 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T09:10:05.321 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T09:10:05.325 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T09:10:05.434 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T09:10:05.456 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T09:10:05.484 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T09:10:05.551 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T09:10:05.595 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T09:10:05.742 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T09:10:05.790 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T09:10:05.845 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T09:10:05.924 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T09:10:05.957 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T09:10:05.994 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T09:10:06.041 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T09:10:06.077 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T09:10:06.120 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T09:10:06.162 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T09:10:06.297 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T09:10:06.343 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T09:10:06.373 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T09:10:06.415 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T09:10:06.508 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T09:10:06.539 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T09:10:06.580 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T09:10:06.622 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T09:10:06.647 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T09:10:06.728 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T09:10:06.855 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T09:10:06.958 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T09:10:06.997 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T09:10:07.046 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T09:10:07.109 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T09:10:07.302 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T09:10:07.352 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T09:10:07.491 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T09:10:07.533 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T09:10:07.565 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T09:10:07.596 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T09:10:07.624 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T09:10:07.658 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T09:10:07.671 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:10:07.697 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T09:10:07.703 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:10:07.729 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T09:10:07.759 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T09:10:07.836 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T09:10:07.878 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T09:10:07.911 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T09:10:07.952 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T09:10:07.987 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T09:10:08.077 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T09:10:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T09:10:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T09:10:08.242 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:10:08.258 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T09:10:08.290 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T09:10:08.305 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T09:10:08.388 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T09:10:08.388 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T09:10:08.389 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T09:10:08.389 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:08.403 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T09:10:08.432 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T09:10:08.451 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T09:10:09.178 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T09:10:09.224 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T09:10:09.254 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T09:10:09.302 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T09:10:09.391 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T09:10:09.430 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T09:10:09.615 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T09:10:09.653 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T09:10:09.730 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T09:10:10.380 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T09:10:10.492 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T09:10:11.123 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-24T09:10:11.124 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:11.282 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T09:10:11.297 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T09:10:11.298 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:11.298 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T09:10:11.298 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:11.712 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T09:10:11.796 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T09:10:11.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T09:10:11.858 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T09:10:11.894 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T09:10:12.205 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T09:10:12.233 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T09:10:12.315 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-24T09:10:12.346 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-24T09:10:12.382 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T09:10:12.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T09:10:12.836 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T09:10:12.836 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T09:10:12.837 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:12.837 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T09:10:12.837 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:10:12.837 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:10:12.837 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:12.869 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T09:10:12.940 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-24T09:10:12.962 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-24T09:10:13.061 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T09:10:13.083 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T09:10:13.083 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:13.083 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T09:10:13.083 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:10:13.083 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:10:13.084 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:13.528 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T09:10:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T09:10:13.551 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:13.551 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T09:10:13.551 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:10:13.551 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:10:13.551 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:13.638 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T09:10:14.089 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T09:10:14.120 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T09:10:14.151 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T09:10:14.211 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T09:10:14.234 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T09:10:14.466 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T09:10:14.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T09:10:15.115 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:10:15.151 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:10:15.466 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T09:10:15.494 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T09:10:15.594 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T09:10:15.686 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-24T09:10:15.721 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T09:10:15.756 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T09:10:15.788 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T09:10:15.796 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-24T09:10:15.797 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T09:10:15.797 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T09:10:15.797 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T09:10:15.811 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:15.811 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T09:10:15.811 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:10:15.811 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:10:15.811 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:15.839 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T09:10:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T09:10:15.866 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T09:10:15.922 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T09:10:16.778 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T09:10:16.807 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T09:10:16.830 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T09:10:16.830 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:16.830 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T09:10:16.830 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:10:16.831 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:10:16.831 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:16.875 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T09:10:16.896 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T09:10:16.897 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:16.897 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T09:10:16.897 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:16.941 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:16.964 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:16.964 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:16.964 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T09:10:16.964 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T09:10:16.964 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T09:10:16.964 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:17.276 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T09:10:17.298 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T09:10:17.298 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:17.299 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T09:10:17.299 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:10:17.299 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:10:17.299 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-24T09:10:20.266 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-24T09:10:20.267 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:20.435 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T09:10:20.450 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T09:10:20.450 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:20.450 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T09:10:20.451 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:21.747 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T09:10:21.797 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T09:10:21.826 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T09:10:21.854 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T09:10:21.965 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T09:10:21.990 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T09:10:21.990 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:21.990 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T09:10:21.990 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:10:21.990 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:10:21.991 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:22.035 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T09:10:22.144 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T09:10:22.184 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T09:10:22.215 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T09:10:22.237 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T09:10:22.237 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:22.237 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T09:10:22.237 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:10:22.237 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:10:22.237 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:22.238 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T09:10:22.302 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T09:10:22.346 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T09:10:22.386 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T09:10:22.426 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T09:10:22.465 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T09:10:22.532 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T09:10:22.591 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T09:10:22.629 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T09:10:22.674 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T09:10:22.697 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T09:10:22.697 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:22.697 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T09:10:22.697 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:10:22.697 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:10:22.697 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:22.792 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T09:10:23.210 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T09:10:23.241 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T09:10:23.258 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T09:10:23.304 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:23.325 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:23.333 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T09:10:23.355 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T09:10:23.584 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T09:10:23.606 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T09:10:24.144 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T09:10:24.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T09:10:24.265 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T09:10:24.365 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T09:10:24.403 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T09:10:24.428 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T09:10:24.453 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T09:10:24.479 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T09:10:24.480 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T09:10:24.564 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T09:10:24.592 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T09:10:24.598 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T09:10:24.622 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T09:10:24.622 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T09:10:24.698 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T09:10:24.707 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T09:10:24.742 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T09:10:24.783 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T09:10:24.808 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T09:10:24.833 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T09:10:24.860 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T09:10:24.867 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T09:10:24.900 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T09:10:24.914 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T09:10:24.923 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T09:10:24.923 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:24.923 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T09:10:24.923 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:10:24.924 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:10:24.924 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:24.954 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T09:10:24.976 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T09:10:24.980 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T09:10:24.987 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T09:10:25.005 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T09:10:25.030 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T09:10:25.038 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T09:10:25.056 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T09:10:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:25.062 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:25.250 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:25.250 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T09:10:25.333 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T09:10:25.366 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T09:10:25.398 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T09:10:25.425 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T09:10:25.451 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T09:10:25.516 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T09:10:25.909 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T09:10:25.936 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T09:10:25.959 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T09:10:25.960 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:25.960 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T09:10:25.960 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:10:25.960 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:10:25.960 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:26.004 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T09:10:26.025 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T09:10:26.026 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:26.026 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T09:10:26.026 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:26.071 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:26.093 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:26.093 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:26.093 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T09:10:26.093 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T09:10:26.094 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T09:10:26.094 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:26.397 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T09:10:26.419 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T09:10:26.419 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:26.419 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T09:10:26.419 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:10:26.419 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:10:26.419 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:27.852 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-24T09:10:27.852 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-03-24T09:10:27.852 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-03-24T09:10:27.852 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-03-24T09:10:27.852 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-03-24T09:10:27.853 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-03-24T09:10:27.853 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /root 2024-03-24T09:10:27.853 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-03-24T09:10:27.853 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:28.013 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T09:10:28.029 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-24T09:10:28.030 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:28.030 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T09:10:28.030 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:29.533 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T09:10:29.559 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-24T09:10:29.559 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:29.559 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T09:10:29.559 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:10:29.559 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T09:10:29.559 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:29.784 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T09:10:29.806 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-24T09:10:29.806 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:29.806 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T09:10:29.807 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:10:29.807 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T09:10:29.807 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:30.243 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T09:10:30.264 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:10:30.265 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T09:10:30.266 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T09:10:30.267 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T09:10:30.267 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-24T09:10:30.268 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:30.268 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T09:10:30.268 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:10:30.268 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T09:10:30.268 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:30.288 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T09:10:30.288 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T09:10:30.289 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T09:10:30.290 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T09:10:30.291 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T09:10:30.292 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T09:10:30.293 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T09:10:30.294 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T09:10:30.295 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T09:10:30.296 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T09:10:30.297 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T09:10:30.298 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T09:10:30.299 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T09:10:30.300 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T09:10:30.301 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T09:10:30.302 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T09:10:30.303 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T09:10:30.304 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T09:10:30.361 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T09:10:30.796 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T09:10:30.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-24T09:10:30.858 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-24T09:10:30.859 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T09:10:30.900 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T09:10:30.918 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-24T09:10:30.931 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T09:10:30.942 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T09:10:30.958 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T09:10:31.137 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T09:10:31.173 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-24T09:10:31.201 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T09:10:31.257 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T09:10:31.298 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T09:10:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T09:10:31.423 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T09:10:31.457 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T09:10:31.490 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T09:10:31.522 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T09:10:31.552 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T09:10:31.620 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T09:10:31.674 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T09:10:31.709 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T09:10:32.160 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-24T09:10:32.185 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T09:10:32.285 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-24T09:10:32.331 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:32.367 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:32.385 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:32.447 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-24T09:10:32.478 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T09:10:32.501 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-24T09:10:32.501 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:32.501 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T09:10:32.501 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:10:32.501 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T09:10:32.501 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:32.546 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T09:10:32.556 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-24T09:10:32.609 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-24T09:10:33.207 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T09:10:33.219 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T09:10:33.219 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:33.219 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T09:10:33.220 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.221 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T09:10:33.222 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.223 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:33.225 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T09:10:33.226 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T09:10:33.227 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T09:10:33.228 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T09:10:33.229 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T09:10:33.230 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T09:10:33.231 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T09:10:33.232 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T09:10:33.233 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T09:10:33.234 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T09:10:33.235 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T09:10:33.236 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T09:10:33.237 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T09:10:33.238 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T09:10:33.238 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:10:33.238 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-24T09:10:33.295 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T09:10:33.337 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T09:10:33.427 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T09:10:33.457 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T09:10:33.475 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T09:10:33.482 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T09:10:33.506 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T09:10:33.508 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T09:10:33.529 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-24T09:10:33.529 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:33.529 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T09:10:33.529 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:10:33.530 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T09:10:33.530 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:33.534 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T09:10:33.534 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T09:10:33.574 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T09:10:33.595 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-24T09:10:33.596 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:33.596 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T09:10:33.596 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:33.630 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T09:10:33.641 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:33.663 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:33.663 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:33.663 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T09:10:33.663 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T09:10:33.663 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T09:10:33.664 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:33.668 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T09:10:33.668 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T09:10:33.722 DEBUG:teuthology.parallel:result is None 2024-03-24T09:10:33.753 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T09:10:33.779 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T09:10:33.812 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T09:10:33.837 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T09:10:33.862 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T09:10:33.896 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T09:10:33.943 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T09:10:33.958 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T09:10:33.976 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T09:10:33.980 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-24T09:10:33.980 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T09:10:33.981 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T09:10:33.981 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:10:33.981 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T09:10:33.981 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:34.001 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T09:10:34.026 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T09:10:34.051 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T09:10:34.077 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T09:10:34.083 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:34.083 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T09:10:34.337 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T09:10:34.362 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T09:10:34.394 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T09:10:34.422 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T09:10:34.447 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T09:10:34.512 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T09:10:38.388 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T09:10:38.429 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-24T09:10:38.460 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-24T09:10:38.488 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T09:10:38.666 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-24T09:10:38.787 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T09:10:38.827 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-24T09:10:38.894 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T09:10:38.944 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T09:10:38.995 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T09:10:39.020 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-24T09:10:39.061 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T09:10:39.099 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T09:10:39.156 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T09:10:39.213 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T09:10:39.250 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T09:10:39.250 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T09:10:39.250 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T09:10:39.251 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T09:10:39.252 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T09:10:39.253 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T09:10:39.253 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T09:10:39.253 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T09:10:39.269 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T09:10:39.270 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T09:10:39.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T09:10:39.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T09:10:39.273 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T09:10:39.274 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T09:10:39.275 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T09:10:39.276 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:39.277 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T09:10:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T09:10:39.279 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T09:10:39.280 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T09:10:39.281 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T09:10:39.834 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:39.879 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:39.898 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T09:10:40.710 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-24T09:10:40.793 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T09:10:40.834 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T09:10:40.925 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T09:10:40.955 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T09:10:40.980 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T09:10:41.006 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T09:10:41.032 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T09:10:41.033 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T09:10:41.111 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T09:10:41.150 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T09:10:41.150 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T09:10:41.243 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T09:10:41.278 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T09:10:41.311 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T09:10:41.336 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T09:10:41.361 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T09:10:41.395 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T09:10:41.442 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T09:10:41.483 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T09:10:41.508 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T09:10:41.534 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T09:10:41.559 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T09:10:41.584 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T09:10:41.591 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:41.591 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:41.770 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T09:10:41.770 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T09:10:41.837 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T09:10:41.886 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T09:10:41.918 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T09:10:41.946 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T09:10:41.971 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T09:10:42.036 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:42.287 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.288 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T09:10:42.289 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.290 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.291 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.292 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-24T09:10:42.293 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T09:10:42.294 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T09:10:42.295 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T09:10:42.296 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.297 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T09:10:42.298 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T09:10:42.299 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T09:10:42.300 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T09:10:42.301 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T09:10:42.302 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T09:10:42.303 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:42.304 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T09:10:42.305 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T09:10:42.305 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T09:10:42.305 INFO:teuthology.orchestra.run.smithi117.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T09:10:42.305 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T09:10:42.305 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T09:10:42.305 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T09:10:42.306 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-24T09:10:42.306 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-24T09:10:42.759 DEBUG:teuthology.parallel:result is None 2024-03-24T09:10:46.892 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T09:10:46.892 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-24T09:10:46.892 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-24T09:10:46.892 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-24T09:10:46.893 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-24T09:10:46.894 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-24T09:10:46.895 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T09:10:46.896 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T09:10:46.897 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T09:10:46.898 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T09:10:46.899 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T09:10:46.900 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T09:10:46.901 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T09:10:46.902 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T09:10:46.903 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T09:10:46.904 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T09:10:46.905 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-24T09:10:46.906 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T09:10:46.907 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T09:10:46.908 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T09:10:46.909 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T09:10:46.910 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T09:10:46.911 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T09:10:49.975 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T09:10:49.975 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:49.975 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T09:10:49.976 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.977 INFO:teuthology.orchestra.run.smithi143.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T09:10:49.978 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.979 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.980 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.981 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T09:10:49.982 INFO:teuthology.orchestra.run.smithi143.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: jq-1.6-16.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T09:10:49.983 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T09:10:49.984 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T09:10:49.985 INFO:teuthology.orchestra.run.smithi143.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T09:10:49.986 INFO:teuthology.orchestra.run.smithi143.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T09:10:49.987 INFO:teuthology.orchestra.run.smithi143.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T09:10:49.988 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T09:10:49.989 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T09:10:49.990 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T09:10:49.991 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T09:10:49.992 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T09:10:49.993 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T09:10:49.994 INFO:teuthology.orchestra.run.smithi143.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.995 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T09:10:49.996 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T09:10:49.997 INFO:teuthology.orchestra.run.smithi143.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T09:10:49.997 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:10:49.997 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-03-24T09:10:50.450 DEBUG:teuthology.parallel:result is None 2024-03-24T09:10:50.450 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:10:50.674 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T09:10:50.700 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T09:10:50.701 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T09:10:50.701 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T09:10:50.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:10:50.918 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T09:10:50.944 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T09:10:50.944 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T09:10:50.945 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T09:10:50.946 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:10:51.160 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T09:10:51.185 INFO:teuthology.orchestra.run.smithi143.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T09:10:51.185 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T09:10:51.185 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T09:10:51.186 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T09:10:51.186 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:10:51.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T09:10:51.221 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:10:51.221 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T09:10:51.256 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:10:51.256 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T09:10:51.289 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T09:10:51.289 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:10:51.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T09:10:51.320 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T09:10:51.390 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:10:51.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T09:10:51.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T09:10:51.492 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:10:51.492 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T09:10:51.523 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T09:10:51.592 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T09:10:51.592 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:10:51.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T09:10:51.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T09:10:51.691 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:10:51.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T09:10:51.722 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T09:10:51.793 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:10:51.793 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T09:10:51.823 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T09:10:51.891 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T09:10:51.892 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:10:51.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T09:10:51.919 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T09:10:51.985 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:10:51.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T09:10:52.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T09:10:52.087 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:10:52.087 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T09:10:52.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T09:10:52.187 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T09:10:52.279 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-24T09:10:52.279 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:10:52.279 INFO:tasks.cephadm:Cluster fsid is 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:10:52.280 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T09:10:52.280 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.116', 'mon.b': '172.21.15.117', 'mon.c': '172.21.15.143'} 2024-03-24T09:10:52.280 INFO:tasks.cephadm:First mon is mon.a on smithi116 2024-03-24T09:10:52.280 INFO:tasks.cephadm:First mgr is x 2024-03-24T09:10:52.280 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T09:10:52.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-24T09:10:52.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo hostname $(hostname -s) 2024-03-24T09:10:52.339 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-03-24T09:10:52.371 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T09:10:52.372 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:10:52.511 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77900/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1855-g21b3bd67', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1855.g21b3bd67'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-23 04:48:23.862772', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240323.003144-squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T09:10:52.749 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-batrick-testing-20240323.003144-squid, sha1 21b3bd672163092b4e542f772797fcba02d3d4de from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:10:52.750 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm 2024-03-24T09:10:52.750 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:53.357 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 09:10 /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:53.358 DEBUG:teuthology.orchestra.run.smithi117:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:53.830 INFO:teuthology.orchestra.run.smithi117.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 09:10 /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:53.830 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:54.666 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r--. 1 ubuntu ubuntu 766828 Mar 24 09:10 /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:54.666 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:54.685 DEBUG:teuthology.orchestra.run.smithi117:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:54.704 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T09:10:54.738 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de on all hosts... 2024-03-24T09:10:54.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T09:10:54.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T09:10:54.746 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de pull 2024-03-24T09:10:55.558 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T09:10:55.621 INFO:teuthology.orchestra.run.smithi117.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T09:10:55.690 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T09:11:21.388 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T09:11:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T09:11:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T09:11:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "repo_digests": [ 2024-03-24T09:11:21.388 INFO:teuthology.orchestra.run.smithi143.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T09:11:21.389 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-03-24T09:11:21.389 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T09:11:28.265 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:11:28.265 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T09:11:28.265 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T09:11:28.265 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-24T09:11:28.266 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T09:11:28.266 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:11:28.266 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:11:29.244 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T09:11:29.245 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T09:11:29.245 INFO:teuthology.orchestra.run.smithi117.stdout: "image_id": "553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12", 2024-03-24T09:11:29.245 INFO:teuthology.orchestra.run.smithi117.stdout: "repo_digests": [ 2024-03-24T09:11:29.245 INFO:teuthology.orchestra.run.smithi117.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919" 2024-03-24T09:11:29.245 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-24T09:11:29.245 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T09:11:29.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-24T09:11:29.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2024-03-24T09:11:29.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-03-24T09:11:29.372 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-24T09:11:29.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /etc/ceph 2024-03-24T09:11:29.429 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-03-24T09:11:29.461 INFO:tasks.cephadm:Writing seed config... 2024-03-24T09:11:29.462 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T09:11:29.462 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T09:11:29.462 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T09:11:29.462 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T09:11:29.462 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T09:11:29.462 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T09:11:29.463 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-24T09:11:29.464 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T09:11:29.465 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-24T09:11:29.466 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:11:29.466 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T09:11:29.484 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 69c76602-e9be-11ee-95cd-87774f69a715 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T09:11:29.484 DEBUG:teuthology.orchestra.run.smithi116:mon.a> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T09:11:29.527 DEBUG:teuthology.orchestra.run.smithi116:mgr.x> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T09:11:29.570 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T09:11:29.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de -v bootstrap --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.116 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:11:29.765 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-24T09:11:29.765 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de', '-v', 'bootstrap', '--fsid', '69c76602-e9be-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.116', '--skip-admin-label'] 2024-03-24T09:11:29.790 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T09:11:29.791 INFO:teuthology.orchestra.run.smithi116.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T09:11:29.791 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying podman|docker is present... 2024-03-24T09:11:29.814 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T09:11:29.814 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying lvm2 is present... 2024-03-24T09:11:29.814 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying time synchronization is in place... 2024-03-24T09:11:29.821 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T09:11:29.822 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T09:11:29.827 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T09:11:29.827 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-24T09:11:29.833 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-24T09:11:29.839 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-24T09:11:29.839 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-24T09:11:29.839 INFO:teuthology.orchestra.run.smithi116.stdout:Repeating the final host check... 2024-03-24T09:11:29.862 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T09:11:29.862 INFO:teuthology.orchestra.run.smithi116.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T09:11:29.862 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl is present 2024-03-24T09:11:29.862 INFO:teuthology.orchestra.run.smithi116.stdout:lvcreate is present 2024-03-24T09:11:29.868 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T09:11:29.868 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T09:11:29.873 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T09:11:29.874 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-24T09:11:29.880 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:Host looks OK 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:Cluster fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:Acquiring lock 140593694512752 on /run/cephadm/69c76602-e9be-11ee-95cd-87774f69a715.lock 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:Lock 140593694512752 acquired on /run/cephadm/69c76602-e9be-11ee-95cd-87774f69a715.lock 2024-03-24T09:11:29.886 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 3300 ... 2024-03-24T09:11:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 6789 ... 2024-03-24T09:11:29.887 INFO:teuthology.orchestra.run.smithi116.stdout:Base mon IP(s) is [172.21.15.116:3300, 172.21.15.116:6789], mon addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-24T09:11:29.890 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.116 metric 100 2024-03-24T09:11:29.890 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 metric 100 2024-03-24T09:11:29.892 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T09:11:29.892 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T09:11:29.893 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1755sec hoplimit 64 pref medium 2024-03-24T09:11:29.895 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T09:11:29.895 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T09:11:29.895 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T09:11:29.895 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T09:11:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-24T09:11:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T09:11:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-24T09:11:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-24T09:11:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T09:11:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T09:11:29.897 INFO:teuthology.orchestra.run.smithi116.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de... 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:30dc77a9f705c02d26e1a1f9e2bfdf9d08cd4086b08b221e8a27d0dd2909924d 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying config sha256:553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 2024-03-24T09:11:30.392 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T09:11:31.347 INFO:teuthology.orchestra.run.smithi116.stdout:ceph: stdout ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T09:11:31.347 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph version: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev) 2024-03-24T09:11:31.347 INFO:teuthology.orchestra.run.smithi116.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T09:11:32.405 INFO:teuthology.orchestra.run.smithi116.stdout:stat: stdout 167 167 2024-03-24T09:11:32.406 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial keys... 2024-03-24T09:11:33.326 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQBE7v9l2X63KxAAL7BcHeH7eMTtWzeCW+dIyA== 2024-03-24T09:11:34.199 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQBF7v9l8SbvJBAARe8CB+Ytg/8uwJS55EhODg== 2024-03-24T09:11:35.288 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQBG7v9l6Rl8IhAAa6bOkJN0ePlP32BTLIUOkg== 2024-03-24T09:11:35.289 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial monmap... 2024-03-24T09:11:36.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T09:11:36.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T09:11:36.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:36.245 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T09:11:36.246 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool for a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T09:11:36.246 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-03-24T09:11:36.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: set fsid to 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:36.246 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T09:11:36.246 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:11:36.246 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mon... 2024-03-24T09:11:37.150 INFO:teuthology.orchestra.run.smithi116.stdout:create mon.a on 2024-03-24T09:11:37.363 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T09:11:37.511 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T09:11:37.713 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-69c76602-e9be-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-69c76602-e9be-11ee-95cd-87774f69a715.target. 2024-03-24T09:11:37.713 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-69c76602-e9be-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-69c76602-e9be-11ee-95cd-87774f69a715.target. 2024-03-24T09:11:37.938 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a 2024-03-24T09:11:37.938 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T09:11:38.104 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-69c76602-e9be-11ee-95cd-87774f69a715.target.wants/ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-69c76602-e9be-11ee-95cd-87774f69a715@.service. 2024-03-24T09:11:38.657 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:11:38.657 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-24T09:11:38.664 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:11:38.664 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-24T09:11:38.664 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-24T09:11:38.664 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T09:11:38.664 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon to start... 2024-03-24T09:11:38.664 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon... 2024-03-24T09:11:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T09:11:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout id: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T09:11:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:39.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout services: 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.594196s) 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout data: 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.143+0000 7f6a1d82c700 1 Processor -- start 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.143+0000 7f6a1d82c700 1 -- start start 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.143+0000 7f6a1d82c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a18106250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.143+0000 7f6a1d82c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a18106820 con 0x7f6a18105e30 2024-03-24T09:11:39.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a16ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a18106250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a16ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a18106250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45374/0 (socket says 172.21.15.116:45374) 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a16ffd700 1 -- 172.21.15.116:0/619304642 learned_addr learned my addr 172.21.15.116:0/619304642 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a16ffd700 1 -- 172.21.15.116:0/619304642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a18108dc0 con 0x7f6a18105e30 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a16ffd700 1 --2- 172.21.15.116:0/619304642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a18106250 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a000098a0 tx=0x7f6a00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f28bdfc1b32696 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a15ffb700 1 -- 172.21.15.116:0/619304642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a00004160 con 0x7f6a18105e30 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a15ffb700 1 -- 172.21.15.116:0/619304642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a000042c0 con 0x7f6a18105e30 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.144+0000 7f6a15ffb700 1 -- 172.21.15.116:0/619304642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a000045a0 con 0x7f6a18105e30 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 -- 172.21.15.116:0/619304642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 msgr2=0x7f6a18106250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 --2- 172.21.15.116:0/619304642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a18106250 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a000098a0 tx=0x7f6a00009bb0 comp rx=0 tx=0).stop 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 -- 172.21.15.116:0/619304642 shutdown_connections 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 --2- 172.21.15.116:0/619304642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a18106250 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 -- 172.21.15.116:0/619304642 >> 172.21.15.116:0/619304642 conn(0x7f6a181017c0 msgr2=0x7f6a18103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 -- 172.21.15.116:0/619304642 shutdown_connections 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.145+0000 7f6a1d82c700 1 -- 172.21.15.116:0/619304642 wait complete. 2024-03-24T09:11:39.744 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a1d82c700 1 Processor -- start 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a1d82c700 1 -- start start 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a1d82c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a1819afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a1d82c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1819b510 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a16ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a1819afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a16ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a1819afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45376/0 (socket says 172.21.15.116:45376) 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a16ffd700 1 -- 172.21.15.116:0/1948197371 learned_addr learned my addr 172.21.15.116:0/1948197371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a16ffd700 1 -- 172.21.15.116:0/1948197371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a00009550 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.146+0000 7f6a16ffd700 1 --2- 172.21.15.116:0/1948197371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a1819afd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a00004000 tx=0x7f6a00004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a00011630 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a00011790 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1819b710 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a00011a30 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a1819bbb0 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a00022070 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.147+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a0001c450 con 0x7f6a18105e30 2024-03-24T09:11:39.745 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.148+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a18194cf0 con 0x7f6a18105e30 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.150+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6a00011b90 con 0x7f6a18105e30 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.220+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f6a181093a0 con 0x7f6a18105e30 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.220+0000 7f6a1c82a700 1 -- 172.21.15.116:0/1948197371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f6a00011d30 con 0x7f6a18105e30 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 msgr2=0x7f6a1819afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 --2- 172.21.15.116:0/1948197371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a1819afd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a00004000 tx=0x7f6a00004750 comp rx=0 tx=0).stop 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 shutdown_connections 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 --2- 172.21.15.116:0/1948197371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a18105e30 0x7f6a1819afd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 >> 172.21.15.116:0/1948197371 conn(0x7f6a181017c0 msgr2=0x7f6a18191e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 shutdown_connections 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:39.221+0000 7f6a1d82c700 1 -- 172.21.15.116:0/1948197371 wait complete. 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:mon is available 2024-03-24T09:11:39.746 INFO:teuthology.orchestra.run.smithi116.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T09:11:40.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:40 smithi116 ceph-mon[31830]: from='client.? 172.21.15.116:0/2414423426' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T09:11:41.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T09:11:41.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab20637700 1 Processor -- start 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab20637700 1 -- start start 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab20637700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1807c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab20637700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab1807dff0 con 0x7fab1807c2c0 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab1e3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1807c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab1e3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1807c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45382/0 (socket says 172.21.15.116:45382) 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab1e3d3700 1 -- 172.21.15.116:0/48802704 learned_addr learned my addr 172.21.15.116:0/48802704 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.151+0000 7fab1e3d3700 1 -- 172.21.15.116:0/48802704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1807cc20 con 0x7fab1807c2c0 2024-03-24T09:11:41.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.152+0000 7fab1e3d3700 1 --2- 172.21.15.116:0/48802704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1807c6e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fab080098a0 tx=0x7fab08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df38a416025a25d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.152+0000 7fab1d3d1700 1 -- 172.21.15.116:0/48802704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab08004160 con 0x7fab1807c2c0 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.152+0000 7fab1d3d1700 1 -- 172.21.15.116:0/48802704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab080042c0 con 0x7fab1807c2c0 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.152+0000 7fab1d3d1700 1 -- 172.21.15.116:0/48802704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab080045a0 con 0x7fab1807c2c0 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.152+0000 7fab20637700 1 -- 172.21.15.116:0/48802704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 msgr2=0x7fab1807c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.152+0000 7fab20637700 1 --2- 172.21.15.116:0/48802704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1807c6e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fab080098a0 tx=0x7fab08009bb0 comp rx=0 tx=0).stop 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 -- 172.21.15.116:0/48802704 shutdown_connections 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 --2- 172.21.15.116:0/48802704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1807c6e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 -- 172.21.15.116:0/48802704 >> 172.21.15.116:0/48802704 conn(0x7fab18103100 msgr2=0x7fab18105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 -- 172.21.15.116:0/48802704 shutdown_connections 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 -- 172.21.15.116:0/48802704 wait complete. 2024-03-24T09:11:41.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 Processor -- start 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 -- start start 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.153+0000 7fab20637700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab1819d820 con 0x7fab1807c2c0 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab1e3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab1e3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45394/0 (socket says 172.21.15.116:45394) 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab1e3d3700 1 -- 172.21.15.116:0/2414423426 learned_addr learned my addr 172.21.15.116:0/2414423426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab1e3d3700 1 -- 172.21.15.116:0/2414423426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab08009550 con 0x7fab1807c2c0 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab1e3d3700 1 --2- 172.21.15.116:0/2414423426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1819d2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fab1807d330 tx=0x7fab08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab08011630 con 0x7fab1807c2c0 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab08011790 con 0x7fab1807c2c0 2024-03-24T09:11:41.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab1819da20 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab1819dec0 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab08011a30 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fab08022070 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.154+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fab0801c450 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.155+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab181970e0 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.157+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fab08011b90 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.225+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fab18063830 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.228+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7fab08011de0 con 0x7fab1807c2c0 2024-03-24T09:11:41.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.228+0000 7fab0f7fe700 1 -- 172.21.15.116:0/2414423426 <== mon.0 v2:172.21.15.116:3300/0 8 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fab0801c610 con 0x7fab1807c2c0 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 msgr2=0x7fab1819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 --2- 172.21.15.116:0/2414423426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1819d2e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fab1807d330 tx=0x7fab08004750 comp rx=0 tx=0).stop 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 shutdown_connections 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 --2- 172.21.15.116:0/2414423426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1807c2c0 0x7fab1819d2e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 >> 172.21.15.116:0/2414423426 conn(0x7fab18103100 msgr2=0x7fab18105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 shutdown_connections 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:40.229+0000 7fab20637700 1 -- 172.21.15.116:0/2414423426 wait complete. 2024-03-24T09:11:41.179 INFO:teuthology.orchestra.run.smithi116.stdout:Generating new minimal ceph.conf... 2024-03-24T09:11:41.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:41 smithi116 ceph-mon[31830]: from='client.? 172.21.15.116:0/1030459439' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:11:42.276 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.582+0000 7f893cea4700 1 Processor -- start 2024-03-24T09:11:42.276 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.582+0000 7f893cea4700 1 -- start start 2024-03-24T09:11:42.276 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.582+0000 7f893cea4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893810a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:42.276 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.582+0000 7f893cea4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f893807be70 con 0x7f89381085f0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893810a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893810a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45410/0 (socket says 172.21.15.116:45410) 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893659c700 1 -- 172.21.15.116:0/796812937 learned_addr learned my addr 172.21.15.116:0/796812937 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893659c700 1 -- 172.21.15.116:0/796812937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893810af20 con 0x7f89381085f0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893659c700 1 --2- 172.21.15.116:0/796812937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893810a9e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8930009e80 tx=0x7f893000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9f2fb1e56d35ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893559a700 1 -- 172.21.15.116:0/796812937 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f893000e070 con 0x7f89381085f0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893559a700 1 -- 172.21.15.116:0/796812937 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f893000b8c0 con 0x7f89381085f0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.583+0000 7f893559a700 1 -- 172.21.15.116:0/796812937 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89300044b0 con 0x7f89381085f0 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 -- 172.21.15.116:0/796812937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 msgr2=0x7f893810a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 --2- 172.21.15.116:0/796812937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893810a9e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8930009e80 tx=0x7f893000b330 comp rx=0 tx=0).stop 2024-03-24T09:11:42.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 -- 172.21.15.116:0/796812937 shutdown_connections 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 --2- 172.21.15.116:0/796812937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893810a9e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 -- 172.21.15.116:0/796812937 >> 172.21.15.116:0/796812937 conn(0x7f8938102020 msgr2=0x7f8938104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 -- 172.21.15.116:0/796812937 shutdown_connections 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.584+0000 7f893cea4700 1 -- 172.21.15.116:0/796812937 wait complete. 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893cea4700 1 Processor -- start 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893cea4700 1 -- start start 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893cea4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893cea4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f893819bd40 con 0x7f89381085f0 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45422/0 (socket says 172.21.15.116:45422) 2024-03-24T09:11:42.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893659c700 1 -- 172.21.15.116:0/1030459439 learned_addr learned my addr 172.21.15.116:0/1030459439 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893659c700 1 -- 172.21.15.116:0/1030459439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8930009b30 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893659c700 1 --2- 172.21.15.116:0/1030459439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893819b800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f893000b7b0 tx=0x7f8930004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f893000e040 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f893000bd70 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893819bf40 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f893001c660 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.585+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893819c440 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.586+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f893001c8d0 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.586+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8930007730 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.587+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8938063830 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.589+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f893002a060 con 0x7f89381085f0 2024-03-24T09:11:42.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.655+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f893810bcb0 con 0x7f89381085f0 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.655+0000 7f892f7fe700 1 -- 172.21.15.116:0/1030459439 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f8930020070 con 0x7f89381085f0 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 msgr2=0x7f893819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 --2- 172.21.15.116:0/1030459439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893819b800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f893000b7b0 tx=0x7f8930004600 comp rx=0 tx=0).stop 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 shutdown_connections 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 --2- 172.21.15.116:0/1030459439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89381085f0 0x7f893819b800 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 >> 172.21.15.116:0/1030459439 conn(0x7f8938102020 msgr2=0x7f8938102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 shutdown_connections 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:41.656+0000 7f893cea4700 1 -- 172.21.15.116:0/1030459439 wait complete. 2024-03-24T09:11:42.280 INFO:teuthology.orchestra.run.smithi116.stdout:Restarting the monitor... 2024-03-24T09:11:42.530 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:42 smithi116 systemd[1]: Stopping Ceph mon.a for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:11:42.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:42 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a[31826]: 2024-03-24T09:11:42.529+0000 7f0b7478c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:11:42.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:42 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a[31826]: 2024-03-24T09:11:42.529+0000 7f0b7478c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T09:11:42.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:42 smithi116 podman[32079]: 2024-03-24 09:11:42.570939079 +0000 UTC m=+0.133785376 container died 9ac30382e3110364acba044cb28e84d616f04ae710b612601c0822f500ceeb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:11:43.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:42 smithi116 podman[32079]: 2024-03-24 09:11:42.937793221 +0000 UTC m=+0.500639477 container cleanup 9ac30382e3110364acba044cb28e84d616f04ae710b612601c0822f500ceeb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T09:11:43.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:42 smithi116 bash[32079]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a 2024-03-24T09:11:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 podman[32093]: 2024-03-24 09:11:43.205408905 +0000 UTC m=+0.625373381 container remove 9ac30382e3110364acba044cb28e84d616f04ae710b612601c0822f500ceeb1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T09:11:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T09:11:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 systemd[1]: Stopped Ceph mon.a for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:11:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 systemd[1]: Starting Ceph mon.a for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:11:43.858 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 podman[32189]: 2024-03-24 09:11:43.473781141 +0000 UTC m=+0.018875868 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 podman[32189]: 2024-03-24 09:11:43.589412002 +0000 UTC m=+0.134506729 container create b745d26f56b3945da7cc8d897d939d3c6b40d94bfcd6ca7e3777f6fd6f9e1a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 podman[32189]: 2024-03-24 09:11:43.758281954 +0000 UTC m=+0.303376686 container init b745d26f56b3945da7cc8d897d939d3c6b40d94bfcd6ca7e3777f6fd6f9e1a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 podman[32189]: 2024-03-24 09:11:43.761084074 +0000 UTC m=+0.306178803 container start b745d26f56b3945da7cc8d897d939d3c6b40d94bfcd6ca7e3777f6fd6f9e1a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev), process ceph-mon, pid 2 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: pidfile_write: ignore empty --pid-file 2024-03-24T09:11:43.859 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: load: jerasure load: lrc 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: RocksDB version: 7.9.2 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Git sha 0 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Compile date 2024-03-23 03:28:38 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: DB SUMMARY 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: DB Session ID: PQ8OLUUTYZQAEPGQYXKC 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: CURRENT file: CURRENT 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T09:11:43.860 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104796 ; 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.error_if_exists: 0 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.create_if_missing: 0 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.paranoid_checks: 1 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.env: 0x55c2b1364c20 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T09:11:43.861 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.info_log: 0x55c2b1c95ac0 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.statistics: (nil) 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.use_fsync: 0 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_log_file_size: 0 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T09:11:43.862 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.allow_fallocate: 1 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.use_direct_reads: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.db_log_dir: 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.wal_dir: 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T09:11:43.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.write_buffer_manager: 0x55c2b1c681e0 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T09:11:43.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T09:11:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T09:11:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T09:11:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T09:11:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T09:11:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.unordered_write: 0 2024-03-24T09:11:43.866 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.row_cache: None 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.wal_filter: None 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.two_write_queues: 0 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.wal_compression: 0 2024-03-24T09:11:43.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.atomic_flush: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.log_readahead_size: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T09:11:43.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_background_jobs: 2 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_background_compactions: -1 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_subcompactions: 1 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T09:11:43.869 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_open_files: -1 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_background_flushes: -1 2024-03-24T09:11:43.870 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Compression algorithms supported: 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kZSTD supported: 0 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kXpressCompression supported: 0 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kZlibCompression supported: 1 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kSnappyCompression supported: 1 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kLZ4Compression supported: 1 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: kBZip2Compression supported: 0 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T09:11:43.871 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.merge_operator: 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_filter: None 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T09:11:43.872 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c2b1c95bc0) 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_type: 0 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_index_type: 0 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_shortening: 1 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: checksum: 4 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: no_block_cache: 0 2024-03-24T09:11:43.873 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache: 0x55c2b1c99090 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_options: 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: capacity : 536870912 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_shard_bits : 4 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: strict_capacity_limit : 0 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_compressed: (nil) 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: persistent_cache: (nil) 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size: 4096 2024-03-24T09:11:43.874 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size_deviation: 10 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_restart_interval: 16 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_block_restart_interval: 1 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: metadata_block_size: 4096 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: partition_filters: 0 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: use_delta_encoding: 1 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: filter_policy: bloomfilter 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: whole_key_filtering: 1 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: verify_compression: 0 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: format_version: 5 2024-03-24T09:11:43.875 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_index_compression: 1 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_align: 0 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout: prepopulate_block_cache: 0 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression: NoCompression 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T09:11:43.876 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.num_levels: 7 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T09:11:43.877 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T09:11:43.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T09:11:43.879 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T09:11:43.880 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.table_properties_collectors: 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.inplace_update_support: 0 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T09:11:43.881 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.bloom_locality: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.max_successive_merges: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.ttl: 2592000 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T09:11:43.882 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.enable_blob_files: false 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.min_blob_size: 0 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T09:11:43.883 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0e15fefd-5287-413a-b30d-44d8f738ed87 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711271503789882, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711271503790822, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99898, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97812, "index_size": 304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14163, "raw_average_key_size": 49, "raw_value_size": 90138, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711271503, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0e15fefd-5287-413a-b30d-44d8f738ed87", "db_session_id": "PQ8OLUUTYZQAEPGQYXKC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711271503790901, "job": 1, "event": "recovery_finished"} 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c2b1d8a000 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: DB pointer 0x55c2b1d74000 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** DB Stats ** 2024-03-24T09:11:43.884 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: L0 2/0 99.39 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:11:43.885 INFO:journalctl@ceph.mon.a.smithi116.stdout: Sum 2/0 99.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T09:11:43.886 INFO:journalctl@ceph.mon.a.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative compaction: 0.00 GB write, 33.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval compaction: 0.00 GB write, 33.40 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache BinnedLRUCache@0x55c2b1c99090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:43.887 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???) e1 preinit fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).mds e1 new map 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).mds e1 print_map 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout: e1 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout: legacy client fscid: -1 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout: No filesystems configured 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T09:11:43.888 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T09:11:43.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T09:11:43.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 bash[32189]: b745d26f56b3945da7cc8d897d939d3c6b40d94bfcd6ca7e3777f6fd6f9e1a3d 2024-03-24T09:11:43.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 systemd[1]: Started Ceph mon.a for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:11:43.889 INFO:teuthology.orchestra.run.smithi116.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T09:11:44.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T09:11:44.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: monmap epoch 1 2024-03-24T09:11:44.169 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: last_changed 2024-03-24T09:11:35.626561+0000 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: created 2024-03-24T09:11:35.626561+0000 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: min_mon_release 19 (squid) 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: election_strategy: 1 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: fsmap 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T09:11:44.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:43 smithi116 ceph-mon[32207]: mgrmap e1: no daemons active 2024-03-24T09:11:44.914 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff797391700 1 Processor -- start 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff797391700 1 -- start start 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff797391700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff7901085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff797391700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff790108b70 con 0x7ff790108180 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff79512d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff7901085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff79512d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff7901085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45434/0 (socket says 172.21.15.116:45434) 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.271+0000 7ff79512d700 1 -- 172.21.15.116:0/71842629 learned_addr learned my addr 172.21.15.116:0/71842629 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.272+0000 7ff79512d700 1 -- 172.21.15.116:0/71842629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7901066c0 con 0x7ff790108180 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.272+0000 7ff79512d700 1 --2- 172.21.15.116:0/71842629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff7901085a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff78c0098a0 tx=0x7ff78c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7495229e040bbdf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:44.915 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.272+0000 7ff787fff700 1 -- 172.21.15.116:0/71842629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff78c004160 con 0x7ff790108180 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.272+0000 7ff787fff700 1 -- 172.21.15.116:0/71842629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff78c0042c0 con 0x7ff790108180 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.272+0000 7ff787fff700 1 -- 172.21.15.116:0/71842629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff78c011a50 con 0x7ff790108180 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- 172.21.15.116:0/71842629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 msgr2=0x7ff7901085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 --2- 172.21.15.116:0/71842629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff7901085a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff78c0098a0 tx=0x7ff78c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- 172.21.15.116:0/71842629 shutdown_connections 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 --2- 172.21.15.116:0/71842629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff7901085a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- 172.21.15.116:0/71842629 >> 172.21.15.116:0/71842629 conn(0x7ff790102020 msgr2=0x7ff790104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- 172.21.15.116:0/71842629 shutdown_connections 2024-03-24T09:11:44.916 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- 172.21.15.116:0/71842629 wait complete. 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 Processor -- start 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- start start 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff790191be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.273+0000 7ff797391700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff790192120 con 0x7ff790108180 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff79512d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff790191be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff79512d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff790191be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45436/0 (socket says 172.21.15.116:45436) 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff79512d700 1 -- 172.21.15.116:0/546338594 learned_addr learned my addr 172.21.15.116:0/546338594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff79512d700 1 -- 172.21.15.116:0/546338594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c009550 con 0x7ff790108180 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff79512d700 1 --2- 172.21.15.116:0/546338594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff790191be0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff78c006af0 tx=0x7ff78c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:44.917 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff78c003a30 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7ff78c003b90 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff790190310 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff78c019750 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7901907b0 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff78c019d50 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.274+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff78c02c6a0 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.275+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7901911c0 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.277+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff78c042dd0 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.346+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7ff79010aea0 con 0x7ff790108180 2024-03-24T09:11:44.918 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.351+0000 7ff7867fc700 1 -- 172.21.15.116:0/546338594 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7ff78c031020 con 0x7ff790108180 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 msgr2=0x7ff790191be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 --2- 172.21.15.116:0/546338594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff790191be0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff78c006af0 tx=0x7ff78c011ab0 comp rx=0 tx=0).stop 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 shutdown_connections 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 --2- 172.21.15.116:0/546338594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff790108180 0x7ff790191be0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 >> 172.21.15.116:0/546338594 conn(0x7ff790102020 msgr2=0x7ff790102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 shutdown_connections 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:44.352+0000 7ff797391700 1 -- 172.21.15.116:0/546338594 wait complete. 2024-03-24T09:11:44.919 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T09:11:44.930 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:11:44.930 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mgr... 2024-03-24T09:11:44.931 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T09:11:44.931 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T09:11:45.178 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x 2024-03-24T09:11:45.178 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-24T09:11:45.331 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-69c76602-e9be-11ee-95cd-87774f69a715.target.wants/ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-69c76602-e9be-11ee-95cd-87774f69a715@.service. 2024-03-24T09:11:45.566 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:45 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/546338594' entity='client.admin' 2024-03-24T09:11:45.566 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 systemd[1]: Starting Ceph mgr.x for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:11:45.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 podman[32425]: 2024-03-24 09:11:45.56652415 +0000 UTC m=+0.113367229 container create 00e781c18f7d81f63ed6eb2638a231e626d4fc12f3cc5786d348bc8282f89d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T09:11:45.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 podman[32425]: 2024-03-24 09:11:45.47131912 +0000 UTC m=+0.018162196 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T09:11:45.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 podman[32425]: 2024-03-24 09:11:45.752432798 +0000 UTC m=+0.299275883 container init 00e781c18f7d81f63ed6eb2638a231e626d4fc12f3cc5786d348bc8282f89d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T09:11:45.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 podman[32425]: 2024-03-24 09:11:45.754952314 +0000 UTC m=+0.301795391 container start 00e781c18f7d81f63ed6eb2638a231e626d4fc12f3cc5786d348bc8282f89d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T09:11:45.899 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:11:45.899 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-24T09:11:45.906 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:11:45.906 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-24T09:11:45.906 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-24T09:11:45.906 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T09:11:45.914 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:11:45.914 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-24T09:11:45.922 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:11:45.922 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-24T09:11:45.923 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-24T09:11:45.923 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T09:11:45.923 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr to start... 2024-03-24T09:11:45.923 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr... 2024-03-24T09:11:46.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 bash[32425]: 00e781c18f7d81f63ed6eb2638a231e626d4fc12f3cc5786d348bc8282f89d51 2024-03-24T09:11:46.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:45 smithi116 systemd[1]: Started Ceph mgr.x for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:11:46.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:46.321+0000 7feb692ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:11:46.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:46.414+0000 7feb692ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:11:47.384 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.056+0000 7feb692ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:11:47.384 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.148+0000 7feb692ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:11:47.384 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.288+0000 7feb692ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:11:47.727 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.383+0000 7feb692ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:11:47.727 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.466+0000 7feb692ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:11:47.728 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.727+0000 7feb692ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:11:48.016 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.834+0000 7feb692ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:11:48.016 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:47.920+0000 7feb692ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:11:48.016 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:48.015+0000 7feb692ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:11:48.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:48.190+0000 7feb692ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:11:49.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:49 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3969858772' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T09:11:49.393 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:49.191+0000 7feb692ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:11:49.393 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:49.331+0000 7feb692ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:11:49.932 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:49.673+0000 7feb692ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:11:49.932 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:49.762+0000 7feb692ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:11:50.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:49.958+0000 7feb692ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:11:50.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:50.142+0000 7feb692ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:11:50.688 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:50.429+0000 7feb692ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:11:50.688 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:50.687+0000 7feb692ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:11:51.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:50.925+0000 7feb692ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:11:52.067 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:51.730+0000 7feb692ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:11:52.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:52.300+0000 7feb692ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:11:52.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:52.399+0000 7feb692ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:11:52.891 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:11:52.617+0000 7feb692ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:11:52.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: Activating manager daemon x 2024-03-24T09:11:52.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: mgrmap e2: x(active, starting, since 0.00290153s) 2024-03-24T09:11:52.891 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: Manager daemon x is now available 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' 2024-03-24T09:11:52.892 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:52 smithi116 ceph-mon[32207]: from='mgr.14100 172.21.15.116:0/2372573831' entity='mgr.x' 2024-03-24T09:11:55.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:54 smithi116 ceph-mon[32207]: mgrmap e3: x(active, since 1.0072s) 2024-03-24T09:11:55.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:55.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:11:55.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:11:55.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T09:11:55.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.519 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T09:11:55.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:55.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T09:11:55.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T09:11:55.520 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T09:11:55.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T09:11:55.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T09:11:38.628462+0000", 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.220+0000 7f870fd3c700 1 Processor -- start 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.221+0000 7f870fd3c700 1 -- start start 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.221+0000 7f870fd3c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f8708109660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.221+0000 7f870fd3c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8708109c30 con 0x7f8708109240 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.221+0000 7f870dad8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f8708109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.221+0000 7f870dad8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f8708109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60422/0 (socket says 172.21.15.116:60422) 2024-03-24T09:11:55.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.221+0000 7f870dad8700 1 -- 172.21.15.116:0/3136362595 learned_addr learned my addr 172.21.15.116:0/3136362595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870dad8700 1 -- 172.21.15.116:0/3136362595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8708107780 con 0x7f8708109240 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870dad8700 1 --2- 172.21.15.116:0/3136362595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f8708109660 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f87040098a0 tx=0x7f8704009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a6d58c9783669b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870cad6700 1 -- 172.21.15.116:0/3136362595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8704004160 con 0x7f8708109240 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870cad6700 1 -- 172.21.15.116:0/3136362595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f87040042c0 con 0x7f8708109240 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870cad6700 1 -- 172.21.15.116:0/3136362595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8704011a50 con 0x7f8708109240 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870fd3c700 1 -- 172.21.15.116:0/3136362595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 msgr2=0x7f8708109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.222+0000 7f870fd3c700 1 --2- 172.21.15.116:0/3136362595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f8708109660 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f87040098a0 tx=0x7f8704009bb0 comp rx=0 tx=0).stop 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 -- 172.21.15.116:0/3136362595 shutdown_connections 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 --2- 172.21.15.116:0/3136362595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f8708109660 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 -- 172.21.15.116:0/3136362595 >> 172.21.15.116:0/3136362595 conn(0x7f8708103100 msgr2=0x7f8708105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 -- 172.21.15.116:0/3136362595 shutdown_connections 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 -- 172.21.15.116:0/3136362595 wait complete. 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 Processor -- start 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 -- start start 2024-03-24T09:11:55.530 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f870819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870fd3c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f870819beb0 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870dad8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f870819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870dad8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f870819b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60432/0 (socket says 172.21.15.116:60432) 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.223+0000 7f870dad8700 1 -- 172.21.15.116:0/3969858772 learned_addr learned my addr 172.21.15.116:0/3969858772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f870dad8700 1 -- 172.21.15.116:0/3969858772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8704009550 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f870dad8700 1 --2- 172.21.15.116:0/3969858772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f870819b970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8704006af0 tx=0x7f8704011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8704003930 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f8704003a90 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870819c0b0 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8704019670 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870819c550 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8704019cc0 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.224+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8704024a40 con 0x7f8708109240 2024-03-24T09:11:55.531 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.225+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8708063830 con 0x7f8708109240 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.227+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f870401aa30 con 0x7f8708109240 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.298+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f870819f040 con 0x7f8708109240 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.298+0000 7f86feffd700 1 -- 172.21.15.116:0/3969858772 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f870401a430 con 0x7f8708109240 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 msgr2=0x7f870819b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 --2- 172.21.15.116:0/3969858772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f870819b970 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8704006af0 tx=0x7f8704011ab0 comp rx=0 tx=0).stop 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 shutdown_connections 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 --2- 172.21.15.116:0/3969858772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8708109240 0x7f870819b970 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 >> 172.21.15.116:0/3969858772 conn(0x7f8708103100 msgr2=0x7f8708103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 shutdown_connections 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:49.299+0000 7f870fd3c700 1 -- 172.21.15.116:0/3969858772 wait complete. 2024-03-24T09:11:55.532 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (1/15)... 2024-03-24T09:11:57.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:56 smithi116 ceph-mon[32207]: mgrmap e4: x(active, since 3s) 2024-03-24T09:11:58.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:58 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/30836219' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T09:11:58.943 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T09:11:58.944 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T09:11:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T09:11:58.946 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T09:11:38.628462+0000", 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.982+0000 7f0f07f1e700 1 Processor -- start 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f07f1e700 1 -- start start 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f07f1e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0010aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f07f1e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0007beb0 con 0x7f0f001086f0 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f05cba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0010aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f05cba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0010aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36598/0 (socket says 172.21.15.116:36598) 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f05cba700 1 -- 172.21.15.116:0/3246019033 learned_addr learned my addr 172.21.15.116:0/3246019033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f05cba700 1 -- 172.21.15.116:0/3246019033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0010b020 con 0x7f0f001086f0 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.983+0000 7f0f05cba700 1 --2- 172.21.15.116:0/3246019033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0010aae0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0098a0 tx=0x7f0efc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b22ae205757911 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f04cb8700 1 -- 172.21.15.116:0/3246019033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0efc004160 con 0x7f0f001086f0 2024-03-24T09:11:58.947 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f04cb8700 1 -- 172.21.15.116:0/3246019033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0efc0042c0 con 0x7f0f001086f0 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f04cb8700 1 -- 172.21.15.116:0/3246019033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0efc011a50 con 0x7f0f001086f0 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f07f1e700 1 -- 172.21.15.116:0/3246019033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 msgr2=0x7f0f0010aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f07f1e700 1 --2- 172.21.15.116:0/3246019033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0010aae0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0098a0 tx=0x7f0efc009bb0 comp rx=0 tx=0).stop 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f07f1e700 1 -- 172.21.15.116:0/3246019033 shutdown_connections 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f07f1e700 1 --2- 172.21.15.116:0/3246019033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0010aae0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.984+0000 7f0f07f1e700 1 -- 172.21.15.116:0/3246019033 >> 172.21.15.116:0/3246019033 conn(0x7f0f00102040 msgr2=0x7f0f001044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f07f1e700 1 -- 172.21.15.116:0/3246019033 shutdown_connections 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f07f1e700 1 -- 172.21.15.116:0/3246019033 wait complete. 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f07f1e700 1 Processor -- start 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f07f1e700 1 -- start start 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f07f1e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0019ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f07f1e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0019bf40 con 0x7f0f001086f0 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f05cba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0019ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f05cba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0019ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36606/0 (socket says 172.21.15.116:36606) 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.985+0000 7f0f05cba700 1 -- 172.21.15.116:0/30836219 learned_addr learned my addr 172.21.15.116:0/30836219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:11:58.948 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0f05cba700 1 -- 172.21.15.116:0/30836219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc009550 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0f05cba700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0019ba00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0efc003f40 tx=0x7f0efc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0efc003a40 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0019c140 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f0efc003ba0 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0efc019710 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.986+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0019c5e0 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.987+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f0efc0199d0 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.987+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0010bdc0 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.987+0000 7f0ef6ffd700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f0eec038e40 0x7f0eec03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.987+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0efc04d030 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.987+0000 7f0f054b9700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f0eec038e40 0x7f0eec03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.988+0000 7f0f054b9700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f0eec038e40 0x7f0eec03b300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0006fd0 tx=0x7f0ef0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:57.992+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0efc019c80 con 0x7f0f001086f0 2024-03-24T09:11:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.278+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f0f0019f2b0 con 0x7f0f001086f0 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.278+0000 7f0ef6ffd700 1 -- 172.21.15.116:0/30836219 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f0efc018350 con 0x7f0f001086f0 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f0eec038e40 msgr2=0x7f0eec03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f0eec038e40 0x7f0eec03b300 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0006fd0 tx=0x7f0ef0006e40 comp rx=0 tx=0).stop 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 msgr2=0x7f0f0019ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0019ba00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0efc003f40 tx=0x7f0efc011ab0 comp rx=0 tx=0).stop 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 shutdown_connections 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f0eec038e40 0x7f0eec03b300 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 --2- 172.21.15.116:0/30836219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f001086f0 0x7f0f0019ba00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.280+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 >> 172.21.15.116:0/30836219 conn(0x7f0f00102040 msgr2=0x7f0f00102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.281+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 shutdown_connections 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:58.281+0000 7f0f07f1e700 1 -- 172.21.15.116:0/30836219 wait complete. 2024-03-24T09:11:58.950 INFO:teuthology.orchestra.run.smithi116.stdout:mgr is available 2024-03-24T09:11:59.878 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:11:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3201581192' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T09:12:00.137 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:12:00.137 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T09:12:00.138 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45edc19700 1 Processor -- start 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45edc19700 1 -- start start 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45edc19700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e8109730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45edc19700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8109d00 con 0x7f45e8109310 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e8109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e8109730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36616/0 (socket says 172.21.15.116:36616) 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.389+0000 7f45e77fe700 1 -- 172.21.15.116:0/500782186 learned_addr learned my addr 172.21.15.116:0/500782186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45e77fe700 1 -- 172.21.15.116:0/500782186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e8105bd0 con 0x7f45e8109310 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45e77fe700 1 --2- 172.21.15.116:0/500782186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e8109730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f45d80098a0 tx=0x7f45d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dc53875a70e1f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45e67fc700 1 -- 172.21.15.116:0/500782186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45d8004160 con 0x7f45e8109310 2024-03-24T09:12:00.139 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45e67fc700 1 -- 172.21.15.116:0/500782186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f45d80042c0 con 0x7f45e8109310 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45e67fc700 1 -- 172.21.15.116:0/500782186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45d8011a50 con 0x7f45e8109310 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45edc19700 1 -- 172.21.15.116:0/500782186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 msgr2=0x7f45e8109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45edc19700 1 --2- 172.21.15.116:0/500782186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e8109730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f45d80098a0 tx=0x7f45d8009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45edc19700 1 -- 172.21.15.116:0/500782186 shutdown_connections 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45edc19700 1 --2- 172.21.15.116:0/500782186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e8109730 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.390+0000 7f45edc19700 1 -- 172.21.15.116:0/500782186 >> 172.21.15.116:0/500782186 conn(0x7f45e81017f0 msgr2=0x7f45e8103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.391+0000 7f45edc19700 1 -- 172.21.15.116:0/500782186 shutdown_connections 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.391+0000 7f45edc19700 1 -- 172.21.15.116:0/500782186 wait complete. 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.391+0000 7f45edc19700 1 Processor -- start 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.391+0000 7f45edc19700 1 -- start start 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45edc19700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45edc19700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45e819bef0 con 0x7f45e8109310 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:00.140 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e819b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36632/0 (socket says 172.21.15.116:36632) 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e77fe700 1 -- 172.21.15.116:0/3201581192 learned_addr learned my addr 172.21.15.116:0/3201581192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e77fe700 1 -- 172.21.15.116:0/3201581192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d8009550 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e77fe700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e819b9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f45d8011710 tx=0x7f45d8011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45d8003af0 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f45d8003c50 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e819c0f0 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45d80197a0 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.392+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e819c590 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.393+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f45d8019a10 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.393+0000 7f45e4ff9700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f45d0038ee0 0x7f45d003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.393+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f45d804ced0 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.393+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e8063830 con 0x7f45e8109310 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.393+0000 7f45e6ffd700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f45d0038ee0 0x7f45d003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:00.141 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.394+0000 7f45e6ffd700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f45d0038ee0 0x7f45d003b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f45dc006fd0 tx=0x7f45dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.398+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f45d8018c20 con 0x7f45e8109310 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.591+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f45e810c120 con 0x7f45e8109310 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.592+0000 7f45e4ff9700 1 -- 172.21.15.116:0/3201581192 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f45d8018350 con 0x7f45e8109310 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f45d0038ee0 msgr2=0x7f45d003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f45d0038ee0 0x7f45d003b3a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f45dc006fd0 tx=0x7f45dc006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 msgr2=0x7f45e819b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e819b9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f45d8011710 tx=0x7f45d8011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 shutdown_connections 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f45d0038ee0 0x7f45d003b3a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 --2- 172.21.15.116:0/3201581192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45e8109310 0x7f45e819b9b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 >> 172.21.15.116:0/3201581192 conn(0x7f45e81017f0 msgr2=0x7f45e81022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 shutdown_connections 2024-03-24T09:12:00.142 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:11:59.594+0000 7f45edc19700 1 -- 172.21.15.116:0/3201581192 wait complete. 2024-03-24T09:12:00.143 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling cephadm module... 2024-03-24T09:12:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:00 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3240513975' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T09:12:01.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: ignoring --setuser ceph since I am not root 2024-03-24T09:12:01.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: ignoring --setgroup ceph since I am not root 2024-03-24T09:12:01.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:01 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:01.470+0000 7f1cbd3b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:12:01.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:01 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:01.564+0000 7f1cbd3b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:12:02.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:01 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3240513975' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T09:12:02.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:01 smithi116 ceph-mon[32207]: mgrmap e5: x(active, since 8s) 2024-03-24T09:12:02.210 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06e113c700 1 Processor -- start 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06e113c700 1 -- start start 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06e113c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc1096d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06e113c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc109ca0 con 0x7f06dc1092b0 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06dad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc1096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06dad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc1096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36642/0 (socket says 172.21.15.116:36642) 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06dad9d700 1 -- 172.21.15.116:0/4094176355 learned_addr learned my addr 172.21.15.116:0/4094176355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.589+0000 7f06dad9d700 1 -- 172.21.15.116:0/4094176355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06dc105b00 con 0x7f06dc1092b0 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06dad9d700 1 --2- 172.21.15.116:0/4094176355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc1096d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0098a0 tx=0x7f06cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e520ab5e7326a1fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06d9d9b700 1 -- 172.21.15.116:0/4094176355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06cc004160 con 0x7f06dc1092b0 2024-03-24T09:12:02.216 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06d9d9b700 1 -- 172.21.15.116:0/4094176355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f06cc0042c0 con 0x7f06dc1092b0 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06d9d9b700 1 -- 172.21.15.116:0/4094176355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06cc011a50 con 0x7f06dc1092b0 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 -- 172.21.15.116:0/4094176355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 msgr2=0x7f06dc1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 --2- 172.21.15.116:0/4094176355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc1096d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f06cc0098a0 tx=0x7f06cc009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 -- 172.21.15.116:0/4094176355 shutdown_connections 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 --2- 172.21.15.116:0/4094176355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc1096d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 -- 172.21.15.116:0/4094176355 >> 172.21.15.116:0/4094176355 conn(0x7f06dc101800 msgr2=0x7f06dc103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 -- 172.21.15.116:0/4094176355 shutdown_connections 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.590+0000 7f06e113c700 1 -- 172.21.15.116:0/4094176355 wait complete. 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.591+0000 7f06e113c700 1 Processor -- start 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.591+0000 7f06e113c700 1 -- start start 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.591+0000 7f06e113c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.591+0000 7f06e113c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06dc19bf50 con 0x7f06dc1092b0 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06dad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:02.217 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06dad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc19ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36652/0 (socket says 172.21.15.116:36652) 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06dad9d700 1 -- 172.21.15.116:0/3240513975 learned_addr learned my addr 172.21.15.116:0/3240513975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06dad9d700 1 -- 172.21.15.116:0/3240513975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc009550 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06dad9d700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc19ba10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06cc003f40 tx=0x7f06cc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06cc003a40 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f06cc003ba0 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06dc19c150 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f06cc0196f0 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.592+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06dc19c5f0 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.593+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 45917+0+0 (secure 0 0 0) 0x7f06cc0199b0 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.593+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06dc063830 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.593+0000 7f06c3fff700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f06c4038e40 0x7f06c403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.593+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f06cc04d120 con 0x7f06dc1092b0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.593+0000 7f06da59c700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f06c4038e40 0x7f06c403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.594+0000 7f06da59c700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f06c4038e40 0x7f06c403b300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006fd0 tx=0x7f06d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.598+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f06cc019c60 con 0x7f06dc1092b0 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.848+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f06dc10c050 con 0x7f06dc1092b0 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.899+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f06cc01abe0 con 0x7f06dc1092b0 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.901+0000 7f06c3fff700 1 -- 172.21.15.116:0/3240513975 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f06cc02d960 con 0x7f06dc1092b0 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f06c4038e40 msgr2=0x7f06c403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f06c4038e40 0x7f06c403b300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f06d0006fd0 tx=0x7f06d0006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 msgr2=0x7f06dc19ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc19ba10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f06cc003f40 tx=0x7f06cc011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 shutdown_connections 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f06c4038e40 0x7f06c403b300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 --2- 172.21.15.116:0/3240513975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06dc1092b0 0x7f06dc19ba10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 >> 172.21.15.116:0/3240513975 conn(0x7f06dc101800 msgr2=0x7f06dc102390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 shutdown_connections 2024-03-24T09:12:02.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:00.902+0000 7f06e113c700 1 -- 172.21.15.116:0/3240513975 wait complete. 2024-03-24T09:12:02.545 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:02.205+0000 7f1cbd3b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:12:02.545 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:02.306+0000 7f1cbd3b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:12:02.545 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:02.453+0000 7f1cbd3b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:12:02.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:02.551+0000 7f1cbd3b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:12:02.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:02.637+0000 7f1cbd3b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:12:03.104 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3104291636' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T09:12:03.104 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:02.903+0000 7f1cbd3b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:12:03.105 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:03.018+0000 7f1cbd3b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:12:03.105 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:03.104+0000 7f1cbd3b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:12:03.381 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:03.204+0000 7f1cbd3b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:12:03.513 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:12:03.513 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.643+0000 7f22789ab700 1 Processor -- start 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.643+0000 7f22789ab700 1 -- start start 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.643+0000 7f22789ab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227407c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.643+0000 7f22789ab700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f227407c870 con 0x7f227407de40 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227407c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227407c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36692/0 (socket says 172.21.15.116:36692) 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227259c700 1 -- 172.21.15.116:0/2609796048 learned_addr learned my addr 172.21.15.116:0/2609796048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227259c700 1 -- 172.21.15.116:0/2609796048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227407c9b0 con 0x7f227407de40 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227259c700 1 --2- 172.21.15.116:0/2609796048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227407c2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f22680098a0 tx=0x7f2268009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c753ddfb88ee830 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:03.514 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227159a700 1 -- 172.21.15.116:0/2609796048 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2268004160 con 0x7f227407de40 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227159a700 1 -- 172.21.15.116:0/2609796048 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f22680042c0 con 0x7f227407de40 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.644+0000 7f227159a700 1 -- 172.21.15.116:0/2609796048 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2268011a50 con 0x7f227407de40 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 -- 172.21.15.116:0/2609796048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 msgr2=0x7f227407c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 --2- 172.21.15.116:0/2609796048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227407c2a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f22680098a0 tx=0x7f2268009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 -- 172.21.15.116:0/2609796048 shutdown_connections 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 --2- 172.21.15.116:0/2609796048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227407c2a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 -- 172.21.15.116:0/2609796048 >> 172.21.15.116:0/2609796048 conn(0x7f22741031a0 msgr2=0x7f22741055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 -- 172.21.15.116:0/2609796048 shutdown_connections 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.645+0000 7f22789ab700 1 -- 172.21.15.116:0/2609796048 wait complete. 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f22789ab700 1 Processor -- start 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f22789ab700 1 -- start start 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f22789ab700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227419dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:03.515 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f22789ab700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f227419dff0 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f227259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227419dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f227259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227419dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36708/0 (socket says 172.21.15.116:36708) 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f227259c700 1 -- 172.21.15.116:0/3104291636 learned_addr learned my addr 172.21.15.116:0/3104291636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f227259c700 1 -- 172.21.15.116:0/3104291636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2268009550 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.646+0000 7f227259c700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227419dab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2268011a00 tx=0x7f2268011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.647+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2268003a40 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.647+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2268003ba0 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.647+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227419e1f0 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.647+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f226801a6e0 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.647+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227419e690 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.647+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7f226801a950 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.648+0000 7f22637fe700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 0x7f225c03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.648+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f226804cd60 con 0x7f227407de40 2024-03-24T09:12:03.516 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.648+0000 7f2271d9b700 1 -- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 msgr2=0x7f225c03b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.648+0000 7f2271d9b700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 0x7f225c03b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.648+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227410df10 con 0x7f227407de40 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.653+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f226801ac00 con 0x7f227407de40 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.849+0000 7f2271d9b700 1 -- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 msgr2=0x7f225c03b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.849+0000 7f2271d9b700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 0x7f225c03b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.910+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f22741a10e0 con 0x7f227407de40 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.911+0000 7f22637fe700 1 -- 172.21.15.116:0/3104291636 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f2268018350 con 0x7f227407de40 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 msgr2=0x7f225c03b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 0x7f225c03b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 msgr2=0x7f227419dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227419dab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2268011a00 tx=0x7f2268011ae0 comp rx=0 tx=0).stop 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 shutdown_connections 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7f225c038f30 0x7f225c03b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 --2- 172.21.15.116:0/3104291636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f227407de40 0x7f227419dab0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:03.517 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 >> 172.21.15.116:0/3104291636 conn(0x7f22741031a0 msgr2=0x7f2274103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:03.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 shutdown_connections 2024-03-24T09:12:03.518 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:02.913+0000 7f22789ab700 1 -- 172.21.15.116:0/3104291636 wait complete. 2024-03-24T09:12:03.518 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-24T09:12:03.518 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 5... 2024-03-24T09:12:03.639 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:03.381+0000 7f1cbd3b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:12:04.488 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:04.358+0000 7f1cbd3b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:12:04.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:04.488+0000 7f1cbd3b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:12:05.104 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:04.820+0000 7f1cbd3b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:12:05.104 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:04.909+0000 7f1cbd3b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:12:05.105 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:05.104+0000 7f1cbd3b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:12:05.411 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:05.290+0000 7f1cbd3b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:12:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:05.575+0000 7f1cbd3b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:12:06.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:05.832+0000 7f1cbd3b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:12:06.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:06 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:06.065+0000 7f1cbd3b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:12:07.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:06 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:06.861+0000 7f1cbd3b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:12:07.736 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:07.423+0000 7f1cbd3b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:12:07.737 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:07.521+0000 7f1cbd3b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:12:07.737 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:07.736+0000 7f1cbd3b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:12:08.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: Active manager daemon x restarted 2024-03-24T09:12:08.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: Activating manager daemon x 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: mgrmap e6: x(active, starting, since 0.00342177s) 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: Manager daemon x is now available 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:08.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:07 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: Found migration_current of "None". Setting to last migration. 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:08 smithi116 ceph-mon[32207]: mgrmap e7: x(active, since 1.00632s) 2024-03-24T09:12:09.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.003+0000 7fdb06250700 1 Processor -- start 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.003+0000 7fdb06250700 1 -- start start 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdb06250700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdb06250700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0007be30 con 0x7fdb00108630 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdaff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:09.350 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdaff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36720/0 (socket says 172.21.15.116:36720) 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdaff7fe700 1 -- 172.21.15.116:0/2900233500 learned_addr learned my addr 172.21.15.116:0/2900233500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdaff7fe700 1 -- 172.21.15.116:0/2900233500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0010af60 con 0x7fdb00108630 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.004+0000 7fdaff7fe700 1 --2- 172.21.15.116:0/2900233500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0010aa20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00098a0 tx=0x7fdaf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686391662030ecce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdafe7fc700 1 -- 172.21.15.116:0/2900233500 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdaf0004160 con 0x7fdb00108630 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdafe7fc700 1 -- 172.21.15.116:0/2900233500 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdaf00042c0 con 0x7fdb00108630 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdafe7fc700 1 -- 172.21.15.116:0/2900233500 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdaf0011a50 con 0x7fdb00108630 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 -- 172.21.15.116:0/2900233500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 msgr2=0x7fdb0010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 --2- 172.21.15.116:0/2900233500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0010aa20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdaf00098a0 tx=0x7fdaf0009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 -- 172.21.15.116:0/2900233500 shutdown_connections 2024-03-24T09:12:09.351 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 --2- 172.21.15.116:0/2900233500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0010aa20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 -- 172.21.15.116:0/2900233500 >> 172.21.15.116:0/2900233500 conn(0x7fdb00102020 msgr2=0x7fdb00104460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 -- 172.21.15.116:0/2900233500 shutdown_connections 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.005+0000 7fdb06250700 1 -- 172.21.15.116:0/2900233500 wait complete. 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdb06250700 1 Processor -- start 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdb06250700 1 -- start start 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdb06250700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdb06250700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0019be10 con 0x7fdb00108630 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdaff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdaff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0019b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36732/0 (socket says 172.21.15.116:36732) 2024-03-24T09:12:09.352 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.006+0000 7fdaff7fe700 1 -- 172.21.15.116:0/1448900702 learned_addr learned my addr 172.21.15.116:0/1448900702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdaff7fe700 1 -- 172.21.15.116:0/1448900702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf0009550 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdaff7fe700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0019b8d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0003f40 tx=0x7fdaf0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdaf0003a40 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdaf0003ba0 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0019c010 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdaf0019710 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.007+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0019c4b0 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.008+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 45928+0+0 (secure 0 0 0) 0x7fdaf00199d0 con 0x7fdb00108630 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.008+0000 7fdafcff9700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.008+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 --> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdb00063830 con 0x7fdae8038e90 2024-03-24T09:12:09.353 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.008+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdaf004d180 con 0x7fdb00108630 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.008+0000 7fdafeffd700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 msgr2=0x7fdae803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.008+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.209+0000 7fdafeffd700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 msgr2=0x7fdae803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.209+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.610+0000 7fdafeffd700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 msgr2=0x7fdae803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:04.610+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:05.411+0000 7fdafeffd700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 msgr2=0x7fdae803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:05.411+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:07.012+0000 7fdafeffd700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 msgr2=0x7fdae803b350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/285563997 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:07.012+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T09:12:09.354 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:07.751+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 6) ==== 45695+0+0 (secure 0 0 0) 0x7fdaf0019d20 con 0x7fdb00108630 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:07.751+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 msgr2=0x7fdae803b350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:07.751+0000 7fdafcff9700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.753+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7fdaf0021d60 con 0x7fdb00108630 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.753+0000 7fdafcff9700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdae803c5d0 0x7fdae803e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.753+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fdb00063830 con 0x7fdae803c5d0 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.753+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdae803c5d0 0x7fdae803e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.754+0000 7fdafeffd700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdae803c5d0 0x7fdae803e9c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4003a10 tx=0x7fdaf4009290 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.754+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fdb00063830 con 0x7fdae803c5d0 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fdb0010bc90 con 0x7fdae803c5d0 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdafcff9700 1 -- 172.21.15.116:0/1448900702 <== mgr.14116 v2:172.21.15.116:6800/3871815735 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fdb0010bc90 con 0x7fdae803c5d0 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdae803c5d0 msgr2=0x7fdae803e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:09.355 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdae803c5d0 0x7fdae803e9c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4003a10 tx=0x7fdaf4009290 comp rx=0 tx=0).stop 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 msgr2=0x7fdb0019b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0019b8d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0003f40 tx=0x7fdaf0011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 shutdown_connections 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdae803c5d0 0x7fdae803e9c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb00108630 0x7fdb0019b8d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 --2- 172.21.15.116:0/1448900702 >> [v2:172.21.15.116:6800/285563997,v1:172.21.15.116:6801/285563997] conn(0x7fdae8038e90 0x7fdae803b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.761+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 >> 172.21.15.116:0/1448900702 conn(0x7fdb00102020 msgr2=0x7fdb00102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.762+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 shutdown_connections 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:08.762+0000 7fdb06250700 1 -- 172.21.15.116:0/1448900702 wait complete. 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 5 is available 2024-03-24T09:12:09.356 INFO:teuthology.orchestra.run.smithi116.stdout:Setting orchestrator backend to cephadm... 2024-03-24T09:12:10.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:09 smithi116 ceph-mon[32207]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T09:12:10.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:09 smithi116 ceph-mon[32207]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T09:12:10.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:09 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:10.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.789+0000 7fab7a8ee700 1 Processor -- start 2024-03-24T09:12:10.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.789+0000 7fab7a8ee700 1 -- start start 2024-03-24T09:12:10.588 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.789+0000 7fab7a8ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab741096a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.789+0000 7fab7a8ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab74109c70 con 0x7fab74109280 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.789+0000 7fab73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab741096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab741096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50790/0 (socket says 172.21.15.116:50790) 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab73fff700 1 -- 172.21.15.116:0/2376995926 learned_addr learned my addr 172.21.15.116:0/2376995926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab73fff700 1 -- 172.21.15.116:0/2376995926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab741077c0 con 0x7fab74109280 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab73fff700 1 --2- 172.21.15.116:0/2376995926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab741096a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab640098a0 tx=0x7fab64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f897d2e70cd2fce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab72ffd700 1 -- 172.21.15.116:0/2376995926 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab64004160 con 0x7fab74109280 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab72ffd700 1 -- 172.21.15.116:0/2376995926 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fab640042c0 con 0x7fab74109280 2024-03-24T09:12:10.589 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.790+0000 7fab72ffd700 1 -- 172.21.15.116:0/2376995926 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab64011a50 con 0x7fab74109280 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 -- 172.21.15.116:0/2376995926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 msgr2=0x7fab741096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 --2- 172.21.15.116:0/2376995926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab741096a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fab640098a0 tx=0x7fab64009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 -- 172.21.15.116:0/2376995926 shutdown_connections 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 --2- 172.21.15.116:0/2376995926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab741096a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 -- 172.21.15.116:0/2376995926 >> 172.21.15.116:0/2376995926 conn(0x7fab74103140 msgr2=0x7fab74105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 -- 172.21.15.116:0/2376995926 shutdown_connections 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 -- 172.21.15.116:0/2376995926 wait complete. 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 Processor -- start 2024-03-24T09:12:10.590 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.791+0000 7fab7a8ee700 1 -- start start 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab7a8ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab7407db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab7a8ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab7407e070 con 0x7fab74109280 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab7407db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab7407db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50804/0 (socket says 172.21.15.116:50804) 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab73fff700 1 -- 172.21.15.116:0/4240338343 learned_addr learned my addr 172.21.15.116:0/4240338343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab73fff700 1 -- 172.21.15.116:0/4240338343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab64009550 con 0x7fab74109280 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab73fff700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab7407db30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab64000c00 tx=0x7fab64011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab64003a40 con 0x7fab74109280 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fab64003ba0 con 0x7fab74109280 2024-03-24T09:12:10.591 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7407c260 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab6401a700 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.792+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7407c700 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.793+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 7) ==== 45822+0+0 (secure 0 0 0) 0x7fab6401a980 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.793+0000 7fab717fa700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fab5c038dc0 0x7fab5c03b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.793+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab74063830 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.793+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fab6404d020 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.793+0000 7fab737fe700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fab5c038dc0 0x7fab5c03b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.794+0000 7fab737fe700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fab5c038dc0 0x7fab5c03b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab68006fd0 tx=0x7fab68006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.799+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fab64018530 con 0x7fab74109280 2024-03-24T09:12:10.592 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:09.806+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fab6402d9b0 con 0x7fab74109280 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.010+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fab7407cc10 con 0x7fab5c038dc0 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.017+0000 7fab717fa700 1 -- 172.21.15.116:0/4240338343 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fab7407cc10 con 0x7fab5c038dc0 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fab5c038dc0 msgr2=0x7fab5c03b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fab5c038dc0 0x7fab5c03b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab68006fd0 tx=0x7fab68006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 msgr2=0x7fab7407db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab7407db30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fab64000c00 tx=0x7fab64011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 shutdown_connections 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fab5c038dc0 0x7fab5c03b280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 --2- 172.21.15.116:0/4240338343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab74109280 0x7fab7407db30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:10.593 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 >> 172.21.15.116:0/4240338343 conn(0x7fab74103140 msgr2=0x7fab74103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:10.594 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 shutdown_connections 2024-03-24T09:12:10.594 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.019+0000 7fab7a8ee700 1 -- 172.21.15.116:0/4240338343 wait complete. 2024-03-24T09:12:10.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:09] ENGINE Bus STARTING 2024-03-24T09:12:10.863 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:09] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T09:12:10.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:09] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T09:12:10.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:09] ENGINE Bus STARTED 2024-03-24T09:12:10.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: mgrmap e8: x(active, since 2s) 2024-03-24T09:12:10.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:10.864 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:10 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:11.778 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T09:12:11.778 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd136399700 1 Processor -- start 2024-03-24T09:12:11.778 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd136399700 1 -- start start 2024-03-24T09:12:11.778 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd136399700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd1301085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:11.778 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd136399700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd130108b90 con 0x7fd1301081a0 2024-03-24T09:12:11.778 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd12ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd1301085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd12ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd1301085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50818/0 (socket says 172.21.15.116:50818) 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd12ffff700 1 -- 172.21.15.116:0/1484456784 learned_addr learned my addr 172.21.15.116:0/1484456784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.962+0000 7fd12ffff700 1 -- 172.21.15.116:0/1484456784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1301066e0 con 0x7fd1301081a0 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd12ffff700 1 --2- 172.21.15.116:0/1484456784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd1301085c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd1200098a0 tx=0x7fd120009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdfddb2e52c76ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd12effd700 1 -- 172.21.15.116:0/1484456784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd120004160 con 0x7fd1301081a0 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd12effd700 1 -- 172.21.15.116:0/1484456784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd1200042c0 con 0x7fd1301081a0 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd12effd700 1 -- 172.21.15.116:0/1484456784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd120011a50 con 0x7fd1301081a0 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 -- 172.21.15.116:0/1484456784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 msgr2=0x7fd1301085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:11.779 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 --2- 172.21.15.116:0/1484456784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd1301085c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd1200098a0 tx=0x7fd120009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 -- 172.21.15.116:0/1484456784 shutdown_connections 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 --2- 172.21.15.116:0/1484456784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd1301085c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 -- 172.21.15.116:0/1484456784 >> 172.21.15.116:0/1484456784 conn(0x7fd130102040 msgr2=0x7fd1301044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 -- 172.21.15.116:0/1484456784 shutdown_connections 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.963+0000 7fd136399700 1 -- 172.21.15.116:0/1484456784 wait complete. 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.964+0000 7fd136399700 1 Processor -- start 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.964+0000 7fd136399700 1 -- start start 2024-03-24T09:12:11.780 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.964+0000 7fd136399700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd13007ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.964+0000 7fd136399700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd13019e760 con 0x7fd1301081a0 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.964+0000 7fd12ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd13007ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.964+0000 7fd12ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd13007ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50824/0 (socket says 172.21.15.116:50824) 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd12ffff700 1 -- 172.21.15.116:0/2726629043 learned_addr learned my addr 172.21.15.116:0/2726629043 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd12ffff700 1 -- 172.21.15.116:0/2726629043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd120009550 con 0x7fd1301081a0 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd12ffff700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd13007ba30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd120006af0 tx=0x7fd120011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd120003a30 con 0x7fd1301081a0 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd13019e8a0 con 0x7fd1301081a0 2024-03-24T09:12:11.781 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fd120003b90 con 0x7fd1301081a0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd120019750 con 0x7fd1301081a0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.965+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd13019ec80 con 0x7fd1301081a0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.966+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fd1200199c0 con 0x7fd1301081a0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.966+0000 7fd12d7fa700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fd118038f30 0x7fd11803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.966+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd130063830 con 0x7fd1301081a0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.966+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd12004d200 con 0x7fd1301081a0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.966+0000 7fd12f7fe700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fd118038f30 0x7fd11803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.967+0000 7fd12f7fe700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fd118038f30 0x7fd11803b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd124006fd0 tx=0x7fd124006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:11.782 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:10.971+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd120023370 con 0x7fd1301081a0 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.170+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fd13010aec0 con 0x7fd118038f30 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.171+0000 7fd12d7fa700 1 -- 172.21.15.116:0/2726629043 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fd13010aec0 con 0x7fd118038f30 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fd118038f30 msgr2=0x7fd11803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fd118038f30 0x7fd11803b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd124006fd0 tx=0x7fd124006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 msgr2=0x7fd13007ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd13007ba30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd120006af0 tx=0x7fd120011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 shutdown_connections 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fd118038f30 0x7fd11803b3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 --2- 172.21.15.116:0/2726629043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1301081a0 0x7fd13007ba30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:11.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 >> 172.21.15.116:0/2726629043 conn(0x7fd130102040 msgr2=0x7fd130102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:11.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 shutdown_connections 2024-03-24T09:12:11.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:11.174+0000 7fd136399700 1 -- 172.21.15.116:0/2726629043 wait complete. 2024-03-24T09:12:11.784 INFO:teuthology.orchestra.run.smithi116.stdout:Generating ssh key... 2024-03-24T09:12:12.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:11 smithi116 ceph-mon[32207]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: Generating public/private rsa key pair. 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: Your identification has been saved in /tmp/tmp6ur98kii/key. 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: Your public key has been saved in /tmp/tmp6ur98kii/key.pub. 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: The key fingerprint is: 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: SHA256:KgJHNNUbvSL18jUwymCwB+TUhabxVQZN40hbKK0KJMA ceph-69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: The key's randomart image is: 2024-03-24T09:12:12.808 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: +---[RSA 3072]----+ 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: |+.B+.=+B* | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: |.E.+B.B*=. | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: |o +*.Bo=.+ | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: |....+ * o o | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: |.... . +S. . | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: | o. .. | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: | . . . | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: | . . | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: | | 2024-03-24T09:12:12.809 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: +----[SHA256]-----+ 2024-03-24T09:12:13.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-mon[32207]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:13.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:12 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f15025700 1 Processor -- start 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f15025700 1 -- start start 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f15025700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f10109660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f15025700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f10109c30 con 0x7f2f10109240 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f0ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f10109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f0ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f10109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50834/0 (socket says 172.21.15.116:50834) 2024-03-24T09:12:13.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.189+0000 7f2f0ed9d700 1 -- 172.21.15.116:0/3983365299 learned_addr learned my addr 172.21.15.116:0/3983365299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.190+0000 7f2f0ed9d700 1 -- 172.21.15.116:0/3983365299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f10107780 con 0x7f2f10109240 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.190+0000 7f2f0ed9d700 1 --2- 172.21.15.116:0/3983365299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f10109660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f000098a0 tx=0x7f2f00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f23c9a0d0c62a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.190+0000 7f2f0e59c700 1 -- 172.21.15.116:0/3983365299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00004160 con 0x7f2f10109240 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.190+0000 7f2f0e59c700 1 -- 172.21.15.116:0/3983365299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2f000042c0 con 0x7f2f10109240 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.190+0000 7f2f0e59c700 1 -- 172.21.15.116:0/3983365299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00011a50 con 0x7f2f10109240 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 -- 172.21.15.116:0/3983365299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 msgr2=0x7f2f10109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 --2- 172.21.15.116:0/3983365299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f10109660 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f000098a0 tx=0x7f2f00009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 -- 172.21.15.116:0/3983365299 shutdown_connections 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 --2- 172.21.15.116:0/3983365299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f10109660 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 -- 172.21.15.116:0/3983365299 >> 172.21.15.116:0/3983365299 conn(0x7f2f10103100 msgr2=0x7f2f10105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 -- 172.21.15.116:0/3983365299 shutdown_connections 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 -- 172.21.15.116:0/3983365299 wait complete. 2024-03-24T09:12:13.116 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.191+0000 7f2f15025700 1 Processor -- start 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f15025700 1 -- start start 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f15025700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f1007db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f15025700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1007e070 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f0ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f1007db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f0ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f1007db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50848/0 (socket says 172.21.15.116:50848) 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f0ed9d700 1 -- 172.21.15.116:0/1942600624 learned_addr learned my addr 172.21.15.116:0/1942600624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f0ed9d700 1 -- 172.21.15.116:0/1942600624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f00009550 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f0ed9d700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f1007db30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f00006af0 tx=0x7f2f00011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.192+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00003930 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.193+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f2f00003a90 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.193+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1007c260 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.193+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2f00019670 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.193+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1007c700 con 0x7f2f10109240 2024-03-24T09:12:13.117 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.193+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f2f00019930 con 0x7f2f10109240 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.193+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f10063830 con 0x7f2f10109240 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.194+0000 7f2f0cd99700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f2efc038e90 0x7f2efc03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.194+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2f0004d270 con 0x7f2f10109240 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.194+0000 7f2f0659c700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f2efc038e90 0x7f2efc03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.194+0000 7f2f0659c700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f2efc038e90 0x7f2efc03b350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8006fd0 tx=0x7f2ef8006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.199+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2f0002b340 con 0x7f2f10109240 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.392+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f2f1007cc40 con 0x7f2efc038e90 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.514+0000 7f2f0cd99700 1 -- 172.21.15.116:0/1942600624 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f2f1007cc40 con 0x7f2efc038e90 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f2efc038e90 msgr2=0x7f2efc03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f2efc038e90 0x7f2efc03b350 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8006fd0 tx=0x7f2ef8006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 msgr2=0x7f2f1007db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:13.118 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f1007db30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f00006af0 tx=0x7f2f00011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 shutdown_connections 2024-03-24T09:12:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f2efc038e90 0x7f2efc03b350 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 --2- 172.21.15.116:0/1942600624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f10109240 0x7f2f1007db30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 >> 172.21.15.116:0/1942600624 conn(0x7f2f10103100 msgr2=0x7f2f10103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 shutdown_connections 2024-03-24T09:12:13.119 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:12.516+0000 7f2f15025700 1 -- 172.21.15.116:0/1942600624 wait complete. 2024-03-24T09:12:14.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:13 smithi116 ceph-mon[32207]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:14.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:13 smithi116 ceph-mon[32207]: Generating ssh key... 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.471+0000 7fdf8832b700 1 Processor -- start 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.471+0000 7fdf8832b700 1 -- start start 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.471+0000 7fdf8832b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf801085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.471+0000 7fdf8832b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80108b90 con 0x7fdf801081a0 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.471+0000 7fdf860c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf801085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf860c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf801085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50862/0 (socket says 172.21.15.116:50862) 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf860c7700 1 -- 172.21.15.116:0/851619724 learned_addr learned my addr 172.21.15.116:0/851619724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:14.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf860c7700 1 -- 172.21.15.116:0/851619724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf801066e0 con 0x7fdf801081a0 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf860c7700 1 --2- 172.21.15.116:0/851619724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf801085c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0098a0 tx=0x7fdf7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55da919ea9b3d3a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf850c5700 1 -- 172.21.15.116:0/851619724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c004160 con 0x7fdf801081a0 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf850c5700 1 -- 172.21.15.116:0/851619724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdf7c0042c0 con 0x7fdf801081a0 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.472+0000 7fdf850c5700 1 -- 172.21.15.116:0/851619724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c011a50 con 0x7fdf801081a0 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 -- 172.21.15.116:0/851619724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 msgr2=0x7fdf801085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 --2- 172.21.15.116:0/851619724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf801085c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0098a0 tx=0x7fdf7c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 -- 172.21.15.116:0/851619724 shutdown_connections 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 --2- 172.21.15.116:0/851619724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf801085c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 -- 172.21.15.116:0/851619724 >> 172.21.15.116:0/851619724 conn(0x7fdf80102040 msgr2=0x7fdf801044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 -- 172.21.15.116:0/851619724 shutdown_connections 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 -- 172.21.15.116:0/851619724 wait complete. 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 Processor -- start 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.473+0000 7fdf8832b700 1 -- start start 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf8832b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf80191c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf8832b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf80192140 con 0x7fdf801081a0 2024-03-24T09:12:14.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf860c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf80191c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf860c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf80191c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50874/0 (socket says 172.21.15.116:50874) 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf860c7700 1 -- 172.21.15.116:0/3021581099 learned_addr learned my addr 172.21.15.116:0/3021581099 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf860c7700 1 -- 172.21.15.116:0/3021581099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c009550 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf860c7700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf80191c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c003f40 tx=0x7fdf7c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c003a40 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fdf7c003ba0 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf80190330 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c0196f0 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.474+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf801907d0 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.475+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fdf7c019960 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.475+0000 7fdf777fe700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdf6c038ee0 0x7fdf6c03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.475+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdf7c04d170 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.475+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf80063830 con 0x7fdf801081a0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.475+0000 7fdf858c6700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdf6c038ee0 0x7fdf6c03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:14.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.476+0000 7fdf858c6700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdf6c038ee0 0x7fdf6c03b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdf70006fd0 tx=0x7fdf70006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.480+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdf7c02b340 con 0x7fdf801081a0 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.675+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fdf8010aec0 con 0x7fdf6c038ee0 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.676+0000 7fdf777fe700 1 -- 172.21.15.116:0/3021581099 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fdf8010aec0 con 0x7fdf6c038ee0 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdf6c038ee0 msgr2=0x7fdf6c03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdf6c038ee0 0x7fdf6c03b3a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdf70006fd0 tx=0x7fdf70006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 msgr2=0x7fdf80191c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf80191c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c003f40 tx=0x7fdf7c011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 shutdown_connections 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fdf6c038ee0 0x7fdf6c03b3a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 --2- 172.21.15.116:0/3021581099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf801081a0 0x7fdf80191c00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 >> 172.21.15.116:0/3021581099 conn(0x7fdf80102040 msgr2=0x7fdf80102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 shutdown_connections 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:13.678+0000 7fdf8832b700 1 -- 172.21.15.116:0/3021581099 wait complete. 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T09:12:14.309 INFO:teuthology.orchestra.run.smithi116.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T09:12:14.310 INFO:teuthology.orchestra.run.smithi116.stdout:Adding host smithi116... 2024-03-24T09:12:14.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:14 smithi116 ceph-mon[32207]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:15.893 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:15 smithi116 ceph-mon[32207]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:17.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:16 smithi116 ceph-mon[32207]: Deploying cephadm binary to smithi116 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Added host 'smithi116' with addr '172.21.15.116' 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f0652251700 1 Processor -- start 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f0652251700 1 -- start start 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f0652251700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f0652251700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f064c07bf30 con 0x7f064c108650 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f064b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f064b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50882/0 (socket says 172.21.15.116:50882) 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.738+0000 7f064b7fe700 1 -- 172.21.15.116:0/3545816618 learned_addr learned my addr 172.21.15.116:0/3545816618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f064b7fe700 1 -- 172.21.15.116:0/3545816618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f064c10af80 con 0x7f064c108650 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f064b7fe700 1 --2- 172.21.15.116:0/3545816618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c10aa40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f063c0098a0 tx=0x7f063c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f264a59211d55cf9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:19.672 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f064a7fc700 1 -- 172.21.15.116:0/3545816618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f063c004160 con 0x7f064c108650 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f064a7fc700 1 -- 172.21.15.116:0/3545816618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f063c0042c0 con 0x7f064c108650 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f064a7fc700 1 -- 172.21.15.116:0/3545816618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f063c011a50 con 0x7f064c108650 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f0652251700 1 -- 172.21.15.116:0/3545816618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 msgr2=0x7f064c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f0652251700 1 --2- 172.21.15.116:0/3545816618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c10aa40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f063c0098a0 tx=0x7f063c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f0652251700 1 -- 172.21.15.116:0/3545816618 shutdown_connections 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f0652251700 1 --2- 172.21.15.116:0/3545816618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c10aa40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.739+0000 7f0652251700 1 -- 172.21.15.116:0/3545816618 >> 172.21.15.116:0/3545816618 conn(0x7f064c102020 msgr2=0x7f064c104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.740+0000 7f0652251700 1 -- 172.21.15.116:0/3545816618 shutdown_connections 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.740+0000 7f0652251700 1 -- 172.21.15.116:0/3545816618 wait complete. 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.740+0000 7f0652251700 1 Processor -- start 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.740+0000 7f0652251700 1 -- start start 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.740+0000 7f0652251700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:19.673 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.740+0000 7f0652251700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f064c19bf20 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f064b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f064b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50898/0 (socket says 172.21.15.116:50898) 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f064b7fe700 1 -- 172.21.15.116:0/2289156945 learned_addr learned my addr 172.21.15.116:0/2289156945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f064b7fe700 1 -- 172.21.15.116:0/2289156945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063c009550 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f064b7fe700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c19b9e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f063c011710 tx=0x7f063c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f063c003af0 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f063c003c50 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f064c19c120 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f063c0197a0 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.741+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f064c19c5c0 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.742+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f063c019ac0 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.742+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f064c10bdd0 con 0x7f064c108650 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.742+0000 7f0648ff9700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f0634038ee0 0x7f063403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:19.674 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.742+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f063c04d280 con 0x7f064c108650 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.742+0000 7f064affd700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f0634038ee0 0x7f063403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.743+0000 7f064affd700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f0634038ee0 0x7f063403b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0640006fd0 tx=0x7f0640006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.747+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f063c018c20 con 0x7f064c108650 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:14.947+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}) -- 0x7f064c103290 con 0x7f0634038ee0 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.114+0000 7f0648ff9700 1 -- 172.21.15.116:0/2289156945 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f064c103290 con 0x7f0634038ee0 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f0634038ee0 msgr2=0x7f063403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f0634038ee0 0x7f063403b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0640006fd0 tx=0x7f0640006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 msgr2=0x7f064c19b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c19b9e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f063c011710 tx=0x7f063c011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 shutdown_connections 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f0634038ee0 0x7f063403b3a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 --2- 172.21.15.116:0/2289156945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f064c108650 0x7f064c19b9e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 >> 172.21.15.116:0/2289156945 conn(0x7f064c102020 msgr2=0x7f064c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:19.675 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 shutdown_connections 2024-03-24T09:12:19.676 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:19.117+0000 7f0652251700 1 -- 172.21.15.116:0/2289156945 wait complete. 2024-03-24T09:12:19.676 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mon service... 2024-03-24T09:12:20.209 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:20 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:20.210 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:20 smithi116 ceph-mon[32207]: Added host smithi116 2024-03-24T09:12:20.210 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:20 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:21.185 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T09:12:21.185 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f3095466700 1 Processor -- start 2024-03-24T09:12:21.185 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f3095466700 1 -- start start 2024-03-24T09:12:21.185 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f3095466700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:21.185 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f3095466700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f309010af40 con 0x7f3090108610 2024-03-24T09:12:21.185 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f308effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f308effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41076/0 (socket says 172.21.15.116:41076) 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f308effd700 1 -- 172.21.15.116:0/166701675 learned_addr learned my addr 172.21.15.116:0/166701675 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.126+0000 7f308effd700 1 -- 172.21.15.116:0/166701675 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309010b080 con 0x7f3090108610 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f308effd700 1 --2- 172.21.15.116:0/166701675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309010aa00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30800098a0 tx=0x7f3080009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e63c300f5aa113 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f308dffb700 1 -- 172.21.15.116:0/166701675 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080004160 con 0x7f3090108610 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f308dffb700 1 -- 172.21.15.116:0/166701675 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f30800042c0 con 0x7f3090108610 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f308dffb700 1 -- 172.21.15.116:0/166701675 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080011a50 con 0x7f3090108610 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 -- 172.21.15.116:0/166701675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 msgr2=0x7f309010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:21.186 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 --2- 172.21.15.116:0/166701675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309010aa00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f30800098a0 tx=0x7f3080009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 -- 172.21.15.116:0/166701675 shutdown_connections 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 --2- 172.21.15.116:0/166701675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309010aa00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 -- 172.21.15.116:0/166701675 >> 172.21.15.116:0/166701675 conn(0x7f3090102020 msgr2=0x7f3090104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 -- 172.21.15.116:0/166701675 shutdown_connections 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.127+0000 7f3095466700 1 -- 172.21.15.116:0/166701675 wait complete. 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f3095466700 1 Processor -- start 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f3095466700 1 -- start start 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f3095466700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f3095466700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f309019bf30 con 0x7f3090108610 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f308effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:21.187 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f308effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309019b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41086/0 (socket says 172.21.15.116:41086) 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.128+0000 7f308effd700 1 -- 172.21.15.116:0/2776209944 learned_addr learned my addr 172.21.15.116:0/2776209944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f308effd700 1 -- 172.21.15.116:0/2776209944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3080009550 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f308effd700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309019b9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3080011710 tx=0x7f3080011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3080003af0 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7f3080003c50 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309019c130 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30800197a0 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.129+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f309019c5d0 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.130+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f3080019a10 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.130+0000 7f3077fff700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f3078038f30 0x7f307803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.130+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f308004d2c0 con 0x7f3090108610 2024-03-24T09:12:21.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.130+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f309010bdb0 con 0x7f3090108610 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.130+0000 7f308e7fc700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f3078038f30 0x7f307803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.130+0000 7f308e7fc700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f3078038f30 0x7f307803b3f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3084006fd0 tx=0x7f3084006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.135+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3080018c20 con 0x7f3090108610 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.344+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f3090103300 con 0x7f3078038f30 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.348+0000 7f3077fff700 1 -- 172.21.15.116:0/2776209944 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f3090103300 con 0x7f3078038f30 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f3078038f30 msgr2=0x7f307803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f3078038f30 0x7f307803b3f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3084006fd0 tx=0x7f3084006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 msgr2=0x7f309019b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309019b9f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3080011710 tx=0x7f3080011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:21.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 shutdown_connections 2024-03-24T09:12:21.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f3078038f30 0x7f307803b3f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:21.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 --2- 172.21.15.116:0/2776209944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3090108610 0x7f309019b9f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:21.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.350+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 >> 172.21.15.116:0/2776209944 conn(0x7f3090102020 msgr2=0x7f3090102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:21.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.351+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 shutdown_connections 2024-03-24T09:12:21.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:20.351+0000 7f3095466700 1 -- 172.21.15.116:0/2776209944 wait complete. 2024-03-24T09:12:21.190 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mgr service... 2024-03-24T09:12:21.477 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:21 smithi116 ceph-mon[32207]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:21.478 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:21 smithi116 ceph-mon[32207]: Saving service mon spec with placement count:5 2024-03-24T09:12:21.478 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:21 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:22.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T09:12:22.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.575+0000 7fcd26910700 1 Processor -- start 2024-03-24T09:12:22.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.575+0000 7fcd26910700 1 -- start start 2024-03-24T09:12:22.551 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd26910700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd20107fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd26910700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd20108580 con 0x7fcd20107b90 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd20107fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd20107fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41096/0 (socket says 172.21.15.116:41096) 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1ffff700 1 -- 172.21.15.116:0/2588244941 learned_addr learned my addr 172.21.15.116:0/2588244941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1ffff700 1 -- 172.21.15.116:0/2588244941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd20108d90 con 0x7fcd20107b90 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1ffff700 1 --2- 172.21.15.116:0/2588244941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd20107fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcd080098a0 tx=0x7fcd08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c4d7b4034a3056 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1effd700 1 -- 172.21.15.116:0/2588244941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd08004160 con 0x7fcd20107b90 2024-03-24T09:12:22.552 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1effd700 1 -- 172.21.15.116:0/2588244941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcd080042c0 con 0x7fcd20107b90 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.576+0000 7fcd1effd700 1 -- 172.21.15.116:0/2588244941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd08011a50 con 0x7fcd20107b90 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 -- 172.21.15.116:0/2588244941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 msgr2=0x7fcd20107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 --2- 172.21.15.116:0/2588244941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd20107fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcd080098a0 tx=0x7fcd08009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 -- 172.21.15.116:0/2588244941 shutdown_connections 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 --2- 172.21.15.116:0/2588244941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd20107fb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 -- 172.21.15.116:0/2588244941 >> 172.21.15.116:0/2588244941 conn(0x7fcd20103150 msgr2=0x7fcd20105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 -- 172.21.15.116:0/2588244941 shutdown_connections 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.577+0000 7fcd26910700 1 -- 172.21.15.116:0/2588244941 wait complete. 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd26910700 1 Processor -- start 2024-03-24T09:12:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd26910700 1 -- start start 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd26910700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd2007db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd26910700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd2007e070 con 0x7fcd20107b90 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd1ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd2007db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd1ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd2007db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41098/0 (socket says 172.21.15.116:41098) 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd1ffff700 1 -- 172.21.15.116:0/17000657 learned_addr learned my addr 172.21.15.116:0/17000657 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd1ffff700 1 -- 172.21.15.116:0/17000657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd08009550 con 0x7fcd20107b90 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.578+0000 7fcd1ffff700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd2007db30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd08006af0 tx=0x7fcd08011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.579+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd08003930 con 0x7fcd20107b90 2024-03-24T09:12:22.554 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.579+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1071+0+0 (secure 0 0 0) 0x7fcd08003a90 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.579+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2007c260 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.579+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd08019670 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.579+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2007c700 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.580+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fcd0801aa30 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.580+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd20063830 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.580+0000 7fcd1d7fa700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fcd0c038e90 0x7fcd0c03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.580+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcd0804d260 con 0x7fcd20107b90 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.580+0000 7fcd1f7fe700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fcd0c038e90 0x7fcd0c03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.581+0000 7fcd1f7fe700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fcd0c038e90 0x7fcd0c03b350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd10006fd0 tx=0x7fcd10006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:22.555 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.585+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd08018760 con 0x7fcd20107b90 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.799+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fcd2010bf90 con 0x7fcd0c038e90 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.806+0000 7fcd1d7fa700 1 -- 172.21.15.116:0/17000657 <== mgr.14116 v2:172.21.15.116:6800/3871815735 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcd2010bf90 con 0x7fcd0c038e90 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.808+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fcd0c038e90 msgr2=0x7fcd0c03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.808+0000 7fcd26910700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fcd0c038e90 0x7fcd0c03b350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd10006fd0 tx=0x7fcd10006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.808+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 msgr2=0x7fcd2007db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.808+0000 7fcd26910700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd2007db30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcd08006af0 tx=0x7fcd08011ab0 comp rx=0 tx=0).stop 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.809+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 shutdown_connections 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.809+0000 7fcd26910700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fcd0c038e90 0x7fcd0c03b350 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.809+0000 7fcd26910700 1 --2- 172.21.15.116:0/17000657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd20107b90 0x7fcd2007db30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.809+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 >> 172.21.15.116:0/17000657 conn(0x7fcd20103150 msgr2=0x7fcd20103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.809+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 shutdown_connections 2024-03-24T09:12:22.556 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:21.809+0000 7fcd26910700 1 -- 172.21.15.116:0/17000657 wait complete. 2024-03-24T09:12:23.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:22 smithi116 ceph-mon[32207]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:22 smithi116 ceph-mon[32207]: Saving service mgr spec with placement count:2 2024-03-24T09:12:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:22 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:24.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:23 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:24.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:23 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:24.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:23 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3773920804' entity='client.admin' 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a82bc700 1 Processor -- start 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a82bc700 1 -- start start 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a82bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a82bc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67a007be10 con 0x7f67a0108660 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a6058700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a6058700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a010aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41112/0 (socket says 172.21.15.116:41112) 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a6058700 1 -- 172.21.15.116:0/1596410382 learned_addr learned my addr 172.21.15.116:0/1596410382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:24.260 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.392+0000 7f67a6058700 1 -- 172.21.15.116:0/1596410382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a010af90 con 0x7f67a0108660 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.393+0000 7f67a6058700 1 --2- 172.21.15.116:0/1596410382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a010aa50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f679c0098a0 tx=0x7f679c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287766a44c0a4436 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.393+0000 7f67a5056700 1 -- 172.21.15.116:0/1596410382 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f679c004160 con 0x7f67a0108660 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.393+0000 7f67a5056700 1 -- 172.21.15.116:0/1596410382 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f679c0042c0 con 0x7f67a0108660 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.393+0000 7f67a5056700 1 -- 172.21.15.116:0/1596410382 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f679c0119b0 con 0x7f67a0108660 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 -- 172.21.15.116:0/1596410382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 msgr2=0x7f67a010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 --2- 172.21.15.116:0/1596410382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a010aa50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f679c0098a0 tx=0x7f679c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 -- 172.21.15.116:0/1596410382 shutdown_connections 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 --2- 172.21.15.116:0/1596410382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a010aa50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 -- 172.21.15.116:0/1596410382 >> 172.21.15.116:0/1596410382 conn(0x7f67a0102010 msgr2=0x7f67a0104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 -- 172.21.15.116:0/1596410382 shutdown_connections 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 -- 172.21.15.116:0/1596410382 wait complete. 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 Processor -- start 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.394+0000 7f67a82bc700 1 -- start start 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.395+0000 7f67a82bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a0107ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.395+0000 7f67a82bc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67a0108420 con 0x7f67a0108660 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.395+0000 7f67a6058700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a0107ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:24.262 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.395+0000 7f67a6058700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a0107ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41118/0 (socket says 172.21.15.116:41118) 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.395+0000 7f67a6058700 1 -- 172.21.15.116:0/3773920804 learned_addr learned my addr 172.21.15.116:0/3773920804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.395+0000 7f67a6058700 1 -- 172.21.15.116:0/3773920804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f679c009550 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.396+0000 7f67a6058700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a0107ee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f679c011980 tx=0x7f679c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.396+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f679c0039c0 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.396+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f679c003b20 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.396+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a0106610 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.396+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f679c019650 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.396+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a0106ab0 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.397+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f679c0198c0 con 0x7f67a0108660 2024-03-24T09:12:24.263 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.397+0000 7f67977fe700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f678c038f30 0x7f678c03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.397+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f679c04d350 con 0x7f67a0108660 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.397+0000 7f67a5857700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f678c038f30 0x7f678c03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.397+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a0063830 con 0x7f67a0108660 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.398+0000 7f67a5857700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f678c038f30 0x7f678c03b3f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6790006fd0 tx=0x7f6790006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.404+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f679c019bc0 con 0x7f67a0108660 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.611+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f67a0106eb0 con 0x7f67a0108660 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.616+0000 7f67977fe700 1 -- 172.21.15.116:0/3773920804 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f679c018350 con 0x7f67a0108660 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f678c038f30 msgr2=0x7f678c03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f678c038f30 0x7f678c03b3f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6790006fd0 tx=0x7f6790006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:24.264 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 msgr2=0x7f67a0107ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a0107ee0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f679c011980 tx=0x7f679c011a60 comp rx=0 tx=0).stop 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 shutdown_connections 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f678c038f30 0x7f678c03b3f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 --2- 172.21.15.116:0/3773920804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67a0108660 0x7f67a0107ee0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 >> 172.21.15.116:0/3773920804 conn(0x7f67a0102010 msgr2=0x7f67a0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 shutdown_connections 2024-03-24T09:12:24.265 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:23.619+0000 7f67a82bc700 1 -- 172.21.15.116:0/3773920804 wait complete. 2024-03-24T09:12:26.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:25 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2078506531' entity='client.admin' 2024-03-24T09:12:26.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:25 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:26.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.655+0000 7f9b5b671700 1 Processor -- start 2024-03-24T09:12:26.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5b671700 1 -- start start 2024-03-24T09:12:26.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5b671700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:26.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5b671700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5407be30 con 0x7f9b54108610 2024-03-24T09:12:26.037 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5940d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5940d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41122/0 (socket says 172.21.15.116:41122) 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5940d700 1 -- 172.21.15.116:0/1691734158 learned_addr learned my addr 172.21.15.116:0/1691734158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.656+0000 7f9b5940d700 1 -- 172.21.15.116:0/1691734158 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5410af40 con 0x7f9b54108610 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.657+0000 7f9b5940d700 1 --2- 172.21.15.116:0/1691734158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5410aa00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b500098a0 tx=0x7f9b50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556b35720d58272b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.657+0000 7f9b4bfff700 1 -- 172.21.15.116:0/1691734158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b50004160 con 0x7f9b54108610 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.657+0000 7f9b4bfff700 1 -- 172.21.15.116:0/1691734158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b500042c0 con 0x7f9b54108610 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.657+0000 7f9b4bfff700 1 -- 172.21.15.116:0/1691734158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b500119b0 con 0x7f9b54108610 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.657+0000 7f9b5b671700 1 -- 172.21.15.116:0/1691734158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 msgr2=0x7f9b5410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.657+0000 7f9b5b671700 1 --2- 172.21.15.116:0/1691734158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5410aa00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b500098a0 tx=0x7f9b50009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 -- 172.21.15.116:0/1691734158 shutdown_connections 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 --2- 172.21.15.116:0/1691734158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5410aa00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 -- 172.21.15.116:0/1691734158 >> 172.21.15.116:0/1691734158 conn(0x7f9b54101fe0 msgr2=0x7f9b54104440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 -- 172.21.15.116:0/1691734158 shutdown_connections 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 -- 172.21.15.116:0/1691734158 wait complete. 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 Processor -- start 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 -- start start 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5419b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.658+0000 7f9b5b671700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5419bdc0 con 0x7f9b54108610 2024-03-24T09:12:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5940d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5419b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5940d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5419b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41134/0 (socket says 172.21.15.116:41134) 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5940d700 1 -- 172.21.15.116:0/2078506531 learned_addr learned my addr 172.21.15.116:0/2078506531 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5940d700 1 -- 172.21.15.116:0/2078506531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b50009550 con 0x7f9b54108610 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5940d700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5419b880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b50011980 tx=0x7f9b50011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b500038b0 con 0x7f9b54108610 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b50003a10 con 0x7f9b54108610 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9b500195e0 con 0x7f9b54108610 2024-03-24T09:12:26.040 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5419bfc0 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.659+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5419c460 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.660+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7f9b50019820 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.660+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b5410bd20 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.660+0000 7f9b4a7fc700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f9b40038e90 0x7f9b4003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.660+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9b5004d160 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.661+0000 7f9b58c0c700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f9b40038e90 0x7f9b4003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.661+0000 7f9b58c0c700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f9b40038e90 0x7f9b4003b350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9b44006fd0 tx=0x7f9b44006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.666+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9b50018dc0 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.885+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f9b5419eda0 con 0x7f9b54108610 2024-03-24T09:12:26.041 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.888+0000 7f9b4a7fc700 1 -- 172.21.15.116:0/2078506531 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f9b50018be0 con 0x7f9b54108610 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f9b40038e90 msgr2=0x7f9b4003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f9b40038e90 0x7f9b4003b350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9b44006fd0 tx=0x7f9b44006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 msgr2=0x7f9b5419b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5419b880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b50011980 tx=0x7f9b50011a60 comp rx=0 tx=0).stop 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 shutdown_connections 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f9b40038e90 0x7f9b4003b350 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 --2- 172.21.15.116:0/2078506531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b54108610 0x7f9b5419b880 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 >> 172.21.15.116:0/2078506531 conn(0x7f9b54101fe0 msgr2=0x7f9b54102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 shutdown_connections 2024-03-24T09:12:26.042 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:24.890+0000 7f9b5b671700 1 -- 172.21.15.116:0/2078506531 wait complete. 2024-03-24T09:12:26.043 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling the dashboard module... 2024-03-24T09:12:26.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:26 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3894682646' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T09:12:26.947 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:26 smithi116 ceph-mon[32207]: from='mgr.14116 172.21.15.116:0/3808278959' entity='mgr.x' 2024-03-24T09:12:27.226 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: ignoring --setuser ceph since I am not root 2024-03-24T09:12:27.226 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: ignoring --setgroup ceph since I am not root 2024-03-24T09:12:27.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa46d426700 1 Processor -- start 2024-03-24T09:12:27.532 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa46d426700 1 -- start start 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa46d426700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa468106260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa46d426700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa468106830 con 0x7fa468105e40 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa466ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa468106260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa466ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa468106260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41144/0 (socket says 172.21.15.116:41144) 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.422+0000 7fa466ffd700 1 -- 172.21.15.116:0/3035911284 learned_addr learned my addr 172.21.15.116:0/3035911284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.423+0000 7fa466ffd700 1 -- 172.21.15.116:0/3035911284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa468108dd0 con 0x7fa468105e40 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.423+0000 7fa466ffd700 1 --2- 172.21.15.116:0/3035911284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa468106260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4580098a0 tx=0x7fa458009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ccffa432137e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.423+0000 7fa465ffb700 1 -- 172.21.15.116:0/3035911284 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa458004160 con 0x7fa468105e40 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.423+0000 7fa465ffb700 1 -- 172.21.15.116:0/3035911284 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4580042c0 con 0x7fa468105e40 2024-03-24T09:12:27.533 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.423+0000 7fa465ffb700 1 -- 172.21.15.116:0/3035911284 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4580119b0 con 0x7fa468105e40 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 -- 172.21.15.116:0/3035911284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 msgr2=0x7fa468106260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 --2- 172.21.15.116:0/3035911284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa468106260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa4580098a0 tx=0x7fa458009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 -- 172.21.15.116:0/3035911284 shutdown_connections 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 --2- 172.21.15.116:0/3035911284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa468106260 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 -- 172.21.15.116:0/3035911284 >> 172.21.15.116:0/3035911284 conn(0x7fa4681017d0 msgr2=0x7fa468103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 -- 172.21.15.116:0/3035911284 shutdown_connections 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.424+0000 7fa46d426700 1 -- 172.21.15.116:0/3035911284 wait complete. 2024-03-24T09:12:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa46d426700 1 Processor -- start 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa46d426700 1 -- start start 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa46d426700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa46819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa46d426700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa46819be10 con 0x7fa468105e40 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa466ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa46819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa466ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa46819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41152/0 (socket says 172.21.15.116:41152) 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa466ffd700 1 -- 172.21.15.116:0/3894682646 learned_addr learned my addr 172.21.15.116:0/3894682646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa466ffd700 1 -- 172.21.15.116:0/3894682646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa458009550 con 0x7fa468105e40 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.425+0000 7fa466ffd700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa46819b8d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa458011980 tx=0x7fa458011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.426+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa458003970 con 0x7fa468105e40 2024-03-24T09:12:27.535 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.426+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46819c010 con 0x7fa468105e40 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.426+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa458003ad0 con 0x7fa468105e40 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.426+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa458019610 con 0x7fa468105e40 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.426+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46819c4b0 con 0x7fa468105e40 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.426+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 45928+0+0 (secure 0 0 0) 0x7fa4580198d0 con 0x7fa468105e40 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.427+0000 7fa44ffff700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fa450038ee0 0x7fa45003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.427+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa45804e100 con 0x7fa468105e40 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.427+0000 7fa4667fc700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fa450038ee0 0x7fa45003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:27.536 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.427+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa46810c000 con 0x7fa468105e40 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.427+0000 7fa4667fc700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fa450038ee0 0x7fa45003b3a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa45c006fd0 tx=0x7fa45c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.432+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa458019bd0 con 0x7fa468105e40 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.694+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa46819eef0 con 0x7fa468105e40 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.891+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa458018be0 con 0x7fa468105e40 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.893+0000 7fa44ffff700 1 -- 172.21.15.116:0/3894682646 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7fa45802d960 con 0x7fa468105e40 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fa450038ee0 msgr2=0x7fa45003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fa450038ee0 0x7fa45003b3a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa45c006fd0 tx=0x7fa45c006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 msgr2=0x7fa46819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:27.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa46819b8d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa458011980 tx=0x7fa458011a60 comp rx=0 tx=0).stop 2024-03-24T09:12:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 shutdown_connections 2024-03-24T09:12:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7fa450038ee0 0x7fa45003b3a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 --2- 172.21.15.116:0/3894682646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa468105e40 0x7fa46819b8d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 >> 172.21.15.116:0/3894682646 conn(0x7fa4681017d0 msgr2=0x7fa4681022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 shutdown_connections 2024-03-24T09:12:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:26.895+0000 7fa46d426700 1 -- 172.21.15.116:0/3894682646 wait complete. 2024-03-24T09:12:27.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:27.479+0000 7f86307de200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:12:27.758 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:27.578+0000 7f86307de200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:12:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:27 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3894682646' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T09:12:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:27 smithi116 ceph-mon[32207]: mgrmap e9: x(active, since 19s) 2024-03-24T09:12:28.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:28.261+0000 7f86307de200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:12:28.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:28.362+0000 7f86307de200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:12:28.508 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:28.507+0000 7f86307de200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:12:28.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:28.610+0000 7f86307de200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:12:28.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:28.699+0000 7f86307de200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:12:29.179 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2484391459' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T09:12:29.180 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:28.978+0000 7f86307de200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:12:29.180 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:29.090+0000 7f86307de200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:12:29.180 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:29.179+0000 7f86307de200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:12:29.298 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.953+0000 7f7cd5a86700 1 Processor -- start 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.953+0000 7f7cd5a86700 1 -- start start 2024-03-24T09:12:29.299 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.953+0000 7f7cd5a86700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd0107f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:29.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.953+0000 7f7cd5a86700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0108560 con 0x7f7cd0107b70 2024-03-24T09:12:29.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7ccf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd0107f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:29.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7ccf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd0107f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38394/0 (socket says 172.21.15.116:38394) 2024-03-24T09:12:29.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7ccf7fe700 1 -- 172.21.15.116:0/4279265600 learned_addr learned my addr 172.21.15.116:0/4279265600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:29.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7ccf7fe700 1 -- 172.21.15.116:0/4279265600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd0108d70 con 0x7f7cd0107b70 2024-03-24T09:12:29.300 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7ccf7fe700 1 --2- 172.21.15.116:0/4279265600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd0107f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40098a0 tx=0x7f7cc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c05a8305eb5cab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7cce7fc700 1 -- 172.21.15.116:0/4279265600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4004160 con 0x7f7cd0107b70 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7cce7fc700 1 -- 172.21.15.116:0/4279265600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cc40042c0 con 0x7f7cd0107b70 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.954+0000 7f7cce7fc700 1 -- 172.21.15.116:0/4279265600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc40119b0 con 0x7f7cd0107b70 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 -- 172.21.15.116:0/4279265600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 msgr2=0x7f7cd0107f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 --2- 172.21.15.116:0/4279265600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd0107f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cc40098a0 tx=0x7f7cc4009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 -- 172.21.15.116:0/4279265600 shutdown_connections 2024-03-24T09:12:29.301 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 --2- 172.21.15.116:0/4279265600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd0107f90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 -- 172.21.15.116:0/4279265600 >> 172.21.15.116:0/4279265600 conn(0x7f7cd0103150 msgr2=0x7f7cd0105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 -- 172.21.15.116:0/4279265600 shutdown_connections 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.955+0000 7f7cd5a86700 1 -- 172.21.15.116:0/4279265600 wait complete. 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7cd5a86700 1 Processor -- start 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7cd5a86700 1 -- start start 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7cd5a86700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd019b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:29.302 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7cd5a86700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd019be00 con 0x7f7cd0107b70 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7ccf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd019b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7ccf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd019b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38410/0 (socket says 172.21.15.116:38410) 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7ccf7fe700 1 -- 172.21.15.116:0/2484391459 learned_addr learned my addr 172.21.15.116:0/2484391459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.956+0000 7f7ccf7fe700 1 -- 172.21.15.116:0/2484391459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4009550 con 0x7f7cd0107b70 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.957+0000 7f7ccf7fe700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd019b8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4011980 tx=0x7f7cc4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.957+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4003970 con 0x7f7cd0107b70 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.957+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd019c000 con 0x7f7cd0107b70 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.957+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7cc4003ad0 con 0x7f7cd0107b70 2024-03-24T09:12:29.303 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.957+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd019c4a0 con 0x7f7cd0107b70 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.957+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cc4019610 con 0x7f7cd0107b70 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.958+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f7cc40198d0 con 0x7f7cd0107b70 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.958+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd010bfa0 con 0x7f7cd0107b70 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.958+0000 7f7cccff9700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 0x7f7cb803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.958+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7cc404ce90 con 0x7f7cd0107b70 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.958+0000 7f7cceffd700 1 -- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 msgr2=0x7f7cb803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.958+0000 7f7cceffd700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 0x7f7cb803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:27.963+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7cc4023e00 con 0x7f7cd0107b70 2024-03-24T09:12:29.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.159+0000 7f7cceffd700 1 -- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 msgr2=0x7f7cb803b3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.159+0000 7f7cceffd700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 0x7f7cb803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.245+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7cd019ed80 con 0x7f7cd0107b70 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.245+0000 7f7cccff9700 1 -- 172.21.15.116:0/2484391459 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f7cc404e070 con 0x7f7cd0107b70 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 msgr2=0x7f7cb803b3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 0x7f7cb803b3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 msgr2=0x7f7cd019b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd019b8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4011980 tx=0x7f7cc4011a60 comp rx=0 tx=0).stop 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 shutdown_connections 2024-03-24T09:12:29.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f7cb8038f30 0x7f7cb803b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 --2- 172.21.15.116:0/2484391459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cd0107b70 0x7f7cd019b8c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 >> 172.21.15.116:0/2484391459 conn(0x7f7cd0103150 msgr2=0x7f7cd0103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 shutdown_connections 2024-03-24T09:12:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:28.248+0000 7f7cd5a86700 1 -- 172.21.15.116:0/2484391459 wait complete. 2024-03-24T09:12:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-24T09:12:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 9... 2024-03-24T09:12:29.454 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:29.278+0000 7f86307de200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:12:29.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:29.463+0000 7f86307de200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:12:30.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:30.439+0000 7f86307de200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:12:30.834 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:30.569+0000 7f86307de200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:12:31.190 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:30.903+0000 7f86307de200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:12:31.190 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:30.993+0000 7f86307de200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:12:31.190 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:31.189+0000 7f86307de200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:12:31.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:31.373+0000 7f86307de200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:12:31.916 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:31.658+0000 7f86307de200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:12:31.916 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:31.915+0000 7f86307de200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:12:32.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:32 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:32.152+0000 7f86307de200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:12:33.237 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:32 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:32.948+0000 7f86307de200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:12:33.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:33.513+0000 7f86307de200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:12:33.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:33.610+0000 7f86307de200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:12:33.825+0000 7f86307de200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: Active manager daemon x restarted 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: Activating manager daemon x 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: mgrmap e10: x(active, starting, since 0.00330975s) 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:12:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T09:12:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:12:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:12:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:12:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: Manager daemon x is now available 2024-03-24T09:12:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:35.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:12:35.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T09:12:35.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:34 smithi116 ceph-mon[32207]: mgrmap e11: x(active, since 1.0066s) 2024-03-24T09:12:35.793 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-24T09:12:35.793 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.227+0000 7f6cb293c700 1 Processor -- start 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.227+0000 7f6cb293c700 1 -- start start 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.227+0000 7f6cb293c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac108030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.227+0000 7f6cb293c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac108600 con 0x7f6cac107c10 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6cabfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac108030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6cabfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac108030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38426/0 (socket says 172.21.15.116:38426) 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6cabfff700 1 -- 172.21.15.116:0/1724900139 learned_addr learned my addr 172.21.15.116:0/1724900139 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6cabfff700 1 -- 172.21.15.116:0/1724900139 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac108e10 con 0x7f6cac107c10 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6cabfff700 1 --2- 172.21.15.116:0/1724900139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac108030 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c940098a0 tx=0x7f6c94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de0a54bf7870273 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6caaffd700 1 -- 172.21.15.116:0/1724900139 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c94004160 con 0x7f6cac107c10 2024-03-24T09:12:35.794 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6caaffd700 1 -- 172.21.15.116:0/1724900139 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c940042c0 con 0x7f6cac107c10 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.228+0000 7f6caaffd700 1 -- 172.21.15.116:0/1724900139 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c940119b0 con 0x7f6cac107c10 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 -- 172.21.15.116:0/1724900139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 msgr2=0x7f6cac108030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 --2- 172.21.15.116:0/1724900139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac108030 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6c940098a0 tx=0x7f6c94009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 -- 172.21.15.116:0/1724900139 shutdown_connections 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 --2- 172.21.15.116:0/1724900139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac108030 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 -- 172.21.15.116:0/1724900139 >> 172.21.15.116:0/1724900139 conn(0x7f6cac1031f0 msgr2=0x7f6cac105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 -- 172.21.15.116:0/1724900139 shutdown_connections 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 -- 172.21.15.116:0/1724900139 wait complete. 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.229+0000 7f6cb293c700 1 Processor -- start 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cb293c700 1 -- start start 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cb293c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cb293c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac19bea0 con 0x7f6cac107c10 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cabfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cabfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac19b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38436/0 (socket says 172.21.15.116:38436) 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cabfff700 1 -- 172.21.15.116:0/3586927474 learned_addr learned my addr 172.21.15.116:0/3586927474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cabfff700 1 -- 172.21.15.116:0/3586927474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c94009550 con 0x7f6cac107c10 2024-03-24T09:12:35.795 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.230+0000 7f6cabfff700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac19b960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6c94011670 tx=0x7f6c94011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.231+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c940038b0 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.231+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac19c0a0 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.231+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c94003a10 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.231+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6c9401a630 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.231+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac19c540 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.232+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) ==== 45941+0+0 (secure 0 0 0) 0x7f6c9401a8f0 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.232+0000 7f6ca97fa700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.232+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 --> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6c9803bab0 con 0x7f6c98038ee0 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.232+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6c9404d4d0 con 0x7f6cac107c10 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.232+0000 7f6cab7fe700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 msgr2=0x7f6c9803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.232+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.433+0000 7f6cab7fe700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 msgr2=0x7f6c9803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.433+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.834+0000 7f6cab7fe700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 msgr2=0x7f6c9803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:35.796 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:30.834+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:31.635+0000 7f6cab7fe700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 msgr2=0x7f6c9803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:31.635+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:33.236+0000 7f6cab7fe700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 msgr2=0x7f6c9803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3871815735 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:33.236+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:33.841+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 10) ==== 45708+0+0 (secure 0 0 0) 0x7f6c9402d960 con 0x7f6cac107c10 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:33.841+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 msgr2=0x7f6c9803b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:33.841+0000 7f6ca97fa700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.843+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7f6c9404e760 con 0x7f6cac107c10 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.843+0000 7f6ca97fa700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c9803c960 0x7f6c9803ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.843+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6c9803bab0 con 0x7f6c9803c960 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.843+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c9803c960 0x7f6c9803ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.844+0000 7f6cab7fe700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c9803c960 0x7f6c9803ed50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c003a10 tx=0x7f6c9c0092b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.844+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f6c9803bab0 con 0x7f6c9803c960 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.851+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f6cac10c040 con 0x7f6c9803c960 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.851+0000 7f6ca97fa700 1 -- 172.21.15.116:0/3586927474 <== mgr.14148 v2:172.21.15.116:6800/2363601294 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f6cac10c040 con 0x7f6c9803c960 2024-03-24T09:12:35.797 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c9803c960 msgr2=0x7f6c9803ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c9803c960 0x7f6c9803ed50 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c003a10 tx=0x7f6c9c0092b0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 msgr2=0x7f6cac19b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac19b960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6c94011670 tx=0x7f6c94011a30 comp rx=0 tx=0).stop 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 shutdown_connections 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c9803c960 0x7f6c9803ed50 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cac107c10 0x7f6cac19b960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 --2- 172.21.15.116:0/3586927474 >> [v2:172.21.15.116:6800/3871815735,v1:172.21.15.116:6801/3871815735] conn(0x7f6c98038ee0 0x7f6c9803b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 >> 172.21.15.116:0/3586927474 conn(0x7f6cac1031f0 msgr2=0x7f6cac103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 shutdown_connections 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:34.852+0000 7f6cb293c700 1 -- 172.21.15.116:0/3586927474 wait complete. 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 9 is available 2024-03-24T09:12:35.798 INFO:teuthology.orchestra.run.smithi116.stdout:Generating a dashboard self-signed certificate... 2024-03-24T09:12:36.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:34] ENGINE Bus STARTING 2024-03-24T09:12:36.170 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:34] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T09:12:36.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:34] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T09:12:36.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: [24/Mar/2024:09:12:34] ENGINE Bus STARTED 2024-03-24T09:12:36.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T09:12:36.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T09:12:36.171 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:37.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T09:12:37.099 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c33ae700 1 Processor -- start 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c33ae700 1 -- start start 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c33ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc108590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c33ae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93bc108b60 con 0x7f93bc108170 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c114a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc108590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c114a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc108590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38512/0 (socket says 172.21.15.116:38512) 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.268+0000 7f93c114a700 1 -- 172.21.15.116:0/1185543510 learned_addr learned my addr 172.21.15.116:0/1185543510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c114a700 1 -- 172.21.15.116:0/1185543510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93bc106c80 con 0x7f93bc108170 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c114a700 1 --2- 172.21.15.116:0/1185543510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc108590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f93b80098a0 tx=0x7f93b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2f55b9f2906a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93b3fff700 1 -- 172.21.15.116:0/1185543510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b8004160 con 0x7f93bc108170 2024-03-24T09:12:37.100 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93b3fff700 1 -- 172.21.15.116:0/1185543510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b80042c0 con 0x7f93bc108170 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93b3fff700 1 -- 172.21.15.116:0/1185543510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b80119b0 con 0x7f93bc108170 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c33ae700 1 -- 172.21.15.116:0/1185543510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 msgr2=0x7f93bc108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c33ae700 1 --2- 172.21.15.116:0/1185543510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc108590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f93b80098a0 tx=0x7f93b8009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c33ae700 1 -- 172.21.15.116:0/1185543510 shutdown_connections 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c33ae700 1 --2- 172.21.15.116:0/1185543510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc108590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.269+0000 7f93c33ae700 1 -- 172.21.15.116:0/1185543510 >> 172.21.15.116:0/1185543510 conn(0x7f93bc1017f0 msgr2=0x7f93bc103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c33ae700 1 -- 172.21.15.116:0/1185543510 shutdown_connections 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c33ae700 1 -- 172.21.15.116:0/1185543510 wait complete. 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c33ae700 1 Processor -- start 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c33ae700 1 -- start start 2024-03-24T09:12:37.101 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c33ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c33ae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93bc19bf80 con 0x7f93bc108170 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c114a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c114a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc19ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38528/0 (socket says 172.21.15.116:38528) 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.270+0000 7f93c114a700 1 -- 172.21.15.116:0/1846237395 learned_addr learned my addr 172.21.15.116:0/1846237395 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93c114a700 1 -- 172.21.15.116:0/1846237395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b8009550 con 0x7f93bc108170 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93c114a700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc19ba40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f93b8011670 tx=0x7f93b8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b8003a40 con 0x7f93bc108170 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93b8003ba0 con 0x7f93bc108170 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93bc19c180 con 0x7f93bc108170 2024-03-24T09:12:37.102 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f93b8019730 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.271+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93bc19c620 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.272+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 11) ==== 45835+0+0 (secure 0 0 0) 0x7f93b80199f0 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.272+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93bc195b30 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.272+0000 7f93b27fc700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93a8038dd0 0x7f93a803b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.272+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f93b804d470 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.272+0000 7f93c0949700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93a8038dd0 0x7f93a803b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.272+0000 7f93c0949700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93a8038dd0 0x7f93a803b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f93ac006fd0 tx=0x7f93ac006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.277+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f93b8018720 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.459+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f93b801abe0 con 0x7f93bc108170 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.486+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f93bc02dc30 con 0x7f93a8038dd0 2024-03-24T09:12:37.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.567+0000 7f93b27fc700 1 -- 172.21.15.116:0/1846237395 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f93bc02dc30 con 0x7f93a8038dd0 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93a8038dd0 msgr2=0x7f93a803b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93a8038dd0 0x7f93a803b290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f93ac006fd0 tx=0x7f93ac006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 msgr2=0x7f93bc19ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc19ba40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f93b8011670 tx=0x7f93b8011a30 comp rx=0 tx=0).stop 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 shutdown_connections 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93a8038dd0 0x7f93a803b290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 --2- 172.21.15.116:0/1846237395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93bc108170 0x7f93bc19ba40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 >> 172.21.15.116:0/1846237395 conn(0x7f93bc1017f0 msgr2=0x7f93bc1026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 shutdown_connections 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:36.569+0000 7f93c33ae700 1 -- 172.21.15.116:0/1846237395 wait complete. 2024-03-24T09:12:37.104 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial admin user... 2024-03-24T09:12:37.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:37 smithi116 ceph-mon[32207]: mgrmap e12: x(active, since 2s) 2024-03-24T09:12:37.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:37 smithi116 ceph-mon[32207]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:37.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:37 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:37.752 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:37 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$p2gQMMwUDTGO07iHJgtgV.LwvJGO8IYkKNibm7oowXBDKWzZ0fVjG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711271557, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb569e6e700 1 Processor -- start 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb569e6e700 1 -- start start 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb569e6e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 0x7fb564109660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb569e6e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb564109c30 con 0x7fb564109240 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb5637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 0x7fb564109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:38.557 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb5637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 0x7fb564109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51024/0 (socket says 172.21.15.116:51024) 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb5637fe700 1 -- 172.21.15.116:0/82868530 learned_addr learned my addr 172.21.15.116:0/82868530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb5637fe700 1 -- 172.21.15.116:0/82868530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb564107780 con 0x7fb564109240 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.514+0000 7fb5637fe700 1 --2- 172.21.15.116:0/82868530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 0x7fb564109660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb5540098a0 tx=0x7fb554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=272030c0484102ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb5627fc700 1 -- 172.21.15.116:0/82868530 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb554004160 con 0x7fb564109240 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb5627fc700 1 -- 172.21.15.116:0/82868530 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5540042c0 con 0x7fb564109240 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb5627fc700 1 -- 172.21.15.116:0/82868530 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5540119b0 con 0x7fb564109240 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 -- 172.21.15.116:0/82868530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 msgr2=0x7fb564109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:38.558 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 --2- 172.21.15.116:0/82868530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 0x7fb564109660 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb5540098a0 tx=0x7fb554009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 -- 172.21.15.116:0/82868530 shutdown_connections 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 --2- 172.21.15.116:0/82868530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb564109240 0x7fb564109660 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 -- 172.21.15.116:0/82868530 >> 172.21.15.116:0/82868530 conn(0x7fb564103100 msgr2=0x7fb564105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 -- 172.21.15.116:0/82868530 shutdown_connections 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.515+0000 7fb569e6e700 1 -- 172.21.15.116:0/82868530 wait complete. 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb569e6e700 1 Processor -- start 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb569e6e700 1 -- start start 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb569e6e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 0x7fb56407c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb569e6e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb56407df10 con 0x7fb56407daf0 2024-03-24T09:12:38.559 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb5637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 0x7fb56407c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb5637fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 0x7fb56407c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51034/0 (socket says 172.21.15.116:51034) 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb5637fe700 1 -- 172.21.15.116:0/1668265895 learned_addr learned my addr 172.21.15.116:0/1668265895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.516+0000 7fb5637fe700 1 -- 172.21.15.116:0/1668265895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb554009550 con 0x7fb56407daf0 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.517+0000 7fb5637fe700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 0x7fb56407c260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb554000c00 tx=0x7fb554012f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.517+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb5540037d0 con 0x7fb56407daf0 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.517+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb55401f070 con 0x7fb56407daf0 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.517+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56407e110 con 0x7fb56407daf0 2024-03-24T09:12:38.560 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.517+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb55401ae20 con 0x7fb56407daf0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.517+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56407c9f0 con 0x7fb56407daf0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.518+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7fb55401b440 con 0x7fb56407daf0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.518+0000 7fb560ff9700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb54c038f30 0x7fb54c03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.518+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56410bf70 con 0x7fb56407daf0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.518+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb554021070 con 0x7fb56407daf0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.518+0000 7fb562ffd700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb54c038f30 0x7fb54c03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.519+0000 7fb562ffd700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb54c038f30 0x7fb54c03b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb558006fd0 tx=0x7fb558006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:38.561 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.523+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb554018940 con 0x7fb56407daf0 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.752+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fb564063650 con 0x7fb54c038f30 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.978+0000 7fb560ff9700 1 -- 172.21.15.116:0/1668265895 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fb564063650 con 0x7fb54c038f30 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.980+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb54c038f30 msgr2=0x7fb54c03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.980+0000 7fb569e6e700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb54c038f30 0x7fb54c03b3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb558006fd0 tx=0x7fb558006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.980+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 msgr2=0x7fb56407c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.980+0000 7fb569e6e700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 0x7fb56407c260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb554000c00 tx=0x7fb554012f60 comp rx=0 tx=0).stop 2024-03-24T09:12:38.562 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.980+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 shutdown_connections 2024-03-24T09:12:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.981+0000 7fb569e6e700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb54c038f30 0x7fb54c03b3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.981+0000 7fb569e6e700 1 --2- 172.21.15.116:0/1668265895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56407daf0 0x7fb56407c260 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.981+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 >> 172.21.15.116:0/1668265895 conn(0x7fb564103100 msgr2=0x7fb564103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.981+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 shutdown_connections 2024-03-24T09:12:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:37.981+0000 7fb569e6e700 1 -- 172.21.15.116:0/1668265895 wait complete. 2024-03-24T09:12:38.563 INFO:teuthology.orchestra.run.smithi116.stdout:Fetching dashboard port number... 2024-03-24T09:12:38.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:38.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:38 smithi116 ceph-mon[32207]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:38.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:39.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:39 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/604787397' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T09:12:40.612 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T09:12:40.612 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.947+0000 7f2187b05700 1 Processor -- start 2024-03-24T09:12:40.612 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f2187b05700 1 -- start start 2024-03-24T09:12:40.612 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f2187b05700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f2180106250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f2187b05700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2180106820 con 0x7f2180105e30 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f21858a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f2180106250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f21858a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f2180106250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51048/0 (socket says 172.21.15.116:51048) 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f21858a1700 1 -- 172.21.15.116:0/1009146276 learned_addr learned my addr 172.21.15.116:0/1009146276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.948+0000 7f21858a1700 1 -- 172.21.15.116:0/1009146276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2180106960 con 0x7f2180105e30 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.949+0000 7f21858a1700 1 --2- 172.21.15.116:0/1009146276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f2180106250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f217c0098a0 tx=0x7f217c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c413e010d96a8fe2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.949+0000 7f218489f700 1 -- 172.21.15.116:0/1009146276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f217c004160 con 0x7f2180105e30 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.949+0000 7f218489f700 1 -- 172.21.15.116:0/1009146276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f217c0042c0 con 0x7f2180105e30 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.949+0000 7f218489f700 1 -- 172.21.15.116:0/1009146276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f217c0119b0 con 0x7f2180105e30 2024-03-24T09:12:40.613 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 -- 172.21.15.116:0/1009146276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 msgr2=0x7f2180106250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 --2- 172.21.15.116:0/1009146276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f2180106250 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f217c0098a0 tx=0x7f217c009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 -- 172.21.15.116:0/1009146276 shutdown_connections 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 --2- 172.21.15.116:0/1009146276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f2180106250 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 -- 172.21.15.116:0/1009146276 >> 172.21.15.116:0/1009146276 conn(0x7f21801017d0 msgr2=0x7f2180103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 -- 172.21.15.116:0/1009146276 shutdown_connections 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.950+0000 7f2187b05700 1 -- 172.21.15.116:0/1009146276 wait complete. 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f2187b05700 1 Processor -- start 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f2187b05700 1 -- start start 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f2187b05700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f218019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f2187b05700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f218019bda0 con 0x7f2180105e30 2024-03-24T09:12:40.614 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f21858a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f218019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f21858a1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f218019b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51056/0 (socket says 172.21.15.116:51056) 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.951+0000 7f21858a1700 1 -- 172.21.15.116:0/604787397 learned_addr learned my addr 172.21.15.116:0/604787397 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.952+0000 7f21858a1700 1 -- 172.21.15.116:0/604787397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c009550 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.952+0000 7f21858a1700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f218019b860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f217c011980 tx=0x7f217c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.952+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f217c003970 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.952+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f217c003ad0 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.952+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f217c019610 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.952+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f218019c000 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.953+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f218019ed20 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.954+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2180063830 con 0x7f2180105e30 2024-03-24T09:12:40.615 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.954+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) ==== 45941+0+0 (secure 0 0 0) 0x7f217c01a430 con 0x7f2180105e30 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.954+0000 7f2176ffd700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f216c038f30 0x7f216c03b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.954+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f217c04d1f0 con 0x7f2180105e30 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.954+0000 7f21850a0700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f216c038f30 0x7f216c03b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.955+0000 7f21850a0700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f216c038f30 0x7f216c03b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2170006fd0 tx=0x7f2170006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:38.964+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f217c019df0 con 0x7f2180105e30 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.171+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f218019f0d0 con 0x7f2180105e30 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.172+0000 7f2176ffd700 1 -- 172.21.15.116:0/604787397 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f217c018350 con 0x7f2180105e30 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f216c038f30 msgr2=0x7f216c03b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f216c038f30 0x7f216c03b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2170006fd0 tx=0x7f2170006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 msgr2=0x7f218019b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:40.616 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f218019b860 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f217c011980 tx=0x7f217c011a60 comp rx=0 tx=0).stop 2024-03-24T09:12:40.617 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 shutdown_connections 2024-03-24T09:12:40.617 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f216c038f30 0x7f216c03b3f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:40.617 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 --2- 172.21.15.116:0/604787397 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2180105e30 0x7f218019b860 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:40.617 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 >> 172.21.15.116:0/604787397 conn(0x7f21801017d0 msgr2=0x7f21801022e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:40.617 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 shutdown_connections 2024-03-24T09:12:40.617 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:39.174+0000 7f2187b05700 1 -- 172.21.15.116:0/604787397 wait complete. 2024-03-24T09:12:40.620 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T09:12:40.621 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-24T09:12:40.628 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T09:12:40.628 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-24T09:12:40.628 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-24T09:12:40.629 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T09:12:40.630 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph Dashboard is now available at: 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: URL: https://smithi116.front.sepia.ceph.com:8443/ 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: User: admin 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: Password: gl6xxals3h 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout:Saving cluster configuration to /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config directory 2024-03-24T09:12:40.631 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling autotune for osd_memory_target 2024-03-24T09:12:41.865 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:41 smithi116 ceph-mon[32207]: mgrmap e13: x(active, since 6s) 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.087+0000 7f820f637700 1 Processor -- start 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820f637700 1 -- start start 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820f637700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f82081096a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820f637700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8208109c70 con 0x7f8208109280 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820d3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f82081096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820d3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f82081096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51060/0 (socket says 172.21.15.116:51060) 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820d3d3700 1 -- 172.21.15.116:0/946096339 learned_addr learned my addr 172.21.15.116:0/946096339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:41.873 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.088+0000 7f820d3d3700 1 -- 172.21.15.116:0/946096339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82081077c0 con 0x7f8208109280 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f820d3d3700 1 --2- 172.21.15.116:0/946096339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f82081096a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f81f80098a0 tx=0x7f81f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7d757ced66b4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f81fffff700 1 -- 172.21.15.116:0/946096339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81f8004160 con 0x7f8208109280 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f81fffff700 1 -- 172.21.15.116:0/946096339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81f80042c0 con 0x7f8208109280 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f81fffff700 1 -- 172.21.15.116:0/946096339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81f80119b0 con 0x7f8208109280 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f820f637700 1 -- 172.21.15.116:0/946096339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 msgr2=0x7f82081096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f820f637700 1 --2- 172.21.15.116:0/946096339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f82081096a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f81f80098a0 tx=0x7f81f8009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f820f637700 1 -- 172.21.15.116:0/946096339 shutdown_connections 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f820f637700 1 --2- 172.21.15.116:0/946096339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f82081096a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.089+0000 7f820f637700 1 -- 172.21.15.116:0/946096339 >> 172.21.15.116:0/946096339 conn(0x7f8208103140 msgr2=0x7f8208105580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820f637700 1 -- 172.21.15.116:0/946096339 shutdown_connections 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820f637700 1 -- 172.21.15.116:0/946096339 wait complete. 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820f637700 1 Processor -- start 2024-03-24T09:12:41.874 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820f637700 1 -- start start 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820f637700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f820819b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820f637700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f820819bea0 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820d3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f820819b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820d3d3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f820819b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51064/0 (socket says 172.21.15.116:51064) 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.090+0000 7f820d3d3700 1 -- 172.21.15.116:0/648606768 learned_addr learned my addr 172.21.15.116:0/648606768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f820d3d3700 1 -- 172.21.15.116:0/648606768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f8009550 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f820d3d3700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f820819b960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81f8011670 tx=0x7f81f8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81f8003970 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81f8003ad0 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820819c0a0 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f81f80196b0 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.091+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820819c540 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.092+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7f81f801a430 con 0x7f8208109280 2024-03-24T09:12:41.875 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.092+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f820810bfd0 con 0x7f8208109280 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.092+0000 7f81fe7fc700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81f4038f80 0x7f81f403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.092+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f81f804d4a0 con 0x7f8208109280 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.092+0000 7f820cbd2700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81f4038f80 0x7f81f403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.093+0000 7f820cbd2700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81f4038f80 0x7f81f403b440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8204006fd0 tx=0x7f8204006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.097+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f81f8018530 con 0x7f8208109280 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.294+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f820810c1e0 con 0x7f8208109280 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.294+0000 7f81fe7fc700 1 -- 172.21.15.116:0/648606768 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f81f8018350 con 0x7f8208109280 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81f4038f80 msgr2=0x7f81f403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81f4038f80 0x7f81f403b440 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8204006fd0 tx=0x7f8204006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 msgr2=0x7f820819b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f820819b960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f81f8011670 tx=0x7f81f8011aa0 comp rx=0 tx=0).stop 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 shutdown_connections 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81f4038f80 0x7f81f403b440 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:41.876 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 --2- 172.21.15.116:0/648606768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208109280 0x7f820819b960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:41.877 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 >> 172.21.15.116:0/648606768 conn(0x7f8208103140 msgr2=0x7f8208103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:41.877 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 shutdown_connections 2024-03-24T09:12:41.877 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:41.297+0000 7f820f637700 1 -- 172.21.15.116:0/648606768 wait complete. 2024-03-24T09:12:43.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e9f62700 1 Processor -- start 2024-03-24T09:12:43.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e9f62700 1 -- start start 2024-03-24T09:12:43.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e9f62700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:43.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e9f62700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e407bf30 con 0x7ff4e4108650 2024-03-24T09:12:43.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51066/0 (socket says 172.21.15.116:51066) 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e37fe700 1 -- 172.21.15.116:0/33009345 learned_addr learned my addr 172.21.15.116:0/33009345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e37fe700 1 -- 172.21.15.116:0/33009345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e410af80 con 0x7ff4e4108650 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.296+0000 7ff4e37fe700 1 --2- 172.21.15.116:0/33009345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e410aa40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0098a0 tx=0x7ff4cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1614d63e8ba841de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e27fc700 1 -- 172.21.15.116:0/33009345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4cc004160 con 0x7ff4e4108650 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e27fc700 1 -- 172.21.15.116:0/33009345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4cc0042c0 con 0x7ff4e4108650 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e27fc700 1 -- 172.21.15.116:0/33009345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4cc0119b0 con 0x7ff4e4108650 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 -- 172.21.15.116:0/33009345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 msgr2=0x7ff4e410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:43.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 --2- 172.21.15.116:0/33009345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e410aa40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0098a0 tx=0x7ff4cc009bb0 comp rx=0 tx=0).stop 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 -- 172.21.15.116:0/33009345 shutdown_connections 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 --2- 172.21.15.116:0/33009345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e410aa40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 -- 172.21.15.116:0/33009345 >> 172.21.15.116:0/33009345 conn(0x7ff4e4102020 msgr2=0x7ff4e4104480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 -- 172.21.15.116:0/33009345 shutdown_connections 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.297+0000 7ff4e9f62700 1 -- 172.21.15.116:0/33009345 wait complete. 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e9f62700 1 Processor -- start 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e9f62700 1 -- start start 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e9f62700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e9f62700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e419bf40 con 0x7ff4e4108650 2024-03-24T09:12:43.175 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e419ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51082/0 (socket says 172.21.15.116:51082) 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.298+0000 7ff4e37fe700 1 -- 172.21.15.116:0/2642564701 learned_addr learned my addr 172.21.15.116:0/2642564701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e37fe700 1 -- 172.21.15.116:0/2642564701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc009550 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e37fe700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e419ba00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc011670 tx=0x7ff4cc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4cc0038b0 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4cc003a10 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e419c140 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4cc019550 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.299+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e419c5e0 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.300+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7ff4cc0197c0 con 0x7ff4e4108650 2024-03-24T09:12:43.176 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.300+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e410be80 con 0x7ff4e4108650 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.300+0000 7ff4e0ff9700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff4d0038fd0 0x7ff4d003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.300+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff4cc04d560 con 0x7ff4e4108650 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.300+0000 7ff4e2ffd700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff4d0038fd0 0x7ff4d003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.301+0000 7ff4e2ffd700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff4d0038fd0 0x7ff4d003b490 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4006fd0 tx=0x7ff4d4006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.305+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff4cc018a40 con 0x7ff4e4108650 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.604+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7ff4e410c090 con 0x7ff4e4108650 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.607+0000 7ff4e0ff9700 1 -- 172.21.15.116:0/2642564701 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7ff4cc018350 con 0x7ff4e4108650 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff4d0038fd0 msgr2=0x7ff4d003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:43.177 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff4d0038fd0 0x7ff4d003b490 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff4d4006fd0 tx=0x7ff4d4006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 msgr2=0x7ff4e419ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e419ba00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc011670 tx=0x7ff4cc011a30 comp rx=0 tx=0).stop 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 shutdown_connections 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff4d0038fd0 0x7ff4d003b490 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 --2- 172.21.15.116:0/2642564701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e4108650 0x7ff4e419ba00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 >> 172.21.15.116:0/2642564701 conn(0x7ff4e4102020 msgr2=0x7ff4e4102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 shutdown_connections 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-24T09:12:42.609+0000 7ff4e9f62700 1 -- 172.21.15.116:0/2642564701 wait complete. 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T09:12:43.178 INFO:teuthology.orchestra.run.smithi116.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: ceph telemetry on 2024-03-24T09:12:43.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.180 INFO:teuthology.orchestra.run.smithi116.stdout:For more information see: 2024-03-24T09:12:43.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.180 INFO:teuthology.orchestra.run.smithi116.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T09:12:43.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:43.180 INFO:teuthology.orchestra.run.smithi116.stdout:Bootstrap complete. 2024-03-24T09:12:43.210 INFO:tasks.cephadm:Fetching config... 2024-03-24T09:12:43.210 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:12:43.210 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T09:12:43.227 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T09:12:43.227 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:12:43.227 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T09:12:43.282 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T09:12:43.283 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:12:43.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T09:12:43.351 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T09:12:43.351 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:12:43.351 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T09:12:43.406 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T09:12:43.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T09:12:43.488 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:12:43.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T09:12:43.548 INFO:teuthology.orchestra.run.smithi117.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:12:43.563 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T09:12:43.610 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBeHChxz3/fomVLFXALOeG54Oy8I4HkrEQq8jkGJbkrqowwM6/MjL5spJNcDCI4Tg3YqRRjCJ3nTZQCJDknAZK5k/uDhOS0bDQeSoXdRNpFhiGJ0PqJDIMmzJg/n1+IDUUKxaOYNFpye6LVEO9F2d95yAuWMBUFdB6AWifOJf0ROpVZyjUaWW67o8RZI6gIelzM+jppZGGVfC2VIliTM/hoLg3DvDoj//EprVMBpMi2ifmsN1u0yB9eDd+vEff2+tfYYwFSwAQ6c5APWBywpc27EDKbu1y/336ozZsjecDZ2jyOVbEGV9yONd3T5o3UPePMIaUpZexqt7Y/sUGktT4bDj9YkU1SGwnh7mnGxRDlvPQcH6ZbQkVWEEe3MwmtEcglLjENVPaCt8ce7sx6p+1hjwjTn9zVMD6qI896dNybP5hrYTAxlWm65h9fXbgdUHIFqwZ4MaM92X6FrFljLxaeMvDdbz7aLwfibnWJz7+CrOGHearzfRnTow2TuVZHk= ceph-69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:12:43.625 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T09:12:43.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:43 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2642564701' entity='client.admin' 2024-03-24T09:12:43.864 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:12:46.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:46.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:46.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:12:46.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:46.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:46.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:46.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 -- 172.21.15.116:0/3391967010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 msgr2=0x7fd75c103690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:46.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 --2- 172.21.15.116:0/3391967010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c103690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd75400b0a0 tx=0x7fd75400b3b0 comp rx=0 tx=0).stop 2024-03-24T09:12:46.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 -- 172.21.15.116:0/3391967010 shutdown_connections 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 --2- 172.21.15.116:0/3391967010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c103690 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 -- 172.21.15.116:0/3391967010 >> 172.21.15.116:0/3391967010 conn(0x7fd75c0fb8b0 msgr2=0x7fd75c0fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 -- 172.21.15.116:0/3391967010 shutdown_connections 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.387+0000 7fd761aba700 1 -- 172.21.15.116:0/3391967010 wait complete. 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd761aba700 1 Processor -- start 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd761aba700 1 -- start start 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd761aba700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:46.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd761aba700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd75c1975a0 con 0x7fd75c103270 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd75b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd75b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c197060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51100/0 (socket says 172.21.15.116:51100) 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd75b7fe700 1 -- 172.21.15.116:0/2955556700 learned_addr learned my addr 172.21.15.116:0/2955556700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd75b7fe700 1 -- 172.21.15.116:0/2955556700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd754009d00 con 0x7fd75c103270 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.388+0000 7fd75b7fe700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c197060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7540062a0 tx=0x7fd75400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.389+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd754021070 con 0x7fd75c103270 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.389+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd75c1977a0 con 0x7fd75c103270 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.389+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd754003d40 con 0x7fd75c103270 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.389+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd75400e040 con 0x7fd75c103270 2024-03-24T09:12:46.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.389+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd75c197c40 con 0x7fd75c103270 2024-03-24T09:12:46.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.390+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7fd754019070 con 0x7fd75c103270 2024-03-24T09:12:46.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.390+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75c190930 con 0x7fd75c103270 2024-03-24T09:12:46.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.390+0000 7fd758ff9700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038f80 0x7fd74403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:46.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.390+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd75404ca90 con 0x7fd75c103270 2024-03-24T09:12:46.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.390+0000 7fd75affd700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038f80 0x7fd74403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:46.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.390+0000 7fd75affd700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038f80 0x7fd74403b440 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd750006fd0 tx=0x7fd750006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:46.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.395+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd754025960 con 0x7fd75c103270 2024-03-24T09:12:46.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.597+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fd75c063830 con 0x7fd75c103270 2024-03-24T09:12:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.600+0000 7fd758ff9700 1 -- 172.21.15.116:0/2955556700 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fd754017030 con 0x7fd75c103270 2024-03-24T09:12:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038f80 msgr2=0x7fd74403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038f80 0x7fd74403b440 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd750006fd0 tx=0x7fd750006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 msgr2=0x7fd75c197060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c197060 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd7540062a0 tx=0x7fd75400bd40 comp rx=0 tx=0).stop 2024-03-24T09:12:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 shutdown_connections 2024-03-24T09:12:46.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038f80 0x7fd74403b440 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 --2- 172.21.15.116:0/2955556700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd75c103270 0x7fd75c197060 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 >> 172.21.15.116:0/2955556700 conn(0x7fd75c0fb8b0 msgr2=0x7fd75c0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 shutdown_connections 2024-03-24T09:12:46.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:46.603+0000 7fd761aba700 1 -- 172.21.15.116:0/2955556700 wait complete. 2024-03-24T09:12:47.348 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T09:12:47.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T09:12:47.817 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:12:47.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:47 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2955556700' entity='client.admin' 2024-03-24T09:12:47.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:47.933 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 -- 172.21.15.116:0/1672221309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 msgr2=0x7fd758102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 --2- 172.21.15.116:0/1672221309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758102050 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd754009910 tx=0x7fd754009c20 comp rx=0 tx=0).stop 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 -- 172.21.15.116:0/1672221309 shutdown_connections 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 --2- 172.21.15.116:0/1672221309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758102050 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 -- 172.21.15.116:0/1672221309 >> 172.21.15.116:0/1672221309 conn(0x7fd7580fd1f0 msgr2=0x7fd7580ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 -- 172.21.15.116:0/1672221309 shutdown_connections 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.153+0000 7fd75f476700 1 -- 172.21.15.116:0/1672221309 wait complete. 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75f476700 1 Processor -- start 2024-03-24T09:12:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75f476700 1 -- start start 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75f476700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758075ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75f476700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd758076400 con 0x7fd758101c30 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75d212700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758075ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75d212700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758075ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44942/0 (socket says 172.21.15.116:44942) 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75d212700 1 -- 172.21.15.116:0/313925241 learned_addr learned my addr 172.21.15.116:0/313925241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.154+0000 7fd75d212700 1 -- 172.21.15.116:0/313925241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7540095f0 con 0x7fd758101c30 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.155+0000 7fd75d212700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758075ec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd754005950 tx=0x7fd7540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.155+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd75401d070 con 0x7fd758101c30 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.155+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd75400bc00 con 0x7fd758101c30 2024-03-24T09:12:49.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.155+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7580745f0 con 0x7fd758101c30 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.155+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7540039e0 con 0x7fd758101c30 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.155+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd758074a90 con 0x7fd758101c30 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.156+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7fd754003c50 con 0x7fd758101c30 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.156+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd758108270 con 0x7fd758101c30 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.156+0000 7fd74e7fc700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038fe0 0x7fd74403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.156+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd75404cdb0 con 0x7fd758101c30 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.156+0000 7fd75ca11700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038fe0 0x7fd74403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:49.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.156+0000 7fd75ca11700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038fe0 0x7fd74403b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd748006fd0 tx=0x7fd748006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:49.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.161+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd75402ac50 con 0x7fd758101c30 2024-03-24T09:12:49.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.357+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7fd758190390 con 0x7fd744038fe0 2024-03-24T09:12:49.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.361+0000 7fd74e7fc700 1 -- 172.21.15.116:0/313925241 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd758190390 con 0x7fd744038fe0 2024-03-24T09:12:49.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.362+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038fe0 msgr2=0x7fd74403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.362+0000 7fd75f476700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038fe0 0x7fd74403b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd748006fd0 tx=0x7fd748006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.362+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 msgr2=0x7fd758075ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.362+0000 7fd75f476700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758075ec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd754005950 tx=0x7fd7540057f0 comp rx=0 tx=0).stop 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.363+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 shutdown_connections 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.363+0000 7fd75f476700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd744038fe0 0x7fd74403b4a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.363+0000 7fd75f476700 1 --2- 172.21.15.116:0/313925241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd758101c30 0x7fd758075ec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.363+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 >> 172.21.15.116:0/313925241 conn(0x7fd7580fd1f0 msgr2=0x7fd7580fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.363+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 shutdown_connections 2024-03-24T09:12:49.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:49.363+0000 7fd75f476700 1 -- 172.21.15.116:0/313925241 wait complete. 2024-03-24T09:12:50.220 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi117 2024-03-24T09:12:50.221 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:12:50.221 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.conf 2024-03-24T09:12:50.240 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:12:50.240 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:12:50.297 INFO:tasks.cephadm:Adding host smithi117 to orchestrator... 2024-03-24T09:12:50.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch host add smithi117 2024-03-24T09:12:50.530 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:12:50.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:50.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:50.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:51.738 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:51 smithi116 ceph-mon[32207]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 -- 172.21.15.116:0/4292604574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 msgr2=0x7f3f98102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 --2- 172.21.15.116:0/4292604574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f98102200 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009910 tx=0x7f3f8c009c20 comp rx=0 tx=0).stop 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 -- 172.21.15.116:0/4292604574 shutdown_connections 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 --2- 172.21.15.116:0/4292604574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f98102200 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 -- 172.21.15.116:0/4292604574 >> 172.21.15.116:0/4292604574 conn(0x7f3f980fd6d0 msgr2=0x7f3f980ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 -- 172.21.15.116:0/4292604574 shutdown_connections 2024-03-24T09:12:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.084+0000 7f3f9d349700 1 -- 172.21.15.116:0/4292604574 wait complete. 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f9d349700 1 Processor -- start 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f9d349700 1 -- start start 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f9d349700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f981974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f9d349700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f98197a30 con 0x7f3f98101e20 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f981974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f96ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f981974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44952/0 (socket says 172.21.15.116:44952) 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f96ffd700 1 -- 172.21.15.116:0/2465957135 learned_addr learned my addr 172.21.15.116:0/2465957135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.085+0000 7f3f96ffd700 1 -- 172.21.15.116:0/2465957135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c0095f0 con 0x7f3f98101e20 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.086+0000 7f3f96ffd700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f981974f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c004f40 tx=0x7f3f8c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.086+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c01d070 con 0x7f3f98101e20 2024-03-24T09:12:52.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.086+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f98197c30 con 0x7f3f98101e20 2024-03-24T09:12:52.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.086+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3f8c00b750 con 0x7f3f98101e20 2024-03-24T09:12:52.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.086+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f8c0037d0 con 0x7f3f98101e20 2024-03-24T09:12:52.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.086+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f98198050 con 0x7f3f98101e20 2024-03-24T09:12:52.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.087+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7f3f8c003a90 con 0x7f3f98101e20 2024-03-24T09:12:52.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.087+0000 7f3f7ffff700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3f80038f80 0x7f3f8003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:52.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.087+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3f8c04cde0 con 0x7f3f98101e20 2024-03-24T09:12:52.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.087+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f98063830 con 0x7f3f98101e20 2024-03-24T09:12:52.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.087+0000 7f3f967fc700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3f80038f80 0x7f3f8003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:52.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.087+0000 7f3f967fc700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3f80038f80 0x7f3f8003b440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006fd0 tx=0x7f3f88006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:52.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.092+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3f8c026080 con 0x7f3f98101e20 2024-03-24T09:12:52.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:52.287+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}) -- 0x7f3f9810bf00 con 0x7f3f80038f80 2024-03-24T09:12:53.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:53 smithi116 ceph-mon[32207]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:12:55.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:54 smithi116 ceph-mon[32207]: Deploying cephadm binary to smithi117 2024-03-24T09:12:55.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:54 smithi116 ceph-mon[32207]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:12:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.594+0000 7f3f7ffff700 1 -- 172.21.15.116:0/2465957135 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3f9810bf00 con 0x7f3f80038f80 2024-03-24T09:12:56.595 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi117' with addr '172.21.15.117' 2024-03-24T09:12:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3f80038f80 msgr2=0x7f3f8003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3f80038f80 0x7f3f8003b440 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f88006fd0 tx=0x7f3f88006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 msgr2=0x7f3f981974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f981974f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c004f40 tx=0x7f3f8c005020 comp rx=0 tx=0).stop 2024-03-24T09:12:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 shutdown_connections 2024-03-24T09:12:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3f80038f80 0x7f3f8003b440 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 --2- 172.21.15.116:0/2465957135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f98101e20 0x7f3f981974f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.596+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 >> 172.21.15.116:0/2465957135 conn(0x7f3f980fd6d0 msgr2=0x7f3f981037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.597+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 shutdown_connections 2024-03-24T09:12:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:56.597+0000 7f3f9d349700 1 -- 172.21.15.116:0/2465957135 wait complete. 2024-03-24T09:12:57.218 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:56 smithi116 ceph-mon[32207]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:12:57.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:57.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:12:57.252 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T09:12:57.488 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:12:57.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:57 smithi116 ceph-mon[32207]: Added host smithi117 2024-03-24T09:12:57.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:57 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:57.955 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:57 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:12:58.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:12:58 smithi116 ceph-mon[32207]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:12:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.990+0000 7ff45a4aa700 1 -- 172.21.15.116:0/3885506882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 msgr2=0x7ff454103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.990+0000 7ff45a4aa700 1 --2- 172.21.15.116:0/3885506882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff454103220 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff43c009910 tx=0x7ff43c009c20 comp rx=0 tx=0).stop 2024-03-24T09:12:58.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.990+0000 7ff45a4aa700 1 -- 172.21.15.116:0/3885506882 shutdown_connections 2024-03-24T09:12:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 --2- 172.21.15.116:0/3885506882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff454103220 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 -- 172.21.15.116:0/3885506882 >> 172.21.15.116:0/3885506882 conn(0x7ff4540fc5f0 msgr2=0x7ff4540fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 -- 172.21.15.116:0/3885506882 shutdown_connections 2024-03-24T09:12:58.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 -- 172.21.15.116:0/3885506882 wait complete. 2024-03-24T09:12:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 Processor -- start 2024-03-24T09:12:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 -- start start 2024-03-24T09:12:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff4541974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.991+0000 7ff45a4aa700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff454197a30 con 0x7ff454102e40 2024-03-24T09:12:58.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff453fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff4541974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff453fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff4541974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51864/0 (socket says 172.21.15.116:51864) 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff453fff700 1 -- 172.21.15.116:0/1903002175 learned_addr learned my addr 172.21.15.116:0/1903002175 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff453fff700 1 -- 172.21.15.116:0/1903002175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c0095f0 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff453fff700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff4541974f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff43c004f40 tx=0x7ff43c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff43c01d070 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff454197c30 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff43c00bc00 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff43c003920 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.992+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff454198050 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.993+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7ff43c003b90 con 0x7ff454102e40 2024-03-24T09:12:58.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.993+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff454063830 con 0x7ff454102e40 2024-03-24T09:12:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.993+0000 7ff4517fa700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff440038fe0 0x7ff44003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:12:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.993+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff43c04ceb0 con 0x7ff454102e40 2024-03-24T09:12:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.993+0000 7ff4537fe700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff440038fe0 0x7ff44003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:12:58.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:58.994+0000 7ff4537fe700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff440038fe0 0x7ff44003b4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff444006fd0 tx=0x7ff444006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:12:59.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.002+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff43c029540 con 0x7ff454102e40 2024-03-24T09:12:59.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.213+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff45410bfa0 con 0x7ff440038fe0 2024-03-24T09:12:59.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.214+0000 7ff4517fa700 1 -- 172.21.15.116:0/1903002175 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7ff45410bfa0 con 0x7ff440038fe0 2024-03-24T09:12:59.215 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:12:59.215 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}] 2024-03-24T09:12:59.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.216+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff440038fe0 msgr2=0x7ff44003b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:59.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.216+0000 7ff45a4aa700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff440038fe0 0x7ff44003b4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff444006fd0 tx=0x7ff444006e40 comp rx=0 tx=0).stop 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.216+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 msgr2=0x7ff4541974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.216+0000 7ff45a4aa700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff4541974f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff43c004f40 tx=0x7ff43c005020 comp rx=0 tx=0).stop 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.217+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 shutdown_connections 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.217+0000 7ff45a4aa700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff440038fe0 0x7ff44003b4a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.217+0000 7ff45a4aa700 1 --2- 172.21.15.116:0/1903002175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff454102e40 0x7ff4541974f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.217+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 >> 172.21.15.116:0/1903002175 conn(0x7ff4540fc5f0 msgr2=0x7ff454103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.217+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 shutdown_connections 2024-03-24T09:12:59.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:12:59.217+0000 7ff45a4aa700 1 -- 172.21.15.116:0/1903002175 wait complete. 2024-03-24T09:12:59.842 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi143 2024-03-24T09:12:59.843 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:12:59.843 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.conf 2024-03-24T09:12:59.863 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:12:59.863 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:12:59.920 INFO:tasks.cephadm:Adding host smithi143 to orchestrator... 2024-03-24T09:12:59.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch host add smithi143 2024-03-24T09:13:00.158 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:13:00.484 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:00.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:00 smithi116 ceph-mon[32207]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:13:00.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:00 smithi116 ceph-mon[32207]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:01.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 -- 172.21.15.116:0/422314275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 msgr2=0x7f9500102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:01.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 --2- 172.21.15.116:0/422314275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f9500102200 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f94f0009910 tx=0x7f94f0009c20 comp rx=0 tx=0).stop 2024-03-24T09:13:01.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 -- 172.21.15.116:0/422314275 shutdown_connections 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 --2- 172.21.15.116:0/422314275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f9500102200 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 -- 172.21.15.116:0/422314275 >> 172.21.15.116:0/422314275 conn(0x7f95000fd6d0 msgr2=0x7f95000ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 -- 172.21.15.116:0/422314275 shutdown_connections 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.633+0000 7f9505d8f700 1 -- 172.21.15.116:0/422314275 wait complete. 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f9505d8f700 1 Processor -- start 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f9505d8f700 1 -- start start 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f9505d8f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f95001974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f9505d8f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9500197a30 con 0x7f9500101e20 2024-03-24T09:13:01.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f94ff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f95001974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f94ff7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f95001974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51878/0 (socket says 172.21.15.116:51878) 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f94ff7fe700 1 -- 172.21.15.116:0/537863814 learned_addr learned my addr 172.21.15.116:0/537863814 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f94ff7fe700 1 -- 172.21.15.116:0/537863814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f00095f0 con 0x7f9500101e20 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.634+0000 7f94ff7fe700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f95001974f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94f0004f40 tx=0x7f94f0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.635+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94f001d070 con 0x7f9500101e20 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.635+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9500197c30 con 0x7f9500101e20 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.635+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94f000b750 con 0x7f9500101e20 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.635+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94f00037d0 con 0x7f9500101e20 2024-03-24T09:13:01.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.635+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9500198050 con 0x7f9500101e20 2024-03-24T09:13:01.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.636+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7f94f0003a90 con 0x7f9500101e20 2024-03-24T09:13:01.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.636+0000 7f94fd7fa700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f94ec038f80 0x7f94ec03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:01.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.636+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f94f004cda0 con 0x7f9500101e20 2024-03-24T09:13:01.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.636+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9500063830 con 0x7f9500101e20 2024-03-24T09:13:01.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.636+0000 7f94f6ffd700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f94ec038f80 0x7f94ec03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:01.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.636+0000 7f94f6ffd700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f94ec038f80 0x7f94ec03b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94e8006fd0 tx=0x7f94e8006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:01.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.641+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f94f0026080 con 0x7f9500101e20 2024-03-24T09:13:01.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:01.838+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}) -- 0x7f950010bf00 con 0x7f94ec038f80 2024-03-24T09:13:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:03.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:13:03.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:03.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:04.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:03 smithi116 ceph-mon[32207]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:13:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:03 smithi116 ceph-mon[32207]: Deploying cephadm binary to smithi143 2024-03-24T09:13:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:03 smithi116 ceph-mon[32207]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:03 smithi116 ceph-mon[32207]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:13:05.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:04 smithi116 ceph-mon[32207]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:13:05.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:04 smithi116 ceph-mon[32207]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:06.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.087+0000 7f94fd7fa700 1 -- 172.21.15.116:0/537863814 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f950010bf00 con 0x7f94ec038f80 2024-03-24T09:13:06.087 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi143' with addr '172.21.15.143' 2024-03-24T09:13:06.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f94ec038f80 msgr2=0x7f94ec03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:06.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f94ec038f80 0x7f94ec03b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94e8006fd0 tx=0x7f94e8006e40 comp rx=0 tx=0).stop 2024-03-24T09:13:06.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 msgr2=0x7f95001974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:06.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f95001974f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94f0004f40 tx=0x7f94f0005020 comp rx=0 tx=0).stop 2024-03-24T09:13:06.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 shutdown_connections 2024-03-24T09:13:06.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f94ec038f80 0x7f94ec03b440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:06.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 --2- 172.21.15.116:0/537863814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9500101e20 0x7f95001974f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:06.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 >> 172.21.15.116:0/537863814 conn(0x7f95000fd6d0 msgr2=0x7f95001037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:06.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 shutdown_connections 2024-03-24T09:13:06.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:06.089+0000 7f9505d8f700 1 -- 172.21.15.116:0/537863814 wait complete. 2024-03-24T09:13:06.702 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:06.703 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:06.703 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:06.703 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:06 smithi116 ceph-mon[32207]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:06.703 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:06.703 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:06.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T09:13:06.967 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:13:07.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:07 smithi116 ceph-mon[32207]: Added host smithi143 2024-03-24T09:13:07.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:07.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.445+0000 7fb1933b2700 1 -- 172.21.15.116:0/3893819044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 msgr2=0x7fb18c1017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.445+0000 7fb1933b2700 1 --2- 172.21.15.116:0/3893819044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c1017f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb17c009910 tx=0x7fb17c009c20 comp rx=0 tx=0).stop 2024-03-24T09:13:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.445+0000 7fb1933b2700 1 -- 172.21.15.116:0/3893819044 shutdown_connections 2024-03-24T09:13:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.445+0000 7fb1933b2700 1 --2- 172.21.15.116:0/3893819044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c1017f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.445+0000 7fb1933b2700 1 -- 172.21.15.116:0/3893819044 >> 172.21.15.116:0/3893819044 conn(0x7fb18c0fbd40 msgr2=0x7fb18c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:08.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb1933b2700 1 -- 172.21.15.116:0/3893819044 shutdown_connections 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb1933b2700 1 -- 172.21.15.116:0/3893819044 wait complete. 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb1933b2700 1 Processor -- start 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb1933b2700 1 -- start start 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb1933b2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb1933b2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb18c197970 con 0x7fb18c101410 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb19114e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.446+0000 7fb19114e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59924/0 (socket says 172.21.15.116:59924) 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb19114e700 1 -- 172.21.15.116:0/2670422546 learned_addr learned my addr 172.21.15.116:0/2670422546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:13:08.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb19114e700 1 -- 172.21.15.116:0/2670422546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17c0095f0 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb19114e700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c197430 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb17c006010 tx=0x7fb17c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb17c01d070 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb17c00bc00 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb17c003930 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb18c197b70 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.447+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb18c198010 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.448+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7fb17c00bd70 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.448+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb18c063830 con 0x7fb18c101410 2024-03-24T09:13:08.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.448+0000 7fb1827fc700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb178038fe0 0x7fb17803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:08.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.448+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb17c04ccf0 con 0x7fb18c101410 2024-03-24T09:13:08.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.448+0000 7fb19094d700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb178038fe0 0x7fb17803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:08.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.449+0000 7fb19094d700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb178038fe0 0x7fb17803b4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb188006fd0 tx=0x7fb188006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:08.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.453+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb17c029360 con 0x7fb18c101410 2024-03-24T09:13:08.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.654+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb18c10af80 con 0x7fb178038fe0 2024-03-24T09:13:08.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.655+0000 7fb1827fc700 1 -- 172.21.15.116:0/2670422546 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7fb18c10af80 con 0x7fb178038fe0 2024-03-24T09:13:08.655 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:13:08.656 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}] 2024-03-24T09:13:08.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb178038fe0 msgr2=0x7fb17803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb178038fe0 0x7fb17803b4a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb188006fd0 tx=0x7fb188006e40 comp rx=0 tx=0).stop 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 msgr2=0x7fb18c197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c197430 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb17c006010 tx=0x7fb17c004ff0 comp rx=0 tx=0).stop 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 shutdown_connections 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb178038fe0 0x7fb17803b4a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 --2- 172.21.15.116:0/2670422546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb18c101410 0x7fb18c197430 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 >> 172.21.15.116:0/2670422546 conn(0x7fb18c0fbd40 msgr2=0x7fb18c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 shutdown_connections 2024-03-24T09:13:08.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:08.657+0000 7fb1933b2700 1 -- 172.21.15.116:0/2670422546 wait complete. 2024-03-24T09:13:08.948 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:08 smithi116 ceph-mon[32207]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:09.249 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T09:13:09.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T09:13:09.488 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:13:09.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:09.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:09 smithi116 ceph-mon[32207]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:13:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 -- 172.21.15.116:0/194275331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 msgr2=0x7f0878106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 --2- 172.21.15.116:0/194275331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878106f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0868009910 tx=0x7f0868009c20 comp rx=0 tx=0).stop 2024-03-24T09:13:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 -- 172.21.15.116:0/194275331 shutdown_connections 2024-03-24T09:13:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 --2- 172.21.15.116:0/194275331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878106f50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:10.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 -- 172.21.15.116:0/194275331 >> 172.21.15.116:0/194275331 conn(0x7f08780fc610 msgr2=0x7f08780fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 -- 172.21.15.116:0/194275331 shutdown_connections 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.810+0000 7f087e536700 1 -- 172.21.15.116:0/194275331 wait complete. 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f087e536700 1 Processor -- start 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f087e536700 1 -- start start 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f087e536700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878196330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f087e536700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0878196870 con 0x7f0878106b70 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f0877fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878196330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f0877fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878196330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59938/0 (socket says 172.21.15.116:59938) 2024-03-24T09:13:10.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.811+0000 7f0877fff700 1 -- 172.21.15.116:0/1114003942 learned_addr learned my addr 172.21.15.116:0/1114003942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f0877fff700 1 -- 172.21.15.116:0/1114003942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08680095f0 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f0877fff700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878196330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0868004f40 tx=0x7f0868005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f086801f070 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f086800b6f0 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0878196a70 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08680037d0 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.812+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0878196ef0 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.813+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7f0868003a90 con 0x7f0878106b70 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.813+0000 7f08757fa700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0860038f80 0x7f086003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:10.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.813+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0878063830 con 0x7f0878106b70 2024-03-24T09:13:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.813+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f086804ef30 con 0x7f0878106b70 2024-03-24T09:13:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.813+0000 7f08777fe700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0860038f80 0x7f086003b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:10.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.813+0000 7f08777fe700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0860038f80 0x7f086003b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f086c006fd0 tx=0x7f086c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:10.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:10.818+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f086802b720 con 0x7f0878106b70 2024-03-24T09:13:11.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.016+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f087810aea0 con 0x7f0878106b70 2024-03-24T09:13:11.017 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:10 smithi116 ceph-mon[32207]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:11.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.662+0000 7f08757fa700 1 -- 172.21.15.116:0/1114003942 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f0868028070 con 0x7f0878106b70 2024-03-24T09:13:11.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0860038f80 msgr2=0x7f086003b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0860038f80 0x7f086003b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f086c006fd0 tx=0x7f086c006e40 comp rx=0 tx=0).stop 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 msgr2=0x7f0878196330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878196330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0868004f40 tx=0x7f0868005020 comp rx=0 tx=0).stop 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 shutdown_connections 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0860038f80 0x7f086003b440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 --2- 172.21.15.116:0/1114003942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0878106b70 0x7f0878196330 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 >> 172.21.15.116:0/1114003942 conn(0x7f08780fc610 msgr2=0x7f08781086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 shutdown_connections 2024-03-24T09:13:11.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:11.664+0000 7f087e536700 1 -- 172.21.15.116:0/1114003942 wait complete. 2024-03-24T09:13:11.666 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-03-24T09:13:12.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1114003942' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T09:13:12.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:12.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:12.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:12.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:12.044 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:13:12.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:12.045 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:12.414 INFO:tasks.cephadm:Adding mon.a on smithi116 2024-03-24T09:13:12.414 INFO:tasks.cephadm:Adding mon.b on smithi117 2024-03-24T09:13:12.414 INFO:tasks.cephadm:Adding mon.c on smithi143 2024-03-24T09:13:12.415 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi116:172.21.15.116=a;smithi117:172.21.15.117=b;smithi143:172.21.15.143=c' 2024-03-24T09:13:12.641 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:13:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1114003942' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T09:13:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:13:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:13.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:13.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:13.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:13.949 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 -- 172.21.15.143:0/3317294834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 msgr2=0x7f87a4102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 --2- 172.21.15.143:0/3317294834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4102200 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8794009910 tx=0x7f8794009c20 comp rx=0 tx=0).stop 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 -- 172.21.15.143:0/3317294834 shutdown_connections 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 --2- 172.21.15.143:0/3317294834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4102200 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 -- 172.21.15.143:0/3317294834 >> 172.21.15.143:0/3317294834 conn(0x7f87a40fd6d0 msgr2=0x7f87a40ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 -- 172.21.15.143:0/3317294834 shutdown_connections 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.947+0000 7f87aa550700 1 -- 172.21.15.143:0/3317294834 wait complete. 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87aa550700 1 Processor -- start 2024-03-24T09:13:13.950 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87aa550700 1 -- start start 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87aa550700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4073cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87aa550700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f87a4074210 con 0x7f87a4101e20 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87a3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4073cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87a3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4073cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.143:49220/0 (socket says 172.21.15.143:49220) 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.948+0000 7f87a3fff700 1 -- 172.21.15.143:0/1226754350 learned_addr learned my addr 172.21.15.143:0/1226754350 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87a3fff700 1 -- 172.21.15.143:0/1226754350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87940095f0 con 0x7f87a4101e20 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87a3fff700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4073cd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8794004f40 tx=0x7f8794005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f879401d070 con 0x7f87a4101e20 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a406bcf0 con 0x7f87a4101e20 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f879400b750 con 0x7f87a4101e20 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f87940037d0 con 0x7f87a4101e20 2024-03-24T09:13:13.951 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.949+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a406c170 con 0x7f87a4101e20 2024-03-24T09:13:13.952 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.950+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7f8794004240 con 0x7f87a4101e20 2024-03-24T09:13:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.950+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a4103300 con 0x7f87a4101e20 2024-03-24T09:13:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.950+0000 7f87a17fa700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f878c038f80 0x7f878c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.950+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f879404ce80 con 0x7f87a4101e20 2024-03-24T09:13:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.950+0000 7f87a37fe700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f878c038f80 0x7f878c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:13.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.951+0000 7f87a37fe700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f878c038f80 0x7f878c03b440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8798006fd0 tx=0x7f8798006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:13.955 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:13.956+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f879402a730 con 0x7f87a4101e20 2024-03-24T09:13:14.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:13 smithi116 ceph-mon[32207]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:13:14.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:13 smithi116 ceph-mon[32207]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:13:14.160 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.160+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi117:172.21.15.117=b;smithi143:172.21.15.143=c", "target": ["mon-mgr", ""]}) -- 0x7f87a410bf00 con 0x7f878c038f80 2024-03-24T09:13:14.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.168+0000 7f87a17fa700 1 -- 172.21.15.143:0/1226754350 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f87a410bf00 con 0x7f878c038f80 2024-03-24T09:13:14.168 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled mon update... 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f878c038f80 msgr2=0x7f878c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f878c038f80 0x7f878c03b440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8798006fd0 tx=0x7f8798006e40 comp rx=0 tx=0).stop 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 msgr2=0x7f87a4073cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4073cd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8794004f40 tx=0x7f8794005020 comp rx=0 tx=0).stop 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 shutdown_connections 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f878c038f80 0x7f878c03b440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 --2- 172.21.15.143:0/1226754350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87a4101e20 0x7f87a4073cd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:14.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 >> 172.21.15.143:0/1226754350 conn(0x7f87a40fd6d0 msgr2=0x7f87a41037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:14.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 shutdown_connections 2024-03-24T09:13:14.171 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:14.170+0000 7f87aa550700 1 -- 172.21.15.143:0/1226754350 wait complete. 2024-03-24T09:13:14.739 DEBUG:teuthology.orchestra.run.smithi117:mon.b> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T09:13:14.741 DEBUG:teuthology.orchestra.run.smithi143:mon.c> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T09:13:14.743 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T09:13:14.744 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:13:15.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:15.408 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:16.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:15 smithi116 ceph-mon[32207]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi117:172.21.15.117=b;smithi143:172.21.15.143=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:16.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:15 smithi116 ceph-mon[32207]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi117:172.21.15.117=b;smithi143:172.21.15.143=c;count:3 2024-03-24T09:13:16.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:15 smithi116 ceph-mon[32207]: Deploying daemon mon.c on smithi143 2024-03-24T09:13:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:16 smithi116 ceph-mon[32207]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:18.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 -- 172.21.15.143:0/405007727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 msgr2=0x7fbd68102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 --2- 172.21.15.143:0/405007727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68102200 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009910 tx=0x7fbd58009c20 comp rx=0 tx=0).stop 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 -- 172.21.15.143:0/405007727 shutdown_connections 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 --2- 172.21.15.143:0/405007727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68102200 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 -- 172.21.15.143:0/405007727 >> 172.21.15.143:0/405007727 conn(0x7fbd680fd6d0 msgr2=0x7fbd680ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 -- 172.21.15.143:0/405007727 shutdown_connections 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.230+0000 7fbd6d227700 1 -- 172.21.15.143:0/405007727 wait complete. 2024-03-24T09:13:18.254 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.231+0000 7fbd6d227700 1 Processor -- start 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.231+0000 7fbd6d227700 1 -- start start 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd6d227700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd6d227700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbd680763f0 con 0x7fbd68101e20 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd66d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd66d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68074500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.143:49242/0 (socket says 172.21.15.143:49242) 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd66d9d700 1 -- 172.21.15.143:0/1063562401 learned_addr learned my addr 172.21.15.143:0/1063562401 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd66d9d700 1 -- 172.21.15.143:0/1063562401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd580095f0 con 0x7fbd68101e20 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.232+0000 7fbd66d9d700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68074500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004f40 tx=0x7fbd58005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.233+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd5801d070 con 0x7fbd68101e20 2024-03-24T09:13:18.255 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.233+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd68074a40 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.233+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbd5800b750 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.233+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd580037d0 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.233+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd68074e60 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.234+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd68063830 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.234+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7fbd58003a40 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.234+0000 7fbd4ffff700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbd50038fe0 0x7fbd5003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.234+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbd5804cda0 con 0x7fbd68101e20 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.234+0000 7fbd6659c700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbd50038fe0 0x7fbd5003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.235+0000 7fbd6659c700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbd50038fe0 0x7fbd5003b4a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c006fd0 tx=0x7fbd5c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:18.256 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.239+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbd5802a3b0 con 0x7fbd68101e20 2024-03-24T09:13:18.573 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.573+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fbd68075110 con 0x7fbd68101e20 2024-03-24T09:13:18.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.574+0000 7fbd4ffff700 1 -- 172.21.15.143:0/1063562401 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7fbd58026070 con 0x7fbd68101e20 2024-03-24T09:13:18.574 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:13:18.574 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","modified":"2024-03-24T09:11:35.626561Z","created":"2024-03-24T09:11:35.626561Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T09:13:18.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbd50038fe0 msgr2=0x7fbd5003b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:18.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbd50038fe0 0x7fbd5003b4a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c006fd0 tx=0x7fbd5c006e40 comp rx=0 tx=0).stop 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 msgr2=0x7fbd68074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68074500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd58004f40 tx=0x7fbd58005020 comp rx=0 tx=0).stop 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 shutdown_connections 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbd50038fe0 0x7fbd5003b4a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 --2- 172.21.15.143:0/1063562401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd68101e20 0x7fbd68074500 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 >> 172.21.15.143:0/1063562401 conn(0x7fbd680fd6d0 msgr2=0x7fbd681037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 shutdown_connections 2024-03-24T09:13:18.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:18.576+0000 7fbd6d227700 1 -- 172.21.15.143:0/1063562401 wait complete. 2024-03-24T09:13:18.578 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-03-24T09:13:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:18 smithi116 ceph-mon[32207]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:18 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/1063562401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:13:20.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:19 smithi143 systemd[1]: Started Ceph mon.c for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:13:20.384 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T09:13:20.385 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:13:20.618 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:13:25.096 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 -- 172.21.15.143:0/2394887520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 msgr2=0x7ffb94005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 --2- 172.21.15.143:0/2394887520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffb94005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffba8100b30 tx=0x7ffb9800feb0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 -- 172.21.15.143:0/2394887520 shutdown_connections 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 --2- 172.21.15.143:0/2394887520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffb94005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 -- 172.21.15.143:0/2394887520 >> 172.21.15.143:0/2394887520 conn(0x7ffba80fc710 msgr2=0x7ffba80feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 -- 172.21.15.143:0/2394887520 shutdown_connections 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.091+0000 7ffbaded7700 1 -- 172.21.15.143:0/2394887520 wait complete. 2024-03-24T09:13:25.097 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffbaded7700 1 Processor -- start 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffbaded7700 1 -- start start 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffbaded7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffba8198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffbaded7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffbaded7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffba806b3b0 con 0x7ffb940035c0 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffbaded7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffba81a14f0 con 0x7ffba81989c0 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffb9effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.092+0000 7ffb9effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:53258/0 (socket says 172.21.15.143:53258) 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffb9effd700 1 -- 172.21.15.143:0/3943605155 learned_addr learned my addr 172.21.15.143:0/3943605155 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:13:25.098 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffb9effd700 1 -- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 msgr2=0x7ffba8198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffb9effd700 1 -- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 msgr2=0x7ffba8198e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffb9effd700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba77fe700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffba8198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffb9effd700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba77fe700 1 -- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 msgr2=0x7ffba8198e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba77fe700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba77fe700 1 -- 172.21.15.143:0/3943605155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9800f900 con 0x7ffb940035c0 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba77fe700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffba8198480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffba8100b30 tx=0x7ffb98005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:25.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ffb98013960 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb98004030 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba81a1720 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.093+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ffb98003bb0 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.094+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba81a1c70 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.094+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7ffb98011550 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.094+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba81902a0 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.095+0000 7ffba57fa700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffb88038cd0 0x7ffb8803b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.095+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffb9804e5d0 con 0x7ffb940035c0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.095+0000 7ffb9effd700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffb88038cd0 0x7ffb8803b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:25.100 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.095+0000 7ffb9effd700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffb88038cd0 0x7ffb8803b190 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009180 tx=0x7ffb90008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:25.101 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.100+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffb980107a0 con 0x7ffb940035c0 2024-03-24T09:13:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:13:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: mon.a calling monitor election 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: mon.c calling monitor election 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: monmap epoch 2 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: last_changed 2024-03-24T09:13:19.834408+0000 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: created 2024-03-24T09:11:35.626561+0000 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: min_mon_release 19 (squid) 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: election_strategy: 1 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.c 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: fsmap 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: mgrmap e13: x(active, since 51s) 2024-03-24T09:13:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: overall HEALTH_OK 2024-03-24T09:13:25.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:25.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:25.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:25.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:25.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:13:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: mon.a calling monitor election 2024-03-24T09:13:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: mon.c calling monitor election 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: monmap epoch 2 2024-03-24T09:13:25.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: last_changed 2024-03-24T09:13:19.834408+0000 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: created 2024-03-24T09:11:35.626561+0000 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: min_mon_release 19 (squid) 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: election_strategy: 1 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.c 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: fsmap 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: mgrmap e13: x(active, since 51s) 2024-03-24T09:13:25.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: overall HEALTH_OK 2024-03-24T09:13:25.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:25.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:25.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:25.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:25.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:25.379 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.378+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ffba81089f0 con 0x7ffb940035c0 2024-03-24T09:13:25.380 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.379+0000 7ffba57fa700 1 -- 172.21.15.143:0/3943605155 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7ffb98023360 con 0x7ffb940035c0 2024-03-24T09:13:25.380 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:13:25.380 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":2,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","modified":"2024-03-24T09:13:19.834408Z","created":"2024-03-24T09:11:35.626561Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffb88038cd0 msgr2=0x7ffb8803b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffb88038cd0 0x7ffb8803b190 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ffb90009180 tx=0x7ffb90008040 comp rx=0 tx=0).stop 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 msgr2=0x7ffba8198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffba8198480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffba8100b30 tx=0x7ffb98005bc0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 shutdown_connections 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffb88038cd0 0x7ffb8803b190 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb940035c0 0x7ffba8198480 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.383 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 --2- 172.21.15.143:0/3943605155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffba81989c0 0x7ffba8198e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:25.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 >> 172.21.15.143:0/3943605155 conn(0x7ffba80fc710 msgr2=0x7ffba80fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:25.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 shutdown_connections 2024-03-24T09:13:25.384 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:25.381+0000 7ffbaded7700 1 -- 172.21.15.143:0/3943605155 wait complete. 2024-03-24T09:13:25.384 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 2 2024-03-24T09:13:26.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:25 smithi143 ceph-mon[35965]: Deploying daemon mon.b on smithi117 2024-03-24T09:13:26.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:25 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/3943605155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:13:26.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:25 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:26.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:25 smithi116 ceph-mon[32207]: Deploying daemon mon.b on smithi117 2024-03-24T09:13:26.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:25 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/3943605155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:13:26.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:25 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:27.065 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T09:13:27.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T09:13:27.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:26 smithi116 ceph-mon[32207]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:26 smithi143 ceph-mon[35965]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:27.310 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:13:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:28 smithi117 systemd[1]: Started Ceph mon.b for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: mon.a calling monitor election 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: mon.c calling monitor election 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: mon.b calling monitor election 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: monmap epoch 3 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: last_changed 2024-03-24T09:13:28.630493+0000 2024-03-24T09:13:34.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: created 2024-03-24T09:11:35.626561+0000 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: min_mon_release 19 (squid) 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: election_strategy: 1 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.c 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: fsmap 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: mgrmap e13: x(active, since 59s) 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: overall HEALTH_OK 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.027 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.028 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.028 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: mon.a calling monitor election 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: mon.c calling monitor election 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: mon.b calling monitor election 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: monmap epoch 3 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:34.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: last_changed 2024-03-24T09:13:28.630493+0000 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: created 2024-03-24T09:11:35.626561+0000 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: min_mon_release 19 (squid) 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: election_strategy: 1 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.c 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: fsmap 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: mgrmap e13: x(active, since 59s) 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: overall HEALTH_OK 2024-03-24T09:13:34.069 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.070 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.070 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.070 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.070 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:13:34.070 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: mon.a calling monitor election 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: mon.c calling monitor election 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: mon.b calling monitor election 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.071 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: monmap epoch 3 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: last_changed 2024-03-24T09:13:28.630493+0000 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: created 2024-03-24T09:11:35.626561+0000 2024-03-24T09:13:34.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: min_mon_release 19 (squid) 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: election_strategy: 1 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: 1: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.c 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: fsmap 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: mgrmap e13: x(active, since 59s) 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: overall HEALTH_OK 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.073 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.074 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.651 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 -- 172.21.15.143:0/877956545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 msgr2=0x7fc7e40086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 --2- 172.21.15.143:0/877956545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7e40086b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806a640 tx=0x7fc7f00095d0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 -- 172.21.15.143:0/877956545 shutdown_connections 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 --2- 172.21.15.143:0/877956545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7e40086b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 -- 172.21.15.143:0/877956545 >> 172.21.15.143:0/877956545 conn(0x7fc7f80fc6c0 msgr2=0x7fc7f80feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 -- 172.21.15.143:0/877956545 shutdown_connections 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.647+0000 7fc800018700 1 -- 172.21.15.143:0/877956545 wait complete. 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 Processor -- start 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 -- start start 2024-03-24T09:13:34.652 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7f819d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 0x7fc7f8198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f819e0d0 con 0x7fc7e4006240 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81a4360 con 0x7fc7f8197bc0 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.648+0000 7fc800018700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f81a44f0 con 0x7fc7f819daf0 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39158/0 (socket says 172.21.15.143:39158) 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 -- 172.21.15.143:0/2690104871 learned_addr learned my addr 172.21.15.143:0/2690104871 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:13:34.653 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7f819d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fe5b5700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 0x7fc7f8198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 msgr2=0x7fc7f8197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 msgr2=0x7fc7f8197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fe5b5700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 msgr2=0x7fc7f8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fd5b3700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:13:34.654 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fe5b5700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 msgr2=0x7fc7f8198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fe5b5700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 0x7fc7f8198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fe5b5700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 0x7fc7f8198070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 msgr2=0x7fc7f8197680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 msgr2=0x7fc7f8198070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 0x7fc7f8198070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 -- 172.21.15.143:0/2690104871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f00098d0 con 0x7fc7e4006240 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.649+0000 7fc7fddb4700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7f819d5b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806a640 tx=0x7fc7f0009160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.650+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00059d0 con 0x7fc7e4006240 2024-03-24T09:13:34.655 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.650+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7f001a8f0 con 0x7fc7e4006240 2024-03-24T09:13:34.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.650+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f8100a20 con 0x7fc7e4006240 2024-03-24T09:13:34.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.650+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00198d0 con 0x7fc7e4006240 2024-03-24T09:13:34.656 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.650+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f8100f10 con 0x7fc7e4006240 2024-03-24T09:13:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.651+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7fc7f0019a50 con 0x7fc7e4006240 2024-03-24T09:13:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.651+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f806acc0 con 0x7fc7e4006240 2024-03-24T09:13:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.651+0000 7fc7eeffd700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7dc039410 0x7fc7dc03b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.651+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc7f004e8b0 con 0x7fc7e4006240 2024-03-24T09:13:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.651+0000 7fc7fd5b3700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7dc039410 0x7fc7dc03b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.652+0000 7fc7fd5b3700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7dc039410 0x7fc7dc03b8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8009180 tx=0x7fc7e8008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.657+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc7f00183c0 con 0x7fc7e4006240 2024-03-24T09:13:34.953 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.951+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc7f81099e0 con 0x7fc7e4006240 2024-03-24T09:13:34.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:34 smithi143 ceph-mon[35965]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:34.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:34.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:34.953 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:34.954 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.952+0000 7fc7eeffd700 1 -- 172.21.15.143:0/2690104871 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7fc7f002cc30 con 0x7fc7e4006240 2024-03-24T09:13:34.954 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-03-24T09:13:34.954 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":3,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","modified":"2024-03-24T09:13:28.630493Z","created":"2024-03-24T09:11:35.626561Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7dc039410 msgr2=0x7fc7dc03b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7dc039410 0x7fc7dc03b8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8009180 tx=0x7fc7e8008040 comp rx=0 tx=0).stop 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 msgr2=0x7fc7f819d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7f819d5b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc7f806a640 tx=0x7fc7f0009160 comp rx=0 tx=0).stop 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 shutdown_connections 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7f8197bc0 0x7fc7f8198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7dc039410 0x7fc7dc03b8d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.956 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7e4006240 0x7fc7f819d5b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 --2- 172.21.15.143:0/2690104871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7f819daf0 0x7fc7f8197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:34.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 >> 172.21.15.143:0/2690104871 conn(0x7fc7f80fc6c0 msgr2=0x7fc7f80fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:34.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 shutdown_connections 2024-03-24T09:13:34.957 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:34.954+0000 7fc800018700 1 -- 172.21.15.143:0/2690104871 wait complete. 2024-03-24T09:13:34.957 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 3 2024-03-24T09:13:35.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:34 smithi117 ceph-mon[35788]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:35.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:35.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:35.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:35.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:34 smithi116 ceph-mon[32207]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:35.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:35.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:35.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:13:35.685 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T09:13:35.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T09:13:35.935 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:13:36.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/2690104871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:13:36.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:36.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:36.057 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:13:35.629+0000 7f85e6988700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/2690104871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:36.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:36.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:36.060 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/2690104871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T09:13:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:36.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:36.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:36.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:37.076 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:13:37.076 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:13:37.076 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:13:37.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T09:13:37.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: Reconfiguring daemon mon.a on smithi116 2024-03-24T09:13:37.077 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:36 smithi116 ceph-mon[32207]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: Reconfiguring daemon mon.a on smithi116 2024-03-24T09:13:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:36 smithi143 ceph-mon[35965]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:13:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T09:13:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: Reconfiguring daemon mon.a on smithi116 2024-03-24T09:13:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:36 smithi117 ceph-mon[35788]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:38.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 -- 172.21.15.116:0/463561228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac106b70 msgr2=0x7ff4ac106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:38.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 --2- 172.21.15.116:0/463561228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac106b70 0x7ff4ac106f50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009910 tx=0x7ff4a8009c20 comp rx=0 tx=0).stop 2024-03-24T09:13:38.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 -- 172.21.15.116:0/463561228 shutdown_connections 2024-03-24T09:13:38.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 --2- 172.21.15.116:0/463561228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac106b70 0x7ff4ac106f50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 -- 172.21.15.116:0/463561228 >> 172.21.15.116:0/463561228 conn(0x7ff4ac0fc610 msgr2=0x7ff4ac0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:38.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 -- 172.21.15.116:0/463561228 shutdown_connections 2024-03-24T09:13:38.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 -- 172.21.15.116:0/463561228 wait complete. 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 Processor -- start 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.049+0000 7ff4b3564700 1 -- start start 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac106b70 0x7ff4ac197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 0x7ff4ac19c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4ac19c540 0x7ff4ac19c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac198360 con 0x7ff4ac197c70 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1a42b0 con 0x7ff4ac106b70 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1a43f0 con 0x7ff4ac19c540 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 0x7ff4ac19c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 0x7ff4ac19c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44462/0 (socket says 172.21.15.116:44462) 2024-03-24T09:13:38.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 -- 172.21.15.116:0/1039407947 learned_addr learned my addr 172.21.15.116:0/1039407947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b1300700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac106b70 0x7ff4ac197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 -- 172.21.15.116:0/1039407947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4ac19c540 msgr2=0x7ff4ac19c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4ac19c540 0x7ff4ac19c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 -- 172.21.15.116:0/1039407947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac106b70 msgr2=0x7ff4ac197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac106b70 0x7ff4ac197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 -- 172.21.15.116:0/1039407947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a80095f0 con 0x7ff4ac197c70 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b1300700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac106b70 0x7ff4ac197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b0aff700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 0x7ff4ac19c000 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00b780 tx=0x7ff49c00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c004230 con 0x7ff4ac197c70 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff49c004390 con 0x7ff4ac197c70 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ac1a46d0 con 0x7ff4ac197c70 2024-03-24T09:13:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c011b10 con 0x7ff4ac197c70 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.050+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ac1a4bd0 con 0x7ff4ac197c70 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.051+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7ff49c011cb0 con 0x7ff4ac197c70 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.051+0000 7ff4a27fc700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff498039410 0x7ff49803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.051+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff49c04b700 con 0x7ff4ac197c70 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.051+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ac063830 con 0x7ff4ac197c70 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.052+0000 7ff4b1300700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff498039410 0x7ff49803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.052+0000 7ff4b1300700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff498039410 0x7ff49803b8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b5c0 tx=0x7ff4a801a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.056+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff49c01a400 con 0x7ff4ac197c70 2024-03-24T09:13:38.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.260+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff4ac10ad80 con 0x7ff4ac197c70 2024-03-24T09:13:38.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.260+0000 7ff4a27fc700 1 -- 172.21.15.116:0/1039407947 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7ff49c0151c0 con 0x7ff4ac197c70 2024-03-24T09:13:38.261 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:38.261 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-24T09:13:38.261 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:13:38.261 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff498039410 msgr2=0x7ff49803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff498039410 0x7ff49803b8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800b5c0 tx=0x7ff4a801a040 comp rx=0 tx=0).stop 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 msgr2=0x7ff4ac19c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 0x7ff4ac19c000 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00b780 tx=0x7ff49c00bb40 comp rx=0 tx=0).stop 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 shutdown_connections 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff4ac19c540 0x7ff4ac19c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac106b70 0x7ff4ac197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff498039410 0x7ff49803b8d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 --2- 172.21.15.116:0/1039407947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4ac197c70 0x7ff4ac19c000 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:38.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 >> 172.21.15.116:0/1039407947 conn(0x7ff4ac0fc610 msgr2=0x7ff4ac108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:38.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 shutdown_connections 2024-03-24T09:13:38.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:38.262+0000 7ff4b3564700 1 -- 172.21.15.116:0/1039407947 wait complete. 2024-03-24T09:13:39.056 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:39.056 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1039407947' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:39.056 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:39.056 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:39.057 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:39.057 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:39.057 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:39.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:39.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1039407947' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:39.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:39.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:39.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:39.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:39.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:39.083 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T09:13:39.083 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:13:39.083 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T09:13:39.113 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:13:39.113 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:13:39.180 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:13:39.181 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T09:13:39.210 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:13:39.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:13:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1039407947' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:39.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:39.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:39.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:39.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:39.279 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:13:39.279 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T09:13:39.313 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:13:39.314 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:13:39.385 INFO:tasks.cephadm:Adding mgr.x on smithi116 2024-03-24T09:13:39.386 INFO:tasks.cephadm:Adding mgr.y on smithi117 2024-03-24T09:13:39.386 INFO:tasks.cephadm:Adding mgr.z on smithi143 2024-03-24T09:13:39.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi116=x;smithi117=y;smithi143=z' 2024-03-24T09:13:39.658 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:13:39.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:39 smithi117 ceph-mon[35788]: Reconfiguring mon.b (monmap changed)... 2024-03-24T09:13:39.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:39 smithi117 ceph-mon[35788]: Reconfiguring daemon mon.b on smithi117 2024-03-24T09:13:40.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:39 smithi143 ceph-mon[35965]: Reconfiguring mon.b (monmap changed)... 2024-03-24T09:13:40.087 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:39 smithi143 ceph-mon[35965]: Reconfiguring daemon mon.b on smithi117 2024-03-24T09:13:40.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:39 smithi116 ceph-mon[32207]: Reconfiguring mon.b (monmap changed)... 2024-03-24T09:13:40.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:39 smithi116 ceph-mon[32207]: Reconfiguring daemon mon.b on smithi117 2024-03-24T09:13:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:40 smithi143 ceph-mon[35965]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:40 smithi117 ceph-mon[35788]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:40 smithi116 ceph-mon[32207]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T09:13:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T09:13:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:42.044 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 -- 172.21.15.143:0/3358637281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 msgr2=0x7f5358109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 --2- 172.21.15.143:0/3358637281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358109580 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f535000b0a0 tx=0x7f535000b3b0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 -- 172.21.15.143:0/3358637281 shutdown_connections 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 --2- 172.21.15.143:0/3358637281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358109580 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 --2- 172.21.15.143:0/3358637281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 0x7f5358101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 --2- 172.21.15.143:0/3358637281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 0x7f5358102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 -- 172.21.15.143:0/3358637281 >> 172.21.15.143:0/3358637281 conn(0x7f53580fc6f0 msgr2=0x7f53580feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 -- 172.21.15.143:0/3358637281 shutdown_connections 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.042+0000 7f535d1f0700 1 -- 172.21.15.143:0/3358637281 wait complete. 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 Processor -- start 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 -- start start 2024-03-24T09:13:42.045 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 0x7f53581050c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 0x7f5358103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358103c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53581043d0 con 0x7f5358102330 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5358104540 con 0x7f5358101130 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535d1f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f53581a21c0 con 0x7f5358102cd0 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358103c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535759e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358103c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57170/0 (socket says 172.21.15.143:57170) 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535759e700 1 -- 172.21.15.143:0/308408106 learned_addr learned my addr 172.21.15.143:0/308408106 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.043+0000 7f535659c700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 0x7f5358103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:42.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f5356d9d700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 0x7f53581050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535759e700 1 -- 172.21.15.143:0/308408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 msgr2=0x7f53581050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535759e700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 0x7f53581050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535759e700 1 -- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 msgr2=0x7f5358103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535759e700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 0x7f5358103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535759e700 1 -- 172.21.15.143:0/308408106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5350009d00 con 0x7f5358102cd0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f5356d9d700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 0x7f53581050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535659c700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 0x7f5358103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535759e700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358103c50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5350004d00 tx=0x7f5350004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535000e040 con 0x7f5358102cd0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f535000cec0 con 0x7f5358102cd0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53581a2440 con 0x7f5358102cd0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535001bb60 con 0x7f5358102cd0 2024-03-24T09:13:42.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.044+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53581a28b0 con 0x7f5358102cd0 2024-03-24T09:13:42.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.045+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7f535001bcc0 con 0x7f5358102cd0 2024-03-24T09:13:42.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.045+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5358068ec0 con 0x7f5358102cd0 2024-03-24T09:13:42.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.045+0000 7f533ffff700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53400391c0 0x7f534003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:42.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.045+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f535004d9d0 con 0x7f5358102cd0 2024-03-24T09:13:42.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.045+0000 7f5356d9d700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53400391c0 0x7f534003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:42.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.046+0000 7f5356d9d700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53400391c0 0x7f534003b680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5358102190 tx=0x7f534c00a680 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:42.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.050+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f535001a790 con 0x7f5358102cd0 2024-03-24T09:13:42.253 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.253+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi117=y;smithi143=z", "target": ["mon-mgr", ""]}) -- 0x7f53580fe810 con 0x7f53400391c0 2024-03-24T09:13:42.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:41 smithi143 ceph-mon[35965]: Reconfiguring mon.c (monmap changed)... 2024-03-24T09:13:42.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:41 smithi143 ceph-mon[35965]: Reconfiguring daemon mon.c on smithi143 2024-03-24T09:13:42.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:41 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:42.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:41 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:42.254 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:41 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:42.261 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.261+0000 7f533ffff700 1 -- 172.21.15.143:0/308408106 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f53580fe810 con 0x7f53400391c0 2024-03-24T09:13:42.261 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled mgr update... 2024-03-24T09:13:42.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53400391c0 msgr2=0x7f534003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:42.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53400391c0 0x7f534003b680 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5358102190 tx=0x7f534c00a680 comp rx=0 tx=0).stop 2024-03-24T09:13:42.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 msgr2=0x7f5358103c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:42.263 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358103c50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5350004d00 tx=0x7f5350004d30 comp rx=0 tx=0).stop 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 shutdown_connections 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5358102cd0 0x7f5358103c50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5358101130 0x7f53581050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53400391c0 0x7f534003b680 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 --2- 172.21.15.143:0/308408106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5358102330 0x7f5358103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 >> 172.21.15.143:0/308408106 conn(0x7f53580fc6f0 msgr2=0x7f53580fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 shutdown_connections 2024-03-24T09:13:42.264 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:13:42.263+0000 7f535d1f0700 1 -- 172.21.15.143:0/308408106 wait complete. 2024-03-24T09:13:42.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:41 smithi117 ceph-mon[35788]: Reconfiguring mon.c (monmap changed)... 2024-03-24T09:13:42.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:41 smithi117 ceph-mon[35788]: Reconfiguring daemon mon.c on smithi143 2024-03-24T09:13:42.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:41 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:42.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:41 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:42.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:41 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:42.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:41 smithi116 ceph-mon[32207]: Reconfiguring mon.c (monmap changed)... 2024-03-24T09:13:42.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:41 smithi116 ceph-mon[32207]: Reconfiguring daemon mon.c on smithi143 2024-03-24T09:13:42.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:41 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:42.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:41 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:42.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:41 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:42.820 DEBUG:teuthology.orchestra.run.smithi117:mgr.y> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T09:13:42.823 DEBUG:teuthology.orchestra.run.smithi143:mgr.z> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T09:13:42.826 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T09:13:42.827 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:13:42.827 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T09:13:42.843 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T09:13:42.843 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-24T09:13:42.898 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T09:13:42.898 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:42.898 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T09:13:42.899 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:42.899 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:42.899 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 09:12:46.577439324 +0000 2024-03-24T09:13:42.899 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 09:08:27.736276139 +0000 2024-03-24T09:13:42.899 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 09:08:27.736276139 +0000 2024-03-24T09:13:42.899 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 09:08:27.736276139 +0000 2024-03-24T09:13:42.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T09:13:42.967 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T09:13:42.967 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T09:13:42.967 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000206277 s, 2.5 MB/s 2024-03-24T09:13:42.969 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T09:13:43.025 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 09:12:46.615438702 +0000 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 09:08:28.075270523 +0000 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 09:08:28.075270523 +0000 2024-03-24T09:13:43.081 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 09:08:28.075270523 +0000 2024-03-24T09:13:43.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T09:13:43.147 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T09:13:43.147 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T09:13:43.147 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000181845 s, 2.8 MB/s 2024-03-24T09:13:43.148 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T09:13:43.203 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-24T09:13:43.259 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T09:13:43.259 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:43.259 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T09:13:43.259 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:43.259 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:43.259 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 09:12:46.656438031 +0000 2024-03-24T09:13:43.260 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 09:08:28.406265039 +0000 2024-03-24T09:13:43.260 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 09:08:28.406265039 +0000 2024-03-24T09:13:43.260 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 09:08:28.406265039 +0000 2024-03-24T09:13:43.260 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T09:13:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:43.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:43.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:43.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:43.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:43.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:43.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:43.325 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T09:13:43.325 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T09:13:43.326 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000206436 s, 2.5 MB/s 2024-03-24T09:13:43.327 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T09:13:43.383 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 09:12:46.688437508 +0000 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 09:08:28.736259572 +0000 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 09:08:28.736259572 +0000 2024-03-24T09:13:43.439 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-24 09:08:28.736259572 +0000 2024-03-24T09:13:43.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T09:13:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-24T09:13:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-24T09:13:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000213906 s, 2.4 MB/s 2024-03-24T09:13:43.506 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T09:13:43.562 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-24T09:13:43.562 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T09:13:43.582 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T09:13:43.582 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-24T09:13:43.640 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 09:13:28.098792615 +0000 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 09:08:31.451613309 +0000 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 09:08:31.451613309 +0000 2024-03-24T09:13:43.641 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 09:08:31.451613309 +0000 2024-03-24T09:13:43.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T09:13:43.711 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T09:13:43.711 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T09:13:43.711 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000271762 s, 1.9 MB/s 2024-03-24T09:13:43.712 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T09:13:43.770 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 09:13:28.098792615 +0000 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 09:08:31.808606215 +0000 2024-03-24T09:13:43.827 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 09:08:31.808606215 +0000 2024-03-24T09:13:43.828 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 09:08:31.808606215 +0000 2024-03-24T09:13:43.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T09:13:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T09:13:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T09:13:43.895 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000211478 s, 2.4 MB/s 2024-03-24T09:13:43.896 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T09:13:43.954 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-24T09:13:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T09:13:44.010 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:44.010 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T09:13:44.010 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:44.011 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:44.011 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 09:13:28.099792596 +0000 2024-03-24T09:13:44.011 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 09:08:32.143599558 +0000 2024-03-24T09:13:44.011 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 09:08:32.143599558 +0000 2024-03-24T09:13:44.011 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 09:08:32.143599558 +0000 2024-03-24T09:13:44.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T09:13:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:43 smithi117 ceph-mon[35788]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi117=y;smithi143=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:43 smithi117 ceph-mon[35788]: Saving service mgr spec with placement smithi116=x;smithi117=y;smithi143=z;count:3 2024-03-24T09:13:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:43 smithi117 ceph-mon[35788]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T09:13:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:43 smithi117 ceph-mon[35788]: Reconfiguring daemon mgr.x on smithi116 2024-03-24T09:13:44.076 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T09:13:44.076 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T09:13:44.076 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000194285 s, 2.6 MB/s 2024-03-24T09:13:44.077 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T09:13:44.134 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-24T09:13:44.190 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-24 09:13:28.099792596 +0000 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-24 09:08:32.458593298 +0000 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-24 09:08:32.458593298 +0000 2024-03-24T09:13:44.191 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-24 09:08:32.458593298 +0000 2024-03-24T09:13:44.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T09:13:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:43 smithi143 ceph-mon[35965]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi117=y;smithi143=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:43 smithi143 ceph-mon[35965]: Saving service mgr spec with placement smithi116=x;smithi117=y;smithi143=z;count:3 2024-03-24T09:13:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:43 smithi143 ceph-mon[35965]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T09:13:44.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:43 smithi143 ceph-mon[35965]: Reconfiguring daemon mgr.x on smithi116 2024-03-24T09:13:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-24T09:13:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-24T09:13:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000233459 s, 2.2 MB/s 2024-03-24T09:13:44.258 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T09:13:44.315 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-03-24T09:13:44.315 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T09:13:44.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:43 smithi116 ceph-mon[32207]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi117=y;smithi143=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:44.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:43 smithi116 ceph-mon[32207]: Saving service mgr spec with placement smithi116=x;smithi117=y;smithi143=z;count:3 2024-03-24T09:13:44.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:43 smithi116 ceph-mon[32207]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T09:13:44.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:43 smithi116 ceph-mon[32207]: Reconfiguring daemon mgr.x on smithi116 2024-03-24T09:13:44.331 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T09:13:44.331 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 09:13:18.803149342 +0000 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 09:08:37.375398423 +0000 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 09:08:37.375398423 +0000 2024-03-24T09:13:44.386 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 09:08:37.375398423 +0000 2024-03-24T09:13:44.387 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T09:13:44.453 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T09:13:44.453 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T09:13:44.454 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00025513 s, 2.0 MB/s 2024-03-24T09:13:44.455 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T09:13:44.512 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 09:13:18.804149315 +0000 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 09:08:37.708389464 +0000 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 09:08:37.708389464 +0000 2024-03-24T09:13:44.567 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 09:08:37.708389464 +0000 2024-03-24T09:13:44.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T09:13:44.647 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T09:13:44.647 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T09:13:44.647 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00021817 s, 2.3 MB/s 2024-03-24T09:13:44.648 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T09:13:44.707 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-03-24T09:13:44.761 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 09:13:18.804149315 +0000 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 09:08:38.042380478 +0000 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 09:08:38.042380478 +0000 2024-03-24T09:13:44.762 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 09:08:38.042380478 +0000 2024-03-24T09:13:44.763 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T09:13:44.828 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T09:13:44.828 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T09:13:44.829 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000191868 s, 2.7 MB/s 2024-03-24T09:13:44.830 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T09:13:44.886 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-03-24 09:13:18.805149288 +0000 2024-03-24T09:13:44.943 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-03-24 09:08:38.362371869 +0000 2024-03-24T09:13:44.944 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-03-24 09:08:38.362371869 +0000 2024-03-24T09:13:44.944 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-03-24 09:08:38.362371869 +0000 2024-03-24T09:13:44.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T09:13:45.009 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-03-24T09:13:45.009 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-03-24T09:13:45.009 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000236883 s, 2.2 MB/s 2024-03-24T09:13:45.010 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T09:13:45.066 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-24T09:13:45.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.077 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:45.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:13:45.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:45.078 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:45.309 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:13:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:45.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:45.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:45.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:45.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:13:45.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:45.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:46.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:45 smithi143 ceph-mon[35965]: Deploying daemon mgr.z on smithi143 2024-03-24T09:13:46.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:45 smithi116 ceph-mon[32207]: Deploying daemon mgr.z on smithi143 2024-03-24T09:13:46.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:45 smithi117 ceph-mon[35788]: Deploying daemon mgr.z on smithi143 2024-03-24T09:13:46.774 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:46 smithi143 systemd[1]: Started Ceph mgr.z for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:13:47.238 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:47.239 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:47.239 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T09:13:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T09:13:47.240 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T09:13:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T09:13:47.240 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129114 s, 812 MB/s 2024-03-24T09:13:47.240 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T09:13:47.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:47.074+0000 7fd36151f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:13:47.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:47.169+0000 7fd36151f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T09:13:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T09:13:47.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T09:13:47.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:47.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-24T09:13:48.054 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:13:48.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:47 smithi116 ceph-mon[32207]: Deploying daemon mgr.y on smithi117 2024-03-24T09:13:48.142 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:47 smithi143 ceph-mon[35965]: Deploying daemon mgr.y on smithi117 2024-03-24T09:13:48.143 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:47.812+0000 7fd36151f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:13:48.143 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:47.904+0000 7fd36151f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:13:48.143 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.047+0000 7fd36151f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:13:48.225 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:47 smithi117 ceph-mon[35788]: Deploying daemon mgr.y on smithi117 2024-03-24T09:13:48.499 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.142+0000 7fd36151f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:13:48.499 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.225+0000 7fd36151f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:13:48.499 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.498+0000 7fd36151f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:13:48.774 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.608+0000 7fd36151f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:13:48.774 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.695+0000 7fd36151f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:13:49.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:48 smithi116 ceph-mon[32207]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-mon[35965]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:49.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.789+0000 7fd36151f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:13:49.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:48.967+0000 7fd36151f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:13:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:48 smithi117 ceph-mon[35788]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:49.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:49 smithi117 systemd[1]: Started Ceph mgr.y for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 -- 172.21.15.116:0/4056705443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0102250 msgr2=0x7fd0d0102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/4056705443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0102250 0x7fd0d0102670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0099c0 tx=0x7fd0cc009cd0 comp rx=0 tx=0).stop 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 -- 172.21.15.116:0/4056705443 shutdown_connections 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/4056705443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0d0103df0 0x7fd0d010a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/4056705443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0102250 0x7fd0d0102670 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/4056705443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0d0103450 0x7fd0d01038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 -- 172.21.15.116:0/4056705443 >> 172.21.15.116:0/4056705443 conn(0x7fd0d00fd810 msgr2=0x7fd0d00ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:13:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 -- 172.21.15.116:0/4056705443 shutdown_connections 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.638+0000 7fd0d84bc700 1 -- 172.21.15.116:0/4056705443 wait complete. 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 Processor -- start 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 -- start start 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0d0102250 0x7fd0d019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0d0103450 0x7fd0d019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 0x7fd0d01a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d01a1eb0 con 0x7fd0d0103df0 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d01a1ff0 con 0x7fd0d0102250 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d84bc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d01a2180 con 0x7fd0d0103450 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 0x7fd0d01a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 0x7fd0d01a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47410/0 (socket says 172.21.15.116:47410) 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 -- 172.21.15.116:0/2713383010 learned_addr learned my addr 172.21.15.116:0/2713383010 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6258700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0d0102250 0x7fd0d019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d5a57700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0d0103450 0x7fd0d019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 -- 172.21.15.116:0/2713383010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0d0103450 msgr2=0x7fd0d019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0d0103450 0x7fd0d019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 -- 172.21.15.116:0/2713383010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0d0102250 msgr2=0x7fd0d019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0d0102250 0x7fd0d019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 -- 172.21.15.116:0/2713383010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc0095f0 con 0x7fd0d0103df0 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.639+0000 7fd0d6a59700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 0x7fd0d01a18e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8007da0 tx=0x7fd0c800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.640+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800ac50 con 0x7fd0d0103df0 2024-03-24T09:13:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.640+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd0c800adb0 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.640+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d01a2460 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.640+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80115e0 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.640+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d01a2960 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.641+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 45985+0+0 (secure 0 0 0) 0x7fd0c8011780 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.641+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d0068ec0 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.641+0000 7fd0c77fe700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0bc0391c0 0x7fd0bc03b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.641+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd0c804bde0 con 0x7fd0d0103df0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.641+0000 7fd0d6258700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0bc0391c0 0x7fd0bc03b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:13:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.641+0000 7fd0d6258700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0bc0391c0 0x7fd0bc03b680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009990 tx=0x7fd0cc009f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:13:49.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.646+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd0c8010d90 con 0x7fd0d0103df0 2024-03-24T09:13:49.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:49.854+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd0d0075b50 con 0x7fd0bc0391c0 2024-03-24T09:13:50.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:49.573+0000 7fe26cd06200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:13:50.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:49.669+0000 7fe26cd06200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:49.984+0000 7fd36151f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:50.119+0000 7fd36151f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:13:50.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:13:50.248 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:50.353+0000 7fe26cd06200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:50.444+0000 7fe26cd06200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.445 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:50.446 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:50.446 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:50.446 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:13:50.446 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:13:50.446 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:13:50.463 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:50.524 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:50.464+0000 7fd36151f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:13:50.776 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:50.592+0000 7fe26cd06200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:13:50.776 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:50.691+0000 7fe26cd06200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:13:50.777 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:50.776+0000 7fe26cd06200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:13:50.949 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:50.555+0000 7fd36151f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:13:50.949 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:50.760+0000 7fd36151f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:13:50.950 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:50.949+0000 7fd36151f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:13:51.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:51.039+0000 7fe26cd06200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:13:51.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:51.146+0000 7fe26cd06200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:13:51.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:51.231+0000 7fe26cd06200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:13:51.492 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:51.231+0000 7fd36151f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:13:51.493 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:51.492+0000 7fd36151f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:13:51.620 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:51.325+0000 7fe26cd06200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:13:51.620 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:51.497+0000 7fe26cd06200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:13:51.774 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:51.727+0000 7fd36151f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:13:51.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:51.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:51.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:13:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:13:52.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:52.816 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:52.448+0000 7fe26cd06200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:13:52.816 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:52.578+0000 7fe26cd06200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:13:52.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-mon[35788]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:52.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2299802844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b186fcf-17c6-409c-9ca6-7c8373830eea"}]: dispatch 2024-03-24T09:13:52.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2299802844' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b186fcf-17c6-409c-9ca6-7c8373830eea"}]': finished 2024-03-24T09:13:52.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-mon[35788]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T09:13:52.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:13:53.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:52.525+0000 7fd36151f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:13:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:52 smithi143 ceph-mon[35965]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:52 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2299802844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b186fcf-17c6-409c-9ca6-7c8373830eea"}]: dispatch 2024-03-24T09:13:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:52 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2299802844' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b186fcf-17c6-409c-9ca6-7c8373830eea"}]': finished 2024-03-24T09:13:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:52 smithi143 ceph-mon[35965]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T09:13:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:13:53.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:52 smithi116 ceph-mon[32207]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:53.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:52 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2299802844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b186fcf-17c6-409c-9ca6-7c8373830eea"}]: dispatch 2024-03-24T09:13:53.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:52 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2299802844' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b186fcf-17c6-409c-9ca6-7c8373830eea"}]': finished 2024-03-24T09:13:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:52 smithi116 ceph-mon[32207]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T09:13:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:13:53.186 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:52.907+0000 7fe26cd06200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:13:53.186 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:52.994+0000 7fe26cd06200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:13:53.186 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:53.185+0000 7fe26cd06200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:13:53.405 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:53.096+0000 7fd36151f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:13:53.405 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:53.191+0000 7fd36151f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:13:53.405 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:13:53.404+0000 7fd36151f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:13:53.566 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:53.366+0000 7fe26cd06200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:13:53.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:53.629+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 14) ==== 91343+0+0 (secure 0 0 0) 0x7fd0c8051020 con 0x7fd0d0103df0 2024-03-24T09:13:53.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/897271292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:13:53.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-mon[35965]: Standby manager daemon z started 2024-03-24T09:13:53.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T09:13:53.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:13:53.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T09:13:53.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:53 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:53.644+0000 7fe26cd06200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:53.898+0000 7fe26cd06200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/897271292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-mon[35788]: Standby manager daemon z started 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:13:53.899 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T09:13:53.900 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:53 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:13:54.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/897271292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:13:54.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:53 smithi116 ceph-mon[32207]: Standby manager daemon z started 2024-03-24T09:13:54.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:53 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T09:13:54.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:53 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:13:54.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:53 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T09:13:54.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:53 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/2986576467' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:13:54.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:54.129+0000 7fe26cd06200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:13:54.916 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:54 smithi117 ceph-mon[35788]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-24T09:13:54.916 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:54 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T09:13:54.916 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:54 smithi117 ceph-mon[35788]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:54.917 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:54 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:55.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:54 smithi143 ceph-mon[35965]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-24T09:13:55.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:54 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T09:13:55.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:54 smithi143 ceph-mon[35965]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:55.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:54 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:55.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:54 smithi116 ceph-mon[32207]: mgrmap e14: x(active, since 79s), standbys: z 2024-03-24T09:13:55.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:54 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T09:13:55.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:54 smithi116 ceph-mon[32207]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:55.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:54 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:13:55.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:54.916+0000 7fe26cd06200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:13:55.775 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:55.469+0000 7fe26cd06200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:13:55.775 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:55.564+0000 7fe26cd06200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:13:55.775 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:13:55.774+0000 7fe26cd06200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:13:55.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:13:55.847+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 15) ==== 136701+0+0 (secure 0 0 0) 0x7fd0c8020cc0 con 0x7fd0d0103df0 2024-03-24T09:13:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-mon[35788]: Standby manager daemon y started 2024-03-24T09:13:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T09:13:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:13:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T09:13:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:55 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:13:56.112 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:55 smithi116 ceph-mon[32207]: Standby manager daemon y started 2024-03-24T09:13:56.112 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:55 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T09:13:56.112 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:55 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:13:56.112 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:55 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T09:13:56.113 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:55 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:13:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:55 smithi143 ceph-mon[35965]: Standby manager daemon y started 2024-03-24T09:13:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:55 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T09:13:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:55 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:13:56.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:55 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T09:13:56.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:55 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/510673859' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:13:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:56 smithi143 ceph-mon[35965]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-24T09:13:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T09:13:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:56 smithi143 ceph-mon[35965]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:57.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:56 smithi116 ceph-mon[32207]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-24T09:13:57.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T09:13:57.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:56 smithi116 ceph-mon[32207]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:56 smithi117 ceph-mon[35788]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-03-24T09:13:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T09:13:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:56 smithi117 ceph-mon[35788]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:58.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:58 smithi116 ceph-mon[32207]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:59.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:58 smithi143 ceph-mon[35965]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:13:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:58 smithi117 ceph-mon[35788]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:00.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T09:14:00.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:13:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:00.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T09:14:00.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:13:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:00.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T09:14:00.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:13:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:01.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:00 smithi116 ceph-mon[32207]: Deploying daemon osd.0 on smithi116 2024-03-24T09:14:01.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:00 smithi116 ceph-mon[32207]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:00 smithi143 ceph-mon[35965]: Deploying daemon osd.0 on smithi116 2024-03-24T09:14:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:00 smithi143 ceph-mon[35965]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:00 smithi117 ceph-mon[35788]: Deploying daemon osd.0 on smithi116 2024-03-24T09:14:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:00 smithi117 ceph-mon[35788]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:03.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:02 smithi116 ceph-mon[32207]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:02 smithi143 ceph-mon[35965]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:02 smithi117 ceph-mon[35788]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:04 smithi143 ceph-mon[35965]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:04 smithi117 ceph-mon[35788]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:04 smithi116 ceph-mon[32207]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:06.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:06.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:06 smithi143 ceph-mon[35965]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:06 smithi117 ceph-mon[35788]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:06.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:06.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:06.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:06.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:06 smithi116 ceph-mon[32207]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:08.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:08.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:08.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:08.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:08.653 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:08.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:08 smithi116 ceph-mon[32207]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:08.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.707+0000 7fd0c77fe700 1 -- 172.21.15.116:0/2713383010 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd0d0075b50 con 0x7fd0bc0391c0 2024-03-24T09:14:08.707 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2024-03-24T09:14:08.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0bc0391c0 msgr2=0x7fd0bc03b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0bc0391c0 0x7fd0bc03b680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc009990 tx=0x7fd0cc009f10 comp rx=0 tx=0).stop 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 msgr2=0x7fd0d01a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 0x7fd0d01a18e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8007da0 tx=0x7fd0c800a340 comp rx=0 tx=0).stop 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 shutdown_connections 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0bc0391c0 0x7fd0bc03b680 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0d0103df0 0x7fd0d01a18e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0d0102250 0x7fd0d019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 --2- 172.21.15.116:0/2713383010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd0d0103450 0x7fd0d019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 >> 172.21.15.116:0/2713383010 conn(0x7fd0d00fd810 msgr2=0x7fd0d0074430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 shutdown_connections 2024-03-24T09:14:08.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:14:08.709+0000 7fd0d84bc700 1 -- 172.21.15.116:0/2713383010 wait complete. 2024-03-24T09:14:09.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:09.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:09.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:08 smithi143 ceph-mon[35965]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:09.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:09.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:09.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:08 smithi117 ceph-mon[35788]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:09.542 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T09:14:09.545 INFO:tasks.cephadm:Deploying osd.1 on smithi117 with /dev/vg_nvme/lv_4... 2024-03-24T09:14:09.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T09:14:09.779 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.779 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.780 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:09.780 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:09 smithi116 ceph-mon[32207]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T09:14:09.792 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:14:09.913 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:09 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.913 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:09 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:09.913 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:09 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:09.914 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:09 smithi117 ceph-mon[35788]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T09:14:10.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:09 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:10.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:09 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:10.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:09 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:10.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:09 smithi143 ceph-mon[35965]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T09:14:10.903 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:10 smithi117 ceph-mon[35788]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T09:14:10.904 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:10 smithi117 ceph-mon[35788]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T09:14:10.904 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:10 smithi117 ceph-mon[35788]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:14:10.904 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:10.904 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:10 smithi117 ceph-mon[35788]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:11.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:10 smithi116 ceph-mon[32207]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T09:14:11.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:10 smithi116 ceph-mon[32207]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T09:14:11.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:10 smithi116 ceph-mon[32207]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:14:11.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:11.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:10 smithi116 ceph-mon[32207]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:11.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:10 smithi143 ceph-mon[35965]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T09:14:11.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:10 smithi143 ceph-mon[35965]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T09:14:11.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:10 smithi143 ceph-mon[35965]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:14:11.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:11.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:10 smithi143 ceph-mon[35965]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:11.318 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0[47373]: 2024-03-24T09:14:11.157+0000 7fd710896700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T09:14:11.318 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0[47373]: 2024-03-24T09:14:11.160+0000 7fd708681700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:14:11.696 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T09:14:11.696 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T09:14:11.697 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T09:14:11.697 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T09:14:11.697 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112765 s, 930 MB/s 2024-03-24T09:14:11.697 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T09:14:12.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:11 smithi143 ceph-mon[35965]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:14:12.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:11 smithi143 ceph-mon[35965]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T09:14:12.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:11 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:11 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:11 smithi143 ceph-mon[35965]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' 2024-03-24T09:14:12.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:11 smithi117 ceph-mon[35788]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:14:12.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:11 smithi117 ceph-mon[35788]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T09:14:12.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:11 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:11 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:11 smithi117 ceph-mon[35788]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' 2024-03-24T09:14:12.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-mon[32207]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:14:12.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-mon[32207]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T09:14:12.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:11 smithi116 ceph-mon[32207]: from='osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569]' entity='osd.0' 2024-03-24T09:14:12.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_4 2024-03-24T09:14:12.549 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:14:12.896 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:14:12.896 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:14:12.896 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.896 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] boot 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:12.897 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:14:13.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:14:13.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:13.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:13.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] boot 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:13.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:14:13.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:14:13.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:13.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T09:14:13.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: osd.0 [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] boot 2024-03-24T09:14:13.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:13.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:13.800 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:13 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi116 2024-03-24T09:14:13.800 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:13 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-24T09:14:14.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:13 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi116 2024-03-24T09:14:14.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:13 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-24T09:14:14.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 -- 172.21.15.117:0/3628779669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 msgr2=0x7f32ec10a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:14.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 --2- 172.21.15.117:0/3628779669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec10a390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f32e400b0a0 tx=0x7f32e400b3b0 comp rx=0 tx=0).stop 2024-03-24T09:14:14.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 -- 172.21.15.117:0/3628779669 shutdown_connections 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 --2- 172.21.15.117:0/3628779669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec10a390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 --2- 172.21.15.117:0/3628779669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec102cc0 0x7f32ec1050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 --2- 172.21.15.117:0/3628779669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32ec1055f0 0x7f32ec1079e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 -- 172.21.15.117:0/3628779669 >> 172.21.15.117:0/3628779669 conn(0x7f32ec0fc6f0 msgr2=0x7f32ec0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 -- 172.21.15.117:0/3628779669 shutdown_connections 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 -- 172.21.15.117:0/3628779669 wait complete. 2024-03-24T09:14:14.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 Processor -- start 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.028+0000 7f32f22e8700 1 -- start start 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec102cc0 0x7f32ec102620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32ec1055f0 0x7f32ec100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec101930 con 0x7f32ec102cc0 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec101aa0 con 0x7f32ec107f20 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f32ec1a2320 con 0x7f32ec1055f0 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:14.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec1011b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44304/0 (socket says 172.21.15.117:44304) 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 -- 172.21.15.117:0/1364621059 learned_addr learned my addr 172.21.15.117:0/1364621059 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32eb7fe700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32ec1055f0 0x7f32ec100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32ebfff700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec102cc0 0x7f32ec102620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 -- 172.21.15.117:0/1364621059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32ec1055f0 msgr2=0x7f32ec100c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32ec1055f0 0x7f32ec100c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 -- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec102cc0 msgr2=0x7f32ec102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec102cc0 0x7f32ec102620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 -- 172.21.15.117:0/1364621059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e4009d00 con 0x7f32ec107f20 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f0885700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec1011b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32e40094c0 tx=0x7f32e40094f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:14:14.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400e040 con 0x7f32ec107f20 2024-03-24T09:14:14.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32e401d930 con 0x7f32ec107f20 2024-03-24T09:14:14.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32ec1a25a0 con 0x7f32ec107f20 2024-03-24T09:14:14.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.029+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e401b960 con 0x7f32ec107f20 2024-03-24T09:14:14.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.030+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32ec1a2a10 con 0x7f32ec107f20 2024-03-24T09:14:14.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.031+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32ec0fe270 con 0x7f32ec107f20 2024-03-24T09:14:14.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.032+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 15) ==== 136701+0+0 (secure 0 0 0) 0x7f32e401daa0 con 0x7f32ec107f20 2024-03-24T09:14:14.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.032+0000 7f32e97fa700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f32d40a20c0 0x7f32d40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:14.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.032+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f32e40ce610 con 0x7f32ec107f20 2024-03-24T09:14:14.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.033+0000 7f32ebfff700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f32d40a20c0 0x7f32d40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:14.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.033+0000 7f32ebfff700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f32d40a20c0 0x7f32d40a4580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32dc009a10 tx=0x7f32dc009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:14:14.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.036+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f32e40990e0 con 0x7f32ec107f20 2024-03-24T09:14:14.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:13 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi116 2024-03-24T09:14:14.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:13 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-24T09:14:14.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:14.236+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f32ec02db60 con 0x7f32d40a20c0 2024-03-24T09:14:15.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:14 smithi143 ceph-mon[35965]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T09:14:15.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:14 smithi143 ceph-mon[35965]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:15.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:14:15.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:14:15.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:15.028 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:14 smithi117 ceph-mon[35788]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T09:14:15.028 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:14 smithi117 ceph-mon[35788]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:15.028 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:14:15.029 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:14:15.029 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:14 smithi116 ceph-mon[32207]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T09:14:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:14 smithi116 ceph-mon[32207]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:14:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:14:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:15.970 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:15 smithi117 ceph-mon[35788]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:14:16.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:15 smithi143 ceph-mon[35965]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:14:16.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:15 smithi116 ceph-mon[32207]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:14:17.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:16 smithi143 ceph-mon[35965]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:16 smithi117 ceph-mon[35788]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:17.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:16 smithi116 ceph-mon[32207]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:17 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/4256144548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]: dispatch 2024-03-24T09:14:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:17 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]: dispatch 2024-03-24T09:14:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:17 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]': finished 2024-03-24T09:14:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:17 smithi143 ceph-mon[35965]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T09:14:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:17 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:17 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/3835848194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:14:18.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:17 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/4256144548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]: dispatch 2024-03-24T09:14:18.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:17 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]: dispatch 2024-03-24T09:14:18.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:17 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]': finished 2024-03-24T09:14:18.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:17 smithi117 ceph-mon[35788]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:17 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:17 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/3835848194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:17 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/4256144548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]: dispatch 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:17 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]: dispatch 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:17 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62462f7a-4125-4c73-beac-3aee728ab940"}]': finished 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:17 smithi116 ceph-mon[32207]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T09:14:18.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:17 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:18.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:17 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/3835848194' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:14:19.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:18 smithi143 ceph-mon[35965]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:19.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:18 smithi117 ceph-mon[35788]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:19.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:18 smithi116 ceph-mon[32207]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:20 smithi117 ceph-mon[35788]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:20 smithi143 ceph-mon[35965]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:21.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:20 smithi116 ceph-mon[32207]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:23.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:22 smithi117 ceph-mon[35788]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:22 smithi143 ceph-mon[35965]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:23.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:22 smithi116 ceph-mon[32207]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:24.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:23 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T09:14:24.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:23 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:24.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:23 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T09:14:24.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:23 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:24.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:23 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T09:14:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:23 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:24 smithi117 ceph-mon[35788]: Deploying daemon osd.1 on smithi117 2024-03-24T09:14:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:24 smithi117 ceph-mon[35788]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:25.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:24 smithi143 ceph-mon[35965]: Deploying daemon osd.1 on smithi117 2024-03-24T09:14:25.187 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:24 smithi143 ceph-mon[35965]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:24 smithi116 ceph-mon[32207]: Deploying daemon osd.1 on smithi117 2024-03-24T09:14:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:24 smithi116 ceph-mon[32207]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:27.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:26 smithi117 ceph-mon[35788]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:26 smithi143 ceph-mon[35965]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:26 smithi116 ceph-mon[32207]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:29.031 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:28 smithi117 ceph-mon[35788]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:28 smithi143 ceph-mon[35965]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:29.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:28 smithi116 ceph-mon[32207]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:30.345 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:30.345 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:30.345 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:30.345 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:30 smithi117 ceph-mon[35788]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:30 smithi143 ceph-mon[35965]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:30.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:30.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:30.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:30.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:30 smithi116 ceph-mon[32207]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:32.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.601+0000 7f32e97fa700 1 -- 172.21.15.117:0/1364621059 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f32ec02db60 con 0x7f32d40a20c0 2024-03-24T09:14:32.602 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 1 on host 'smithi117' 2024-03-24T09:14:32.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.603+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f32d40a20c0 msgr2=0x7f32d40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:32.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.603+0000 7f32f22e8700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f32d40a20c0 0x7f32d40a4580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32dc009a10 tx=0x7f32dc009190 comp rx=0 tx=0).stop 2024-03-24T09:14:32.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.603+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 msgr2=0x7f32ec1011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:32.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.603+0000 7f32f22e8700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec1011b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32e40094c0 tx=0x7f32e40094f0 comp rx=0 tx=0).stop 2024-03-24T09:14:32.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 shutdown_connections 2024-03-24T09:14:32.604 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32ec107f20 0x7f32ec1011b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:32.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f32d40a20c0 0x7f32d40a4580 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:32.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32ec102cc0 0x7f32ec102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:32.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 --2- 172.21.15.117:0/1364621059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32ec1055f0 0x7f32ec100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:32.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 >> 172.21.15.117:0/1364621059 conn(0x7f32ec0fc6f0 msgr2=0x7f32ec1036a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:14:32.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 shutdown_connections 2024-03-24T09:14:32.605 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:14:32.604+0000 7f32f22e8700 1 -- 172.21.15.117:0/1364621059 wait complete. 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:32.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.307 DEBUG:teuthology.orchestra.run.smithi117:osd.1> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T09:14:33.309 INFO:tasks.cephadm:Deploying osd.2 on smithi143 with /dev/vg_nvme/lv_4... 2024-03-24T09:14:33.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:33.555 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:14:33.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:33 smithi143 ceph-mon[35965]: from='osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:14:33.946 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:33 smithi143 ceph-mon[35965]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:14:34.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:33 smithi117 ceph-mon[35788]: from='osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:14:34.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:33 smithi117 ceph-mon[35788]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:14:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:33 smithi116 ceph-mon[32207]: from='osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:14:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:33 smithi116 ceph-mon[32207]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T09:14:35.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:34 smithi143 ceph-mon[35965]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T09:14:35.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:34 smithi143 ceph-mon[35965]: from='osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:14:35.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:34 smithi143 ceph-mon[35965]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T09:14:35.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:34 smithi143 ceph-mon[35965]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:14:35.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:34 smithi143 ceph-mon[35965]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:35.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-mon[35788]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T09:14:35.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-mon[35788]: from='osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:14:35.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-mon[35788]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T09:14:35.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-mon[35788]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:14:35.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-mon[35788]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:35.067 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1[40401]: 2024-03-24T09:14:34.866+0000 7fecca572700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T09:14:35.067 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:14:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1[40401]: 2024-03-24T09:14:34.869+0000 7fecc6365700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:14:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:34 smithi116 ceph-mon[32207]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T09:14:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:34 smithi116 ceph-mon[32207]: from='osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:14:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:34 smithi116 ceph-mon[32207]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T09:14:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:34 smithi116 ceph-mon[32207]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:14:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:34 smithi116 ceph-mon[32207]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T09:14:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T09:14:35.350 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T09:14:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T09:14:35.350 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113674 s, 922 MB/s 2024-03-24T09:14:35.350 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T09:14:35.954 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:35 smithi143 ceph-mon[35965]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:14:35.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:35 smithi143 ceph-mon[35965]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T09:14:35.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.955 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.962 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:35 smithi117 ceph-mon[35788]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:14:35.962 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:35 smithi117 ceph-mon[35788]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T09:14:35.962 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.962 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:35.982 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_4 2024-03-24T09:14:36.229 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:14:36.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:35 smithi116 ceph-mon[32207]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:14:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:35 smithi116 ceph-mon[32207]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T09:14:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] boot 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:14:37.185 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:37.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:37.186 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:14:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] boot 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:37.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:37.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:14:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:14:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T09:14:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: osd.1 [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] boot 2024-03-24T09:14:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:14:37.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:37.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/4030309845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28102230 msgr2=0x7f6c28102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:37.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/4030309845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28102230 0x7f6c28102650 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009990 tx=0x7f6c18009ca0 comp rx=0 tx=0).stop 2024-03-24T09:14:37.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/4030309845 shutdown_connections 2024-03-24T09:14:37.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/4030309845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c28103dd0 0x7f6c2810a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:37.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/4030309845 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28102230 0x7f6c28102650 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:37.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/4030309845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c28103430 0x7f6c28103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:37.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/4030309845 >> 172.21.15.143:0/4030309845 conn(0x7f6c280fd810 msgr2=0x7f6c280ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:14:37.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/4030309845 shutdown_connections 2024-03-24T09:14:37.692 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.691+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/4030309845 wait complete. 2024-03-24T09:14:37.694 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 Processor -- start 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 -- start start 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c28102230 0x7f6c281981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 0x7f6c281986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c28103dd0 0x7f6c2819f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2819fd40 con 0x7f6c28103dd0 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2819fe80 con 0x7f6c28102230 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2d0b7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6c281a0010 con 0x7f6c28103430 2024-03-24T09:14:37.695 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 0x7f6c281986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 0x7f6c281986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:53458/0 (socket says 172.21.15.143:53458) 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 -- 172.21.15.143:0/1525193266 learned_addr learned my addr 172.21.15.143:0/1525193266 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c26d9d700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c28102230 0x7f6c281981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c2759e700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c28103dd0 0x7f6c2819f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 -- 172.21.15.143:0/1525193266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c28102230 msgr2=0x7f6c281981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c28102230 0x7f6c281981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 -- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c28103dd0 msgr2=0x7f6c2819f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c28103dd0 0x7f6c2819f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:37.696 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.692+0000 7f6c1e59c700 1 -- 172.21.15.143:0/1525193266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c180095f0 con 0x7f6c28103430 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.693+0000 7f6c1e59c700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 0x7f6c281986f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000d980 tx=0x7f6c1000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.693+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10009b40 con 0x7f6c28103430 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.693+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6c1000b6e0 con 0x7f6c28103430 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.693+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c281a02f0 con 0x7f6c28103430 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.693+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000f620 con 0x7f6c28103430 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.693+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c281a0840 con 0x7f6c28103430 2024-03-24T09:14:37.697 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.694+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c28109710 con 0x7f6c28103430 2024-03-24T09:14:37.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.695+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 15) ==== 136701+0+0 (secure 0 0 0) 0x7f6c10010460 con 0x7f6c28103430 2024-03-24T09:14:37.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.695+0000 7f6c24d99700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c140a2140 0x7f6c140a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:14:37.698 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.696+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f6c100cc210 con 0x7f6c28103430 2024-03-24T09:14:37.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.696+0000 7f6c26d9d700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c140a2140 0x7f6c140a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:14:37.699 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.696+0000 7f6c26d9d700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c140a2140 0x7f6c140a4600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c18000c00 tx=0x7f6c18005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:14:37.700 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.700+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6c10096a30 con 0x7f6c28103430 2024-03-24T09:14:37.907 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:37.907+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6c2802db60 con 0x7f6c140a2140 2024-03-24T09:14:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:37 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi117 2024-03-24T09:14:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:37 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi117 to 17088M 2024-03-24T09:14:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:37 smithi143 ceph-mon[35965]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T09:14:38.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:37 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi117 2024-03-24T09:14:38.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:37 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi117 to 17088M 2024-03-24T09:14:38.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:37 smithi117 ceph-mon[35788]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T09:14:38.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:37 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi117 2024-03-24T09:14:38.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:37 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi117 to 17088M 2024-03-24T09:14:38.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:37 smithi116 ceph-mon[32207]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T09:14:39.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:38 smithi143 ceph-mon[35965]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:39.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:38 smithi143 ceph-mon[35965]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:14:39.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:14:39.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:14:39.118 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:38 smithi117 ceph-mon[35788]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:38 smithi117 ceph-mon[35788]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:14:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:14:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:14:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:38 smithi116 ceph-mon[32207]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:38 smithi116 ceph-mon[32207]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:14:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:14:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:14:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:40 smithi143 ceph-mon[35965]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:40 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/2516166173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]: dispatch 2024-03-24T09:14:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:40 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]: dispatch 2024-03-24T09:14:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:40 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]': finished 2024-03-24T09:14:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:40 smithi143 ceph-mon[35965]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T09:14:41.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:40 smithi117 ceph-mon[35788]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:40 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/2516166173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]: dispatch 2024-03-24T09:14:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:40 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]: dispatch 2024-03-24T09:14:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:40 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]': finished 2024-03-24T09:14:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:40 smithi117 ceph-mon[35788]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T09:14:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:40 smithi116 ceph-mon[32207]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:40 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/2516166173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]: dispatch 2024-03-24T09:14:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:40 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]: dispatch 2024-03-24T09:14:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:40 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f4d76a1-5218-4904-8b3e-0d782ada3628"}]': finished 2024-03-24T09:14:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:40 smithi116 ceph-mon[32207]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T09:14:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:42.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:41 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/798626009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:14:42.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:41 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/798626009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:14:42.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:41 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/798626009' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:14:43.175 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:42 smithi143 ceph-mon[35965]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:42 smithi117 ceph-mon[35788]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:43.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:42 smithi116 ceph-mon[32207]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:45.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:44 smithi143 ceph-mon[35965]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:44 smithi117 ceph-mon[35788]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:45.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:44 smithi116 ceph-mon[32207]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:47.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:46 smithi143 ceph-mon[35965]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:46 smithi117 ceph-mon[35788]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:47.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:46 smithi116 ceph-mon[32207]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:48.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:47 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T09:14:48.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:47 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:47 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T09:14:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:47 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:48.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T09:14:48.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:14:49.253 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:48 smithi143 ceph-mon[35965]: Deploying daemon osd.2 on smithi143 2024-03-24T09:14:49.253 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:48 smithi143 ceph-mon[35965]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:48 smithi117 ceph-mon[35788]: Deploying daemon osd.2 on smithi143 2024-03-24T09:14:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:48 smithi117 ceph-mon[35788]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:49.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:48 smithi116 ceph-mon[32207]: Deploying daemon osd.2 on smithi143 2024-03-24T09:14:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:48 smithi116 ceph-mon[32207]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:50.998 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:50 smithi143 ceph-mon[35965]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:50 smithi117 ceph-mon[35788]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:51.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:50 smithi116 ceph-mon[32207]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:53.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:52 smithi143 ceph-mon[35965]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:53.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:53.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:53.223 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:52 smithi117 ceph-mon[35788]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:53.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:52 smithi116 ceph-mon[32207]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:14:55.200 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:54 smithi143 ceph-mon[35965]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:54 smithi117 ceph-mon[35788]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:54 smithi116 ceph-mon[32207]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:55.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.819+0000 7f6c24d99700 1 -- 172.21.15.143:0/1525193266 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6c2802db60 con 0x7f6c140a2140 2024-03-24T09:14:55.820 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 2 on host 'smithi143' 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c140a2140 msgr2=0x7f6c140a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c140a2140 0x7f6c140a4600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6c18000c00 tx=0x7f6c18005fb0 comp rx=0 tx=0).stop 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 msgr2=0x7f6c281986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 0x7f6c281986f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000d980 tx=0x7f6c1000dc90 comp rx=0 tx=0).stop 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 shutdown_connections 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6c140a2140 0x7f6c140a4600 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c28103dd0 0x7f6c2819f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:55.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c28102230 0x7f6c281981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:55.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 --2- 172.21.15.143:0/1525193266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6c28103430 0x7f6c281986f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:14:55.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 >> 172.21.15.143:0/1525193266 conn(0x7f6c280fd810 msgr2=0x7f6c28106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:14:55.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 shutdown_connections 2024-03-24T09:14:55.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:14:55.822+0000 7f6c2d0b7700 1 -- 172.21.15.143:0/1525193266 wait complete. 2024-03-24T09:14:56.467 DEBUG:teuthology.orchestra.run.smithi143:osd.2> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T09:14:56.469 INFO:tasks.cephadm:Deploying osd.3 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-24T09:14:56.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T09:14:56.717 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:14:56.741 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2[41138]: 2024-03-24T09:14:56.557+0000 7f522bd957c0 -1 osd.2 0 log_to_monitors true 2024-03-24T09:14:57.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:57.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: from='osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:14:57.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:56 smithi143 ceph-mon[35965]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: from='osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:14:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:56 smithi117 ceph-mon[35788]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: from='osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:14:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:56 smithi116 ceph-mon[32207]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T09:14:58.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-mon[35965]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T09:14:58.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-mon[35965]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T09:14:58.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:58.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-mon[35965]: from='osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:14:58.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-mon[35965]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:14:58.025 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2[41138]: 2024-03-24T09:14:57.898+0000 7f52213fa700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T09:14:58.026 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:14:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2[41138]: 2024-03-24T09:14:57.901+0000 7f521d1ed700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:14:58.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:57 smithi116 ceph-mon[32207]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T09:14:58.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:57 smithi116 ceph-mon[32207]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T09:14:58.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:57 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:58.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:57 smithi116 ceph-mon[32207]: from='osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:14:58.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:57 smithi116 ceph-mon[32207]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:14:58.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:57 smithi117 ceph-mon[35788]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T09:14:58.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:57 smithi117 ceph-mon[35788]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T09:14:58.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:57 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:58.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:57 smithi117 ceph-mon[35788]: from='osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:14:58.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:57 smithi117 ceph-mon[35788]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:14:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T09:14:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T09:14:58.681 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T09:14:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T09:14:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112793 s, 930 MB/s 2024-03-24T09:14:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T09:14:59.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:58 smithi143 ceph-mon[35965]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:59.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:58 smithi143 ceph-mon[35965]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:14:59.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:58 smithi143 ceph-mon[35965]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T09:14:59.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:59.130 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:59.243 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:58 smithi116 ceph-mon[32207]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:59.243 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:58 smithi116 ceph-mon[32207]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:14:59.244 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:58 smithi116 ceph-mon[32207]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T09:14:59.244 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:59.244 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:58 smithi117 ceph-mon[35788]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:14:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:58 smithi117 ceph-mon[35788]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:14:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:58 smithi117 ceph-mon[35788]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T09:14:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:14:59.343 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-24T09:14:59.589 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:15:00.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:15:00.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] boot 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:00.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:00.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:00.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:00.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:00.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:14:59 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:15:00.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] boot 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:00.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:00.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:00.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:00.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:00.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:14:59 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: osd.2 [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] boot 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:00.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:00.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:00.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:00.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:00.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:14:59 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:00.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi143 2024-03-24T09:15:00.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi143 to 17088M 2024-03-24T09:15:00.993 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:15:00.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:15:00.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T09:15:00.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T09:15:00.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:15:01.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 -- 172.21.15.116:0/1695793237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100cc0 msgr2=0x7f843010a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:01.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 --2- 172.21.15.116:0/1695793237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100cc0 0x7f843010a870 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8428009990 tx=0x7f8428009ca0 comp rx=0 tx=0).stop 2024-03-24T09:15:01.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 -- 172.21.15.116:0/1695793237 shutdown_connections 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 --2- 172.21.15.116:0/1695793237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100cc0 0x7f843010a870 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 --2- 172.21.15.116:0/1695793237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8430103880 0x7f8430103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 --2- 172.21.15.116:0/1695793237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8430100300 0x7f8430100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 -- 172.21.15.116:0/1695793237 >> 172.21.15.116:0/1695793237 conn(0x7f84300fbea0 msgr2=0x7f84300fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 -- 172.21.15.116:0/1695793237 shutdown_connections 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 -- 172.21.15.116:0/1695793237 wait complete. 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.220+0000 7f8436ab7700 1 Processor -- start 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 -- start start 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 0x7f8430197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8430100cc0 0x7f8430198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8430103880 0x7f843019f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f843019fb60 con 0x7f8430100300 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f843019fca0 con 0x7f8430103880 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8436ab7700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f843019fe30 con 0x7f8430100cc0 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 0x7f8430197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 0x7f8430197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57372/0 (socket says 172.21.15.116:57372) 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 -- 172.21.15.116:0/3668616542 learned_addr learned my addr 172.21.15.116:0/3668616542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8427fff700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8430100cc0 0x7f8430198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8435054700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8430103880 0x7f843019f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 -- 172.21.15.116:0/3668616542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8430100cc0 msgr2=0x7f8430198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8430100cc0 0x7f8430198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 -- 172.21.15.116:0/3668616542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8430103880 msgr2=0x7f843019f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8430103880 0x7f843019f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 -- 172.21.15.116:0/3668616542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84280095f0 con 0x7f8430100300 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8434853700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 0x7f8430197fd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f842c00add0 tx=0x7f842c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.221+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c00f070 con 0x7f8430100300 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.222+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84301015b0 con 0x7f8430100300 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.222+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f842c007be0 con 0x7f8430100300 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.222+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c004280 con 0x7f8430100300 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.222+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8430101b00 con 0x7f8430100300 2024-03-24T09:15:01.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.223+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430068ec0 con 0x7f8430100300 2024-03-24T09:15:01.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.224+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) ==== 136701+0+0 (secure 0 0 0) 0x7f842c003a20 con 0x7f8430100300 2024-03-24T09:15:01.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.224+0000 7f8425ffb700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f84180a20c0 0x7f84180a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:01.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.224+0000 7f8427fff700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f84180a20c0 0x7f84180a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:01.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.224+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f842c0cbcf0 con 0x7f8430100300 2024-03-24T09:15:01.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.225+0000 7f8427fff700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f84180a20c0 0x7f84180a4580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f843006a890 tx=0x7f842000a3b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:15:01.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.229+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f842c096210 con 0x7f8430100300 2024-03-24T09:15:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi143 2024-03-24T09:15:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi143 to 17088M 2024-03-24T09:15:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:15:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:15:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T09:15:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T09:15:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:15:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi143 2024-03-24T09:15:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi143 to 17088M 2024-03-24T09:15:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T09:15:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:15:01.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T09:15:01.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T09:15:01.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:15:01.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:01.434+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f843002dce0 con 0x7f84180a20c0 2024-03-24T09:15:02.049 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:01 smithi116 sudo[51695]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:15:02.049 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:01 smithi116 sudo[51695]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:02.049 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:01 smithi116 sudo[51695]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:02.049 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:01 smithi116 sudo[51695]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:02.317 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:02 smithi116 sudo[51752]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:15:02.317 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:02 smithi116 sudo[51752]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:02.317 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:02 smithi116 sudo[51752]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:02.318 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:15:02 smithi116 sudo[51752]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:02.578 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44257]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:15:02.578 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44257]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:02.578 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44257]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:02.578 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44257]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:02.578 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:15:02.578 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:02.578 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T09:15:02.578 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:02.578 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:02.579 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:02.579 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:02.579 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:02 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:15:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T09:15:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:02 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:02.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:02.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:03.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44260]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:15:03.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44260]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:03.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44260]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:03.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:15:02 smithi117 sudo[44260]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:03.099 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:02 smithi143 sudo[44745]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:15:03.099 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:02 smithi143 sudo[44745]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:03.100 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:02 smithi143 sudo[44745]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:03.100 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:02 smithi143 sudo[44745]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:03.524 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44748]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:15:03.524 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44748]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:03.524 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44748]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:03.524 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44748]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:03.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51803]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:15:03.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51803]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:03.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51803]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:03.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51803]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:03.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:03.653+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f842c0965e0 con 0x7f8430100300 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 ceph-mon[35965]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44751]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44751]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44751]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:03.903 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:03 smithi143 sudo[44751]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:04.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:03 smithi117 ceph-mon[35788]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T09:15:04.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:03 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:04.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:03 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:04.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51819]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:15:04.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:04.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51819]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:04.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 sudo[51819]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:04.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 ceph-mon[32207]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T09:15:04.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:04.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:03 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 sudo[44754]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:15:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 sudo[44754]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 sudo[44754]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 sudo[44754]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:04.636 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44263]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T09:15:04.636 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:04.636 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44263]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:04.636 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44263]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:04.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2116027838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2f25e8-51f7-4048-a704-c27813a60056"}]: dispatch 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2116027838' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f2f25e8-51f7-4048-a704-c27813a60056"}]': finished 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:04.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:04 smithi116 ceph-mon[32207]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:05.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:05.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:05.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:05.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:05.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:05.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2116027838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2f25e8-51f7-4048-a704-c27813a60056"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2116027838' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f2f25e8-51f7-4048-a704-c27813a60056"}]': finished 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:05.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:04 smithi143 ceph-mon[35965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:05.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44266]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T09:15:05.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44266]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T09:15:05.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44266]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T09:15:05.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 sudo[44266]: pam_unix(sudo:session): session closed for user root 2024-03-24T09:15:05.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2116027838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3f2f25e8-51f7-4048-a704-c27813a60056"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:15:05.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2116027838' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f2f25e8-51f7-4048-a704-c27813a60056"}]': finished 2024-03-24T09:15:05.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T09:15:05.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:05.068 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:04 smithi117 ceph-mon[35788]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T09:15:06.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:05 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2781038378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:06.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:05 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2781038378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:06.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:05 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2781038378' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:07.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:06 smithi143 ceph-mon[35965]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:07.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:06 smithi117 ceph-mon[35788]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:07.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:06 smithi116 ceph-mon[32207]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:09.177 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:08 smithi116 ceph-mon[32207]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:08 smithi143 ceph-mon[35965]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:08 smithi117 ceph-mon[35788]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:11.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:10 smithi116 ceph-mon[32207]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:10 smithi143 ceph-mon[35965]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:10 smithi117 ceph-mon[35788]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:12.247 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T09:15:12.248 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:11 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:12.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:11 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T09:15:12.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:11 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:12.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:11 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T09:15:12.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:11 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:12.985 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:12 smithi116 ceph-mon[32207]: Deploying daemon osd.3 on smithi116 2024-03-24T09:15:12.986 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:12 smithi116 ceph-mon[32207]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:13.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:12 smithi143 ceph-mon[35965]: Deploying daemon osd.3 on smithi116 2024-03-24T09:15:13.193 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:12 smithi143 ceph-mon[35965]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:12 smithi117 ceph-mon[35788]: Deploying daemon osd.3 on smithi116 2024-03-24T09:15:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:12 smithi117 ceph-mon[35788]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:15.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:14 smithi116 ceph-mon[32207]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:15.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:14 smithi143 ceph-mon[35965]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:14 smithi117 ceph-mon[35788]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:16 smithi116 ceph-mon[32207]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:16 smithi143 ceph-mon[35965]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:16 smithi117 ceph-mon[35788]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:18.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:17 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:18.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:17 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:18.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:17 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:18 smithi143 ceph-mon[35965]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:18 smithi117 ceph-mon[35788]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:19.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:18 smithi116 ceph-mon[32207]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:20.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.695+0000 7f8425ffb700 1 -- 172.21.15.116:0/3668616542 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f843002dce0 con 0x7f84180a20c0 2024-03-24T09:15:20.697 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 3 on host 'smithi116' 2024-03-24T09:15:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f84180a20c0 msgr2=0x7f84180a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f84180a20c0 0x7f84180a4580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f843006a890 tx=0x7f842000a3b0 comp rx=0 tx=0).stop 2024-03-24T09:15:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 msgr2=0x7f8430197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 0x7f8430197fd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f842c00add0 tx=0x7f842c00c330 comp rx=0 tx=0).stop 2024-03-24T09:15:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 shutdown_connections 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8430103880 0x7f843019f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f84180a20c0 0x7f84180a4580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8430100300 0x7f8430197fd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 --2- 172.21.15.116:0/3668616542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8430100cc0 0x7f8430198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 >> 172.21.15.116:0/3668616542 conn(0x7f84300fbea0 msgr2=0x7f8430106850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 shutdown_connections 2024-03-24T09:15:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:15:20.697+0000 7f8436ab7700 1 -- 172.21.15.116:0/3668616542 wait complete. 2024-03-24T09:15:21.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:21.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:21.400 DEBUG:teuthology.orchestra.run.smithi116:osd.3> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T09:15:21.403 INFO:tasks.cephadm:Deploying osd.4 on smithi117 with /dev/vg_nvme/lv_3... 2024-03-24T09:15:21.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T09:15:21.650 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:15:21.909 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:21 smithi116 ceph-mon[32207]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:15:22.021 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:21 smithi117 ceph-mon[35788]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:15:22.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:21 smithi143 ceph-mon[35965]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T09:15:23.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:22 smithi117 ceph-mon[35788]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T09:15:23.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:22 smithi117 ceph-mon[35788]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T09:15:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:22 smithi117 ceph-mon[35788]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:15:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:22 smithi117 ceph-mon[35788]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:23.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:22 smithi116 ceph-mon[32207]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T09:15:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:22 smithi116 ceph-mon[32207]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T09:15:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:22 smithi116 ceph-mon[32207]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:15:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:22 smithi116 ceph-mon[32207]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:22 smithi143 ceph-mon[35965]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T09:15:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:22 smithi143 ceph-mon[35965]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T09:15:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:22 smithi143 ceph-mon[35965]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:15:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:22 smithi143 ceph-mon[35965]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:23.568 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:15:23.234+0000 7fe30b682700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T09:15:23.569 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:15:23.240+0000 7fe30346d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:15:23.659 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T09:15:23.659 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T09:15:23.659 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T09:15:23.659 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T09:15:23.659 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113114 s, 927 MB/s 2024-03-24T09:15:23.659 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T09:15:24.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-mon[32207]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:15:24.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-mon[32207]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T09:15:24.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:23 smithi116 ceph-mon[32207]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' 2024-03-24T09:15:24.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:23 smithi117 ceph-mon[35788]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:15:24.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:23 smithi117 ceph-mon[35788]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T09:15:24.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:23 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:23 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:23 smithi117 ceph-mon[35788]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' 2024-03-24T09:15:24.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:23 smithi143 ceph-mon[35965]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:15:24.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:23 smithi143 ceph-mon[35965]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T09:15:24.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:23 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:23 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:23 smithi143 ceph-mon[35965]: from='osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673]' entity='osd.3' 2024-03-24T09:15:24.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_3 2024-03-24T09:15:24.610 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:15:24.981 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:15:24.981 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:15:24.981 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.981 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] boot 2024-03-24T09:15:24.981 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T09:15:24.981 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:24.982 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:24.983 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] boot 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:25.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: osd.3 [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] boot 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: osdmap e25: 4 total, 4 up, 4 in 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:25.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:25.931 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:25 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi116 2024-03-24T09:15:25.931 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:25 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-24T09:15:25.932 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:25 smithi117 ceph-mon[35788]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T09:15:26.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 -- 172.21.15.117:0/2197575208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92581008b0 msgr2=0x7f9258100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:26.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 --2- 172.21.15.117:0/2197575208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92581008b0 0x7f9258100cd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f92480099c0 tx=0x7f9248009cd0 comp rx=0 tx=0).stop 2024-03-24T09:15:26.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 -- 172.21.15.117:0/2197575208 shutdown_connections 2024-03-24T09:15:26.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 --2- 172.21.15.117:0/2197575208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9258102450 0x7f9258108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:26.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 --2- 172.21.15.117:0/2197575208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92581008b0 0x7f9258100cd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:26.159 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 --2- 172.21.15.117:0/2197575208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 0x7f9258101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 -- 172.21.15.117:0/2197575208 >> 172.21.15.117:0/2197575208 conn(0x7f92580fbe70 msgr2=0x7f92580fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 -- 172.21.15.117:0/2197575208 shutdown_connections 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.155+0000 7f925f0ea700 1 -- 172.21.15.117:0/2197575208 wait complete. 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 Processor -- start 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 -- start start 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92581008b0 0x7f925810a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 0x7f925819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 0x7f925819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:26.160 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f925819fd90 con 0x7f9258101ab0 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f925819ff00 con 0x7f9258102450 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925f0ea700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f92581a00c0 con 0x7f92581008b0 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925d687700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 0x7f925819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925d687700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 0x7f925819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47340/0 (socket says 172.21.15.117:47340) 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925d687700 1 -- 172.21.15.117:0/3393291744 learned_addr learned my addr 172.21.15.117:0/3393291744 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925ce86700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92581008b0 0x7f925810a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f924ffff700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 0x7f925819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.156+0000 7f925d687700 1 -- 172.21.15.117:0/3393291744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92581008b0 msgr2=0x7f925810a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:26.161 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925d687700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92581008b0 0x7f925810a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925d687700 1 -- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 msgr2=0x7f925819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925d687700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 0x7f925819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925d687700 1 -- 172.21.15.117:0/3393291744 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92480095f0 con 0x7f9258102450 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f924ffff700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 0x7f925819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925d687700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 0x7f925819f5f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f925000b970 tx=0x7f925000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925000c930 con 0x7f9258102450 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92581a03a0 con 0x7f9258102450 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9250014440 con 0x7f9258102450 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500075d0 con 0x7f9258102450 2024-03-24T09:15:26.162 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.157+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92581a0920 con 0x7f9258102450 2024-03-24T09:15:26.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.158+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9258107d90 con 0x7f9258102450 2024-03-24T09:15:26.163 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.159+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f92500145b0 con 0x7f9258102450 2024-03-24T09:15:26.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.159+0000 7f92567fc700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f92440a2150 0x7f92440a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:26.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.159+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f9250006e40 con 0x7f9258102450 2024-03-24T09:15:26.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.160+0000 7f925ce86700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f92440a2150 0x7f92440a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:26.164 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.160+0000 7f925ce86700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f92440a2150 0x7f92440a4610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9248006010 tx=0x7f92480037b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:15:26.165 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.164+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9250100280 con 0x7f9258102450 2024-03-24T09:15:26.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:25 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi116 2024-03-24T09:15:26.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:25 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-24T09:15:26.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:25 smithi143 ceph-mon[35965]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T09:15:26.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:25 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi116 2024-03-24T09:15:26.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:25 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-24T09:15:26.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:25 smithi116 ceph-mon[32207]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T09:15:26.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:26.368+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f925802dc00 con 0x7f92440a2150 2024-03-24T09:15:27.102 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:26 smithi117 ceph-mon[35788]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T09:15:27.102 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:26 smithi117 ceph-mon[35788]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:27.102 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:27.102 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:27.103 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:27.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:26 smithi116 ceph-mon[32207]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T09:15:27.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:26 smithi116 ceph-mon[32207]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:27.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:27.219 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:27.220 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:26 smithi143 ceph-mon[35965]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T09:15:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:26 smithi143 ceph-mon[35965]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:28.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:27 smithi117 ceph-mon[35788]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:28.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:27 smithi117 ceph-mon[35788]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T09:15:28.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:27 smithi143 ceph-mon[35965]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:28.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:27 smithi143 ceph-mon[35965]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T09:15:28.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:27 smithi116 ceph-mon[32207]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:28.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:27 smithi116 ceph-mon[32207]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T09:15:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:28 smithi143 ceph-mon[35965]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-24T09:15:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:28 smithi117 ceph-mon[35788]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-24T09:15:29.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:28 smithi116 ceph-mon[32207]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-24T09:15:30.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:29 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/4271284658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]: dispatch 2024-03-24T09:15:30.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:29 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]: dispatch 2024-03-24T09:15:30.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:29 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]': finished 2024-03-24T09:15:30.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:29 smithi143 ceph-mon[35965]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T09:15:30.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:29 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:30.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:29 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/2746884968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:29 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/4271284658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]: dispatch 2024-03-24T09:15:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:29 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]: dispatch 2024-03-24T09:15:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:29 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]': finished 2024-03-24T09:15:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:29 smithi117 ceph-mon[35788]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T09:15:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:29 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:29 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/2746884968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:30.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:29 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/4271284658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]: dispatch 2024-03-24T09:15:30.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:29 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]: dispatch 2024-03-24T09:15:30.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:29 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a944b4a-1532-4713-a537-792acac8425c"}]': finished 2024-03-24T09:15:30.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:29 smithi116 ceph-mon[32207]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T09:15:30.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:29 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:30.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:29 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/2746884968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:30 smithi143 ceph-mon[35965]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-24T09:15:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:30 smithi117 ceph-mon[35788]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-24T09:15:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:30 smithi116 ceph-mon[32207]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-24T09:15:33.050 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:32 smithi117 ceph-mon[35788]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:15:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:32 smithi143 ceph-mon[35965]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:15:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:32 smithi116 ceph-mon[32207]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:15:34.991 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:34 smithi117 ceph-mon[35788]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:15:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:34 smithi143 ceph-mon[35965]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:15:35.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:34 smithi116 ceph-mon[32207]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:15:37.208 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:36 smithi117 ceph-mon[35788]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:37.209 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T09:15:37.209 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:36 smithi143 ceph-mon[35965]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T09:15:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:37.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:36 smithi116 ceph-mon[32207]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T09:15:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:38.210 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:37 smithi117 ceph-mon[35788]: Deploying daemon osd.4 on smithi117 2024-03-24T09:15:38.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:37 smithi143 ceph-mon[35965]: Deploying daemon osd.4 on smithi117 2024-03-24T09:15:38.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:37 smithi116 ceph-mon[32207]: Deploying daemon osd.4 on smithi117 2024-03-24T09:15:39.260 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:38 smithi117 ceph-mon[35788]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:38 smithi143 ceph-mon[35965]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:38 smithi116 ceph-mon[32207]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:41.050 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:40 smithi117 ceph-mon[35788]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:40 smithi143 ceph-mon[35965]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:40 smithi116 ceph-mon[32207]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:43.259 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:42 smithi117 ceph-mon[35788]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:43.259 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:43.259 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:43.260 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:42 smithi143 ceph-mon[35965]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:42 smithi116 ceph-mon[32207]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:45.270 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:44 smithi117 ceph-mon[35788]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:45.270 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.270 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.270 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:45.270 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:45.270 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:44 smithi143 ceph-mon[35965]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:44 smithi116 ceph-mon[32207]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:45.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:45.636 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 4 on host 'smithi117' 2024-03-24T09:15:45.636 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.635+0000 7f92567fc700 1 -- 172.21.15.117:0/3393291744 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f925802dc00 con 0x7f92440a2150 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f92440a2150 msgr2=0x7f92440a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f92440a2150 0x7f92440a4610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9248006010 tx=0x7f92480037b0 comp rx=0 tx=0).stop 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 msgr2=0x7f925819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 0x7f925819f5f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f925000b970 tx=0x7f925000bc80 comp rx=0 tx=0).stop 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 shutdown_connections 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9258102450 0x7f925819f5f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:45.638 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92581008b0 0x7f925810a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:45.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f92440a2150 0x7f92440a4610 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:45.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 --2- 172.21.15.117:0/3393291744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9258101ab0 0x7f925819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:45.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 >> 172.21.15.117:0/3393291744 conn(0x7f92580fbe70 msgr2=0x7f9258104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:15:45.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 shutdown_connections 2024-03-24T09:15:45.639 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:15:45.638+0000 7f925f0ea700 1 -- 172.21.15.117:0/3393291744 wait complete. 2024-03-24T09:15:46.315 DEBUG:teuthology.orchestra.run.smithi117:osd.4> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T09:15:46.317 INFO:tasks.cephadm:Deploying osd.5 on smithi143 with /dev/vg_nvme/lv_3... 2024-03-24T09:15:46.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T09:15:46.563 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:15:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:46 smithi117 ceph-mon[35788]: from='osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:15:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:46 smithi117 ceph-mon[35788]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:15:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:46 smithi117 ceph-mon[35788]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:46.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:46.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:46.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:46.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:46 smithi143 ceph-mon[35965]: from='osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:15:46.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:46 smithi143 ceph-mon[35965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:15:46.920 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:46 smithi143 ceph-mon[35965]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:47.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:47.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:47.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:15:47.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:46 smithi116 ceph-mon[32207]: from='osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:15:47.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:46 smithi116 ceph-mon[32207]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T09:15:47.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:46 smithi116 ceph-mon[32207]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:47.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:47 smithi143 ceph-mon[35965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T09:15:47.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:47 smithi143 ceph-mon[35965]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T09:15:47.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:47 smithi143 ceph-mon[35965]: from='osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:15:47.838 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:47 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:47.839 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:47 smithi143 ceph-mon[35965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:15:47.974 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:47 smithi117 ceph-mon[35788]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T09:15:47.975 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:47 smithi117 ceph-mon[35788]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T09:15:47.975 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:47 smithi117 ceph-mon[35788]: from='osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:15:47.975 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:47 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:47.975 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:47 smithi117 ceph-mon[35788]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:15:48.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:47 smithi116 ceph-mon[32207]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T09:15:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:47 smithi116 ceph-mon[32207]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T09:15:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:47 smithi116 ceph-mon[32207]: from='osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:15:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:47 smithi116 ceph-mon[32207]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:15:48.316 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4[47090]: 2024-03-24T09:15:48.027+0000 7f27ef8db700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T09:15:48.316 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4[47090]: 2024-03-24T09:15:48.032+0000 7f27e76c6700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:15:48.410 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T09:15:48.410 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T09:15:48.410 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T09:15:48.410 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T09:15:48.410 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113766 s, 922 MB/s 2024-03-24T09:15:48.411 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T09:15:49.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:48 smithi143 ceph-mon[35965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:15:49.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:48 smithi143 ceph-mon[35965]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T09:15:49.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:48 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:48 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:48 smithi143 ceph-mon[35965]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:49.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:48 smithi143 ceph-mon[35965]: from='osd.4 ' entity='osd.4' 2024-03-24T09:15:49.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-mon[35788]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:15:49.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-mon[35788]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T09:15:49.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-mon[35788]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:49.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:48 smithi117 ceph-mon[35788]: from='osd.4 ' entity='osd.4' 2024-03-24T09:15:49.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:48 smithi116 ceph-mon[32207]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:15:49.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:48 smithi116 ceph-mon[32207]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T09:15:49.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:48 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:48 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:48 smithi116 ceph-mon[32207]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-24T09:15:49.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:48 smithi116 ceph-mon[32207]: from='osd.4 ' entity='osd.4' 2024-03-24T09:15:49.091 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_3 2024-03-24T09:15:49.331 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:15:49.702 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] boot 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:49.703 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.704 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:49.704 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:49.704 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:49 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:15:49.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] boot 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:49.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:49.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:49 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: osd.4 [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] boot 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:50.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:15:50.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:50.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:50.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:15:50.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:49 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:15:50.737 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:50 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi117 2024-03-24T09:15:50.737 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:50 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi117 to 8544M 2024-03-24T09:15:50.737 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:50 smithi143 ceph-mon[35965]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:50.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 -- 172.21.15.143:0/3598330558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 msgr2=0x7fbe90102650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:50.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/3598330558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90102650 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009990 tx=0x7fbe80009ca0 comp rx=0 tx=0).stop 2024-03-24T09:15:50.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 -- 172.21.15.143:0/3598330558 shutdown_connections 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/3598330558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe90103dd0 0x7fbe9010a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/3598330558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90102650 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/3598330558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90103430 0x7fbe90103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 -- 172.21.15.143:0/3598330558 >> 172.21.15.143:0/3598330558 conn(0x7fbe900fd810 msgr2=0x7fbe900ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 -- 172.21.15.143:0/3598330558 shutdown_connections 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.932+0000 7fbe95a5c700 1 -- 172.21.15.143:0/3598330558 wait complete. 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 Processor -- start 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 -- start start 2024-03-24T09:15:50.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe90103430 0x7fbe90198660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90103dd0 0x7fbe9019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9019fcb0 con 0x7fbe90103dd0 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9019fdf0 con 0x7fbe90103430 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe95a5c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9019ff80 con 0x7fbe90102230 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:36910/0 (socket says 172.21.15.143:36910) 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 -- 172.21.15.143:0/178558197 learned_addr learned my addr 172.21.15.143:0/178558197 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8f7fe700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90103dd0 0x7fbe9019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:50.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe87fff700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe90103430 0x7fbe90198660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 -- 172.21.15.143:0/178558197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe90103430 msgr2=0x7fbe90198660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe90103430 0x7fbe90198660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 -- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90103dd0 msgr2=0x7fbe9019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90103dd0 0x7fbe9019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 -- 172.21.15.143:0/178558197 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe800095f0 con 0x7fbe90102230 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.933+0000 7fbe8effd700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90198120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbe80000c00 tx=0x7fbe800047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.934+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8001d070 con 0x7fbe90102230 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.934+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbe80004010 con 0x7fbe90102230 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.934+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe80022bf0 con 0x7fbe90102230 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.934+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe901a0200 con 0x7fbe90102230 2024-03-24T09:15:50.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.934+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe901a06f0 con 0x7fbe90102230 2024-03-24T09:15:50.939 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.935+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe90109760 con 0x7fbe90102230 2024-03-24T09:15:50.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.936+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fbe80003b60 con 0x7fbe90102230 2024-03-24T09:15:50.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.936+0000 7fbe94a5a700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbe7c0a21d0 0x7fbe7c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:15:50.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.936+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fbe800ce140 con 0x7fbe90102230 2024-03-24T09:15:50.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.936+0000 7fbe87fff700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbe7c0a21d0 0x7fbe7c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:15:50.940 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.937+0000 7fbe87fff700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbe7c0a21d0 0x7fbe7c0a4690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbe78005950 tx=0x7fbe780058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:15:50.941 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:50.941+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbe80098210 con 0x7fbe90102230 2024-03-24T09:15:51.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:50 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi117 2024-03-24T09:15:51.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:50 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi117 to 8544M 2024-03-24T09:15:51.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:50 smithi117 ceph-mon[35788]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:51.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:50 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi117 2024-03-24T09:15:51.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:50 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi117 to 8544M 2024-03-24T09:15:51.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:50 smithi116 ceph-mon[32207]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:51.143 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:15:51.144+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fbe9002dc00 con 0x7fbe7c0a21d0 2024-03-24T09:15:51.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:51 smithi143 ceph-mon[35965]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T09:15:51.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:51.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:51.935 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:51 smithi117 ceph-mon[35788]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T09:15:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:52.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:51 smithi116 ceph-mon[32207]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T09:15:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:15:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:15:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:15:52.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:52 smithi143 ceph-mon[35965]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:52.959 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:52 smithi143 ceph-mon[35965]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:53.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:52 smithi117 ceph-mon[35788]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:52 smithi117 ceph-mon[35788]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:53.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:52 smithi116 ceph-mon[32207]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:15:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:52 smithi116 ceph-mon[32207]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:54.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/4088016165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]: dispatch 2024-03-24T09:15:54.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:53 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]: dispatch 2024-03-24T09:15:54.172 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:53 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]': finished 2024-03-24T09:15:54.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:53 smithi143 ceph-mon[35965]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T09:15:54.173 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:53 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:15:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/4088016165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]: dispatch 2024-03-24T09:15:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:53 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]: dispatch 2024-03-24T09:15:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:53 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]': finished 2024-03-24T09:15:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:53 smithi117 ceph-mon[35788]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T09:15:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:53 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:15:54.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/4088016165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]: dispatch 2024-03-24T09:15:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:53 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]: dispatch 2024-03-24T09:15:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:53 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaadd9c9-b1bc-4ca0-bb20-760ee737abf9"}]': finished 2024-03-24T09:15:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:53 smithi116 ceph-mon[32207]: osdmap e35: 6 total, 5 up, 6 in 2024-03-24T09:15:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:53 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:15:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:54 smithi143 ceph-mon[35965]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:55.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/2112948481' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:54 smithi117 ceph-mon[35788]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/2112948481' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:54 smithi116 ceph-mon[32207]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:55.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/2112948481' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:15:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:56 smithi143 ceph-mon[35965]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:56 smithi117 ceph-mon[35788]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:57.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:56 smithi116 ceph-mon[32207]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:15:58 smithi143 ceph-mon[35965]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:15:58 smithi117 ceph-mon[35788]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:15:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:15:58 smithi116 ceph-mon[32207]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:01.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:00 smithi143 ceph-mon[35965]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:01.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T09:16:01.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:00 smithi117 ceph-mon[35788]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T09:16:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:00 smithi116 ceph-mon[32207]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T09:16:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:02.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:01 smithi143 ceph-mon[35965]: Deploying daemon osd.5 on smithi143 2024-03-24T09:16:02.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:01 smithi117 ceph-mon[35788]: Deploying daemon osd.5 on smithi143 2024-03-24T09:16:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:01 smithi116 ceph-mon[32207]: Deploying daemon osd.5 on smithi143 2024-03-24T09:16:03.207 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:02 smithi143 ceph-mon[35965]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:02 smithi117 ceph-mon[35788]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:02 smithi116 ceph-mon[32207]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:05.202 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:04 smithi143 ceph-mon[35965]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:04 smithi117 ceph-mon[35788]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:04 smithi116 ceph-mon[32207]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:07.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:06 smithi143 ceph-mon[35965]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:07.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:07.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:07.075 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:06 smithi117 ceph-mon[35788]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:06 smithi116 ceph-mon[32207]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:09.247 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:08 smithi143 ceph-mon[35965]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:08 smithi117 ceph-mon[35788]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:09.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:08 smithi116 ceph-mon[32207]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:10.312 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.310+0000 7fbe94a5a700 1 -- 172.21.15.143:0/178558197 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbe9002dc00 con 0x7fbe7c0a21d0 2024-03-24T09:16:10.312 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 5 on host 'smithi143' 2024-03-24T09:16:10.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbe7c0a21d0 msgr2=0x7fbe7c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:10.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbe7c0a21d0 0x7fbe7c0a4690 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbe78005950 tx=0x7fbe780058e0 comp rx=0 tx=0).stop 2024-03-24T09:16:10.313 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 msgr2=0x7fbe90198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90198120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbe80000c00 tx=0x7fbe800047b0 comp rx=0 tx=0).stop 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 shutdown_connections 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbe7c0a21d0 0x7fbe7c0a4690 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe90103dd0 0x7fbe9019f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe90102230 0x7fbe90198120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 --2- 172.21.15.143:0/178558197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbe90103430 0x7fbe90198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 >> 172.21.15.143:0/178558197 conn(0x7fbe900fd810 msgr2=0x7fbe901068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 shutdown_connections 2024-03-24T09:16:10.314 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:16:10.313+0000 7fbe95a5c700 1 -- 172.21.15.143:0/178558197 wait complete. 2024-03-24T09:16:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:10.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:16:10.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:16:10.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:16:10.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:16:10.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:10.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:10.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:10.961 DEBUG:teuthology.orchestra.run.smithi143:osd.5> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T09:16:10.963 INFO:tasks.cephadm:Deploying osd.6 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-24T09:16:10.963 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T09:16:11.211 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:16:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:12 smithi117 ceph-mon[35788]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T09:16:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:12 smithi117 ceph-mon[35788]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T09:16:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:12 smithi117 ceph-mon[35788]: from='osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:16:12.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:12 smithi117 ceph-mon[35788]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:16:12.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:12 smithi117 ceph-mon[35788]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:12.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:12 smithi116 ceph-mon[32207]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T09:16:12.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:12 smithi116 ceph-mon[32207]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T09:16:12.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:12.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:12 smithi116 ceph-mon[32207]: from='osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:16:12.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:12 smithi116 ceph-mon[32207]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:16:12.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:12 smithi116 ceph-mon[32207]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-mon[35965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T09:16:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-mon[35965]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T09:16:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-mon[35965]: from='osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:16:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-mon[35965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:16:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-mon[35965]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:13.128 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T09:16:13.128 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T09:16:13.128 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T09:16:13.128 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T09:16:13.128 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113418 s, 925 MB/s 2024-03-24T09:16:13.129 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T09:16:13.156 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:16:12.803+0000 7f526c9d3700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T09:16:13.156 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:16:12 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:16:12.811+0000 7f52647be700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:16:13.439 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:16:13.439 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:16:13.439 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:16:13.440 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T09:16:13.440 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:13.440 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:13.440 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:13 smithi116 ceph-mon[32207]: from='osd.5 ' entity='osd.5' 2024-03-24T09:16:13.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:16:13.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:16:13.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:16:13.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T09:16:13.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:13.458 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:13.459 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:13 smithi143 ceph-mon[35965]: from='osd.5 ' entity='osd.5' 2024-03-24T09:16:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:16:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:16:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:16:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T09:16:13.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:13.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:13.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:13 smithi117 ceph-mon[35788]: from='osd.5 ' entity='osd.5' 2024-03-24T09:16:13.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-24T09:16:14.059 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:16:14.413 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:14.413 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] boot 2024-03-24T09:16:14.413 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T09:16:14.413 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:14.414 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:14 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.440 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] boot 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:14.441 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.442 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:14.442 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:14.442 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:14 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: osd.5 [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] boot 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: osdmap e38: 6 total, 6 up, 6 in 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:14.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:14 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:15.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 -- 172.21.15.116:0/1834269386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8101130 msgr2=0x7f29d8101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:15.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 --2- 172.21.15.116:0/1834269386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8101130 0x7f29d8101550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f29c80099c0 tx=0x7f29c8009cd0 comp rx=0 tx=0).stop 2024-03-24T09:16:15.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 -- 172.21.15.116:0/1834269386 shutdown_connections 2024-03-24T09:16:15.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 --2- 172.21.15.116:0/1834269386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29d8102cd0 0x7f29d8109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:15.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 --2- 172.21.15.116:0/1834269386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8101130 0x7f29d8101550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 --2- 172.21.15.116:0/1834269386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8102330 0x7f29d8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 -- 172.21.15.116:0/1834269386 >> 172.21.15.116:0/1834269386 conn(0x7f29d80fc6d0 msgr2=0x7f29d80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 -- 172.21.15.116:0/1834269386 shutdown_connections 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.752+0000 7f29df7ee700 1 -- 172.21.15.116:0/1834269386 wait complete. 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 Processor -- start 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 -- start start 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29d8101130 0x7f29d819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 0x7f29d819a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8102cd0 0x7f29d81a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:15.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29d81a1f40 con 0x7f29d8102330 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29d81a2080 con 0x7f29d8101130 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29df7ee700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f29d81a2210 con 0x7f29d8102cd0 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 0x7f29d819a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 0x7f29d819a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35666/0 (socket says 172.21.15.116:35666) 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 -- 172.21.15.116:0/240042538 learned_addr learned my addr 172.21.15.116:0/240042538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dd58a700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29d8101130 0x7f29d819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:15.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29ddd8b700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8102cd0 0x7f29d81a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 -- 172.21.15.116:0/240042538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8102cd0 msgr2=0x7f29d81a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8102cd0 0x7f29d81a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 -- 172.21.15.116:0/240042538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29d8101130 msgr2=0x7f29d819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29d8101130 0x7f29d819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 -- 172.21.15.116:0/240042538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c80095f0 con 0x7f29d8102330 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.753+0000 7f29dcd89700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 0x7f29d819a940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f29d400b560 tx=0x7f29d400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.754+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4011830 con 0x7f29d8102330 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.754+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29d4011e40 con 0x7f29d8102330 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.754+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d81a24f0 con 0x7f29d8102330 2024-03-24T09:16:15.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.754+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d4010e40 con 0x7f29d8102330 2024-03-24T09:16:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.754+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d81a2a40 con 0x7f29d8102330 2024-03-24T09:16:15.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.755+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d8068ec0 con 0x7f29d8102330 2024-03-24T09:16:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.756+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f29d4011990 con 0x7f29d8102330 2024-03-24T09:16:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.757+0000 7f29ce7fc700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29c40a21d0 0x7f29c40a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.757+0000 7f29dd58a700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29c40a21d0 0x7f29c40a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.757+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f29d40ccd00 con 0x7f29d8102330 2024-03-24T09:16:15.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.757+0000 7f29dd58a700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29c40a21d0 0x7f29c40a4690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009990 tx=0x7f29c8009f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:16:15.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.761+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f29d4096b70 con 0x7f29d8102330 2024-03-24T09:16:15.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:15.968+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f29d806a5c0 con 0x7f29c40a21d0 2024-03-24T09:16:16.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:15 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi143 2024-03-24T09:16:16.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:15 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi143 to 8544M 2024-03-24T09:16:16.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:15 smithi143 ceph-mon[35965]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T09:16:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:15 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi143 2024-03-24T09:16:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:15 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi143 to 8544M 2024-03-24T09:16:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:15 smithi117 ceph-mon[35788]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T09:16:16.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:15 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi143 2024-03-24T09:16:16.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:15 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi143 to 8544M 2024-03-24T09:16:16.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:15 smithi116 ceph-mon[32207]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T09:16:17.255 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:16 smithi116 ceph-mon[32207]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T09:16:17.255 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:16 smithi116 ceph-mon[32207]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:17.255 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:16 smithi116 ceph-mon[32207]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:16:17.256 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:16:17.256 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:16:17.256 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:16 smithi143 ceph-mon[35965]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T09:16:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:16 smithi143 ceph-mon[35965]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:16 smithi143 ceph-mon[35965]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:16:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:16:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:16:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:16 smithi117 ceph-mon[35788]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T09:16:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:16 smithi117 ceph-mon[35788]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:16 smithi117 ceph-mon[35788]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:16:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:16:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:16:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:18.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:17 smithi143 ceph-mon[35965]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T09:16:18.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:17 smithi117 ceph-mon[35788]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T09:16:18.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:17 smithi116 ceph-mon[32207]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T09:16:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:18 smithi143 ceph-mon[35965]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-24T09:16:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:18 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1583551246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09e3ad38-6286-4d42-9ddc-ccfa80ace379"}]: dispatch 2024-03-24T09:16:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:18 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1583551246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09e3ad38-6286-4d42-9ddc-ccfa80ace379"}]': finished 2024-03-24T09:16:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:18 smithi143 ceph-mon[35965]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T09:16:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:18 smithi117 ceph-mon[35788]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-24T09:16:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:18 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1583551246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09e3ad38-6286-4d42-9ddc-ccfa80ace379"}]: dispatch 2024-03-24T09:16:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:18 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1583551246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09e3ad38-6286-4d42-9ddc-ccfa80ace379"}]': finished 2024-03-24T09:16:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:18 smithi117 ceph-mon[35788]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T09:16:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:19.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:18 smithi116 ceph-mon[32207]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 143 KiB/s, 0 objects/s recovering 2024-03-24T09:16:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:18 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1583551246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09e3ad38-6286-4d42-9ddc-ccfa80ace379"}]: dispatch 2024-03-24T09:16:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:18 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1583551246' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09e3ad38-6286-4d42-9ddc-ccfa80ace379"}]': finished 2024-03-24T09:16:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:18 smithi116 ceph-mon[32207]: osdmap e43: 7 total, 6 up, 7 in 2024-03-24T09:16:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:20.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:19 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3221274586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:16:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:19 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3221274586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:16:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:19 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3221274586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:16:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:20 smithi143 ceph-mon[35965]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-24T09:16:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:20 smithi117 ceph-mon[35788]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-24T09:16:21.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:20 smithi116 ceph-mon[32207]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-03-24T09:16:22.989 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:22 smithi116 ceph-mon[32207]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:16:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:22 smithi143 ceph-mon[35965]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:16:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:22 smithi117 ceph-mon[35788]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:16:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:24 smithi143 ceph-mon[35965]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:16:25.276 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:24 smithi116 ceph-mon[32207]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:16:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:24 smithi117 ceph-mon[35788]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:16:27.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:26 smithi116 ceph-mon[32207]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:27.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T09:16:27.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:27.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:26 smithi116 ceph-mon[32207]: Deploying daemon osd.6 on smithi116 2024-03-24T09:16:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:26 smithi143 ceph-mon[35965]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T09:16:27.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:27.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:26 smithi143 ceph-mon[35965]: Deploying daemon osd.6 on smithi116 2024-03-24T09:16:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:26 smithi117 ceph-mon[35788]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T09:16:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:26 smithi117 ceph-mon[35788]: Deploying daemon osd.6 on smithi116 2024-03-24T09:16:29.006 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:28 smithi116 ceph-mon[32207]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:28 smithi143 ceph-mon[35965]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:28 smithi117 ceph-mon[35788]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:30 smithi143 ceph-mon[35965]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:31.296 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:30 smithi116 ceph-mon[32207]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:30 smithi117 ceph-mon[35788]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:33.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:33.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:33.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:33.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:32 smithi143 ceph-mon[35965]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:33.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:33.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:33.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:33.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:32 smithi116 ceph-mon[32207]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:33.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:33.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:33.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:33.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:32 smithi117 ceph-mon[35788]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:34 smithi143 ceph-mon[35965]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:34 smithi116 ceph-mon[32207]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:35.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:35.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:35.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:34 smithi117 ceph-mon[35788]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:35.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.729+0000 7f29ce7fc700 1 -- 172.21.15.116:0/240042538 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f29d806a5c0 con 0x7f29c40a21d0 2024-03-24T09:16:35.730 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 6 on host 'smithi116' 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29c40a21d0 msgr2=0x7f29c40a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29c40a21d0 0x7f29c40a4690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009990 tx=0x7f29c8009f10 comp rx=0 tx=0).stop 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 msgr2=0x7f29d819a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 0x7f29d819a940 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f29d400b560 tx=0x7f29d400b870 comp rx=0 tx=0).stop 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 shutdown_connections 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29d8102cd0 0x7f29d81a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29d8101130 0x7f29d819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29c40a21d0 0x7f29c40a4690 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 --2- 172.21.15.116:0/240042538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29d8102330 0x7f29d819a940 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 >> 172.21.15.116:0/240042538 conn(0x7f29d80fc6d0 msgr2=0x7f29d8069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:16:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 shutdown_connections 2024-03-24T09:16:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:16:35.731+0000 7f29df7ee700 1 -- 172.21.15.116:0/240042538 wait complete. 2024-03-24T09:16:36.052 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:35 smithi116 ceph-mon[32207]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T09:16:36.052 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:36.052 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:36.052 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:36.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:35 smithi143 ceph-mon[35965]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T09:16:36.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:36.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:36.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:35 smithi117 ceph-mon[35788]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T09:16:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:36.537 DEBUG:teuthology.orchestra.run.smithi116:osd.6> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T09:16:36.540 INFO:tasks.cephadm:Deploying osd.7 on smithi117 with /dev/vg_nvme/lv_2... 2024-03-24T09:16:36.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T09:16:36.786 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:16:37.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-mon[32207]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:37.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-mon[32207]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T09:16:37.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-mon[32207]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T09:16:37.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-mon[32207]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:16:37.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:37.069 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:16:36.952+0000 7f36e6f2e700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T09:16:37.069 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:16:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:16:36.962+0000 7f36e2d21700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:16:37.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:36 smithi117 ceph-mon[35788]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:37.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:36 smithi117 ceph-mon[35788]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T09:16:37.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:36 smithi117 ceph-mon[35788]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T09:16:37.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:36 smithi117 ceph-mon[35788]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:16:37.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:36 smithi143 ceph-mon[35965]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:36 smithi143 ceph-mon[35965]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T09:16:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:36 smithi143 ceph-mon[35965]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T09:16:37.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:36 smithi143 ceph-mon[35965]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:16:37.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:37 smithi116 ceph-mon[32207]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:16:38.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:37 smithi116 ceph-mon[32207]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T09:16:38.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:37 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:37 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.011 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:37 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.172 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:37 smithi117 ceph-mon[35788]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:16:38.172 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:37 smithi117 ceph-mon[35788]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T09:16:38.172 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:37 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.172 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:37 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.173 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:37 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:37 smithi143 ceph-mon[35965]: from='osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:16:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:37 smithi143 ceph-mon[35965]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T09:16:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:37 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:37 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:37 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:38.736 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T09:16:38.736 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T09:16:38.736 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T09:16:38.736 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T09:16:38.736 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127286 s, 824 MB/s 2024-03-24T09:16:38.736 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T09:16:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:38 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:16:39.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:38 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:16:39.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:38 smithi143 ceph-mon[35965]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:39.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:38 smithi143 ceph-mon[35965]: osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] boot 2024-03-24T09:16:39.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:38 smithi143 ceph-mon[35965]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T09:16:39.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:39.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:38 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:16:39.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:38 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:16:39.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:38 smithi116 ceph-mon[32207]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:39.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:38 smithi116 ceph-mon[32207]: osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] boot 2024-03-24T09:16:39.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:38 smithi116 ceph-mon[32207]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T09:16:39.285 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:38 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:16:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:38 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:16:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:38 smithi117 ceph-mon[35788]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-24T09:16:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:38 smithi117 ceph-mon[35788]: osd.6 [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] boot 2024-03-24T09:16:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:38 smithi117 ceph-mon[35788]: osdmap e46: 7 total, 7 up, 7 in 2024-03-24T09:16:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:16:39.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_2 2024-03-24T09:16:39.657 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:16:40.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T09:16:40.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:39 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:39 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T09:16:40.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:16:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:39 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:41.149 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:40 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi116 2024-03-24T09:16:41.149 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:40 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-24T09:16:41.150 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:40 smithi117 ceph-mon[35788]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:40 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi116 2024-03-24T09:16:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:40 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-24T09:16:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:40 smithi143 ceph-mon[35965]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:40 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi116 2024-03-24T09:16:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:40 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-24T09:16:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:40 smithi116 ceph-mon[32207]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:41.377 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 -- 172.21.15.117:0/804767520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 msgr2=0x7f7674109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:41.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 --2- 172.21.15.117:0/804767520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f7674109570 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f766c00b0d0 tx=0x7f766c00b3e0 comp rx=0 tx=0).stop 2024-03-24T09:16:41.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 -- 172.21.15.117:0/804767520 shutdown_connections 2024-03-24T09:16:41.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 --2- 172.21.15.117:0/804767520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f7674109570 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:41.393 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 --2- 172.21.15.117:0/804767520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7674101120 0x7f7674101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:41.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 --2- 172.21.15.117:0/804767520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7674102320 0x7f7674102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:41.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 -- 172.21.15.117:0/804767520 >> 172.21.15.117:0/804767520 conn(0x7f76740fc6c0 msgr2=0x7f76740feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:16:41.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 -- 172.21.15.117:0/804767520 shutdown_connections 2024-03-24T09:16:41.394 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.376+0000 7f767a3e1700 1 -- 172.21.15.117:0/804767520 wait complete. 2024-03-24T09:16:41.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 Processor -- start 2024-03-24T09:16:41.396 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 -- start start 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7674101120 0x7f76741981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7674102320 0x7f7674198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f767419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f767419fd80 con 0x7f7674101120 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f767419fec0 con 0x7f7674102cc0 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767a3e1700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f76741a0050 con 0x7f7674102320 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767897e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f767419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767897e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f767419f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:35742/0 (socket says 172.21.15.117:35742) 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.377+0000 7f767897e700 1 -- 172.21.15.117:0/947577055 learned_addr learned my addr 172.21.15.117:0/947577055 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f76737fe700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7674102320 0x7f7674198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f7673fff700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7674101120 0x7f76741981f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767897e700 1 -- 172.21.15.117:0/947577055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7674102320 msgr2=0x7f7674198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:41.397 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767897e700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7674102320 0x7f7674198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767897e700 1 -- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7674101120 msgr2=0x7f76741981f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767897e700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7674101120 0x7f76741981f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767897e700 1 -- 172.21.15.117:0/947577055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766c009d00 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767897e700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f767419f7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f766c007bc0 tx=0x7f766c009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c00e040 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76741a02d0 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f766c01dd40 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766c012e80 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.378+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76741a0740 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.379+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7674068ec0 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.380+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f766c01d890 con 0x7f7674102cc0 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.381+0000 7f76717fa700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f765c0a2150 0x7f765c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:16:41.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.381+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f766c0cf3e0 con 0x7f7674102cc0 2024-03-24T09:16:41.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.381+0000 7f7673fff700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f765c0a2150 0x7f765c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:16:41.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.382+0000 7f7673fff700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f765c0a2150 0x7f765c0a4610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7664005950 tx=0x7f76640058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:16:41.399 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.384+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f766c099150 con 0x7f7674102cc0 2024-03-24T09:16:41.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:16:41.585+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f76740fe870 con 0x7f765c0a2150 2024-03-24T09:16:42.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:41 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:16:42.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:41 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:16:42.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:41 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:42.309 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:41 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:16:42.309 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:41 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:16:42.310 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:41 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:42.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:41 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:16:42.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:41 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:16:42.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:41 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:43.232 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:42 smithi117 ceph-mon[35788]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:16:43.232 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:42 smithi117 ceph-mon[35788]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:42 smithi143 ceph-mon[35965]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:16:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:42 smithi143 ceph-mon[35965]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:43.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:42 smithi116 ceph-mon[32207]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:16:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:42 smithi116 ceph-mon[32207]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:44 smithi117 ceph-mon[35788]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:44 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/1920579160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]: dispatch 2024-03-24T09:16:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:44 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]: dispatch 2024-03-24T09:16:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:44 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]': finished 2024-03-24T09:16:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:44 smithi117 ceph-mon[35788]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T09:16:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:16:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:44 smithi143 ceph-mon[35965]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:44 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/1920579160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]: dispatch 2024-03-24T09:16:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:44 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]: dispatch 2024-03-24T09:16:45.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:44 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]': finished 2024-03-24T09:16:45.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:44 smithi143 ceph-mon[35965]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T09:16:45.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:16:45.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:44 smithi116 ceph-mon[32207]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:44 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/1920579160' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]: dispatch 2024-03-24T09:16:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:44 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]: dispatch 2024-03-24T09:16:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:44 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc83d789-61b0-4c26-93ff-c7ddcba3cc21"}]': finished 2024-03-24T09:16:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:44 smithi116 ceph-mon[32207]: osdmap e48: 8 total, 7 up, 8 in 2024-03-24T09:16:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:16:46.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:45 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/1971497490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:16:46.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:45 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/1971497490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:16:46.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:45 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/1971497490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:16:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:46 smithi143 ceph-mon[35965]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:46 smithi117 ceph-mon[35788]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:47.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:46 smithi116 ceph-mon[32207]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:49.272 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:48 smithi117 ceph-mon[35788]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:48 smithi143 ceph-mon[35965]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:48 smithi116 ceph-mon[32207]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:50.297 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:50 smithi117 ceph-mon[35788]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:50.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:50 smithi116 ceph-mon[32207]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:50.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:50 smithi143 ceph-mon[35965]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:52.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T09:16:52.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:51 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T09:16:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:51 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:52.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T09:16:52.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:51 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:16:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:52 smithi143 ceph-mon[35965]: Deploying daemon osd.7 on smithi117 2024-03-24T09:16:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:52 smithi143 ceph-mon[35965]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:53.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:52 smithi117 ceph-mon[35788]: Deploying daemon osd.7 on smithi117 2024-03-24T09:16:53.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:52 smithi117 ceph-mon[35788]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:53.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:52 smithi116 ceph-mon[32207]: Deploying daemon osd.7 on smithi117 2024-03-24T09:16:53.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:52 smithi116 ceph-mon[32207]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:54 smithi143 ceph-mon[35965]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:54 smithi117 ceph-mon[35788]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:54 smithi116 ceph-mon[32207]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:57.194 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:56 smithi117 ceph-mon[35788]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:56 smithi143 ceph-mon[35965]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:57.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:56 smithi116 ceph-mon[32207]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:58.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:58.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:58.401 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:58.402 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:16:58 smithi116 ceph-mon[32207]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:16:58 smithi117 ceph-mon[35788]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:16:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:16:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:16:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:16:58 smithi143 ceph-mon[35965]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:00.901 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.900+0000 7f76717fa700 1 -- 172.21.15.117:0/947577055 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f76740fe870 con 0x7f765c0a2150 2024-03-24T09:17:00.902 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 7 on host 'smithi117' 2024-03-24T09:17:00.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f765c0a2150 msgr2=0x7f765c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:00.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f765c0a2150 0x7f765c0a4610 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7664005950 tx=0x7f76640058e0 comp rx=0 tx=0).stop 2024-03-24T09:17:00.903 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 msgr2=0x7f767419f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f767419f7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f766c007bc0 tx=0x7f766c009440 comp rx=0 tx=0).stop 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 shutdown_connections 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674102cc0 0x7f767419f7b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f765c0a2150 0x7f765c0a4610 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7674101120 0x7f76741981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 --2- 172.21.15.117:0/947577055 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7674102320 0x7f7674198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 >> 172.21.15.117:0/947577055 conn(0x7f76740fc6c0 msgr2=0x7f76740fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 shutdown_connections 2024-03-24T09:17:00.904 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:00.903+0000 7f767a3e1700 1 -- 172.21.15.117:0/947577055 wait complete. 2024-03-24T09:17:01.211 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:00 smithi117 ceph-mon[35788]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:01.211 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:01.211 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:00 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:00 smithi143 ceph-mon[35965]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:00 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:00 smithi116 ceph-mon[32207]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:00 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:01.570 DEBUG:teuthology.orchestra.run.smithi117:osd.7> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T09:17:01.574 INFO:tasks.cephadm:Deploying osd.8 on smithi143 with /dev/vg_nvme/lv_2... 2024-03-24T09:17:01.574 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T09:17:01.826 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:17:02.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:01 smithi143 ceph-mon[35965]: from='osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T09:17:02.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:01 smithi143 ceph-mon[35965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T09:17:02.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:02.194 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:02.298 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:01 smithi117 ceph-mon[35788]: from='osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T09:17:02.298 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:01 smithi117 ceph-mon[35788]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T09:17:02.299 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:02.299 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:01 smithi116 ceph-mon[32207]: from='osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T09:17:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:01 smithi116 ceph-mon[32207]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T09:17:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:03.212 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:02 smithi143 ceph-mon[35965]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:03.212 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:02 smithi143 ceph-mon[35965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T09:17:03.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:02 smithi143 ceph-mon[35965]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T09:17:03.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:02 smithi143 ceph-mon[35965]: from='osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:17:03.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:02 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:03.213 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:02 smithi143 ceph-mon[35965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:17:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:02 smithi117 ceph-mon[35788]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:02 smithi117 ceph-mon[35788]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T09:17:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:02 smithi117 ceph-mon[35788]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T09:17:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:02 smithi117 ceph-mon[35788]: from='osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:17:03.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:02 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:03.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:02 smithi117 ceph-mon[35788]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:17:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:02 smithi116 ceph-mon[32207]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:02 smithi116 ceph-mon[32207]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T09:17:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:02 smithi116 ceph-mon[32207]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T09:17:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:02 smithi116 ceph-mon[32207]: from='osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:17:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:02 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:02 smithi116 ceph-mon[32207]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:17:03.668 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:17:03.415+0000 7fd6d278e700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T09:17:03.668 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:17:03.426+0000 7fd6ca579700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:17:03.886 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T09:17:03.886 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T09:17:03.886 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T09:17:03.886 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T09:17:03.886 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011296 s, 928 MB/s 2024-03-24T09:17:03.886 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: from='osd.7 ' entity='osd.7' 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] boot 2024-03-24T09:17:04.210 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T09:17:04.211 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:03 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:17:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T09:17:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: from='osd.7 ' entity='osd.7' 2024-03-24T09:17:04.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] boot 2024-03-24T09:17:04.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T09:17:04.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:03 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: from='osd.7 ' entity='osd.7' 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: osd.7 [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] boot 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: osdmap e51: 8 total, 8 up, 8 in 2024-03-24T09:17:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:03 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:17:04.539 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_2 2024-03-24T09:17:04.788 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:17:05.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:04 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:17:05.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:04 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:17:05.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:04 smithi143 ceph-mon[35965]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:05.148 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:04 smithi143 ceph-mon[35965]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T09:17:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:04 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:17:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:04 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:17:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:04 smithi117 ceph-mon[35788]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:04 smithi117 ceph-mon[35788]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T09:17:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:04 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:17:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:04 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:17:05.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:04 smithi116 ceph-mon[32207]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-24T09:17:05.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:04 smithi116 ceph-mon[32207]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T09:17:06.419 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 -- 172.21.15.143:0/552969671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 msgr2=0x7f003410e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 --2- 172.21.15.143:0/552969671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f003410e280 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f00240099c0 tx=0x7f0024009cd0 comp rx=0 tx=0).stop 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 -- 172.21.15.143:0/552969671 shutdown_connections 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 --2- 172.21.15.143:0/552969671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003410e7c0 0x7f0034115070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 --2- 172.21.15.143:0/552969671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410cc20 0x7f003410d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 --2- 172.21.15.143:0/552969671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f003410e280 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 -- 172.21.15.143:0/552969671 >> 172.21.15.143:0/552969671 conn(0x7f003406ccd0 msgr2=0x7f003406d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 -- 172.21.15.143:0/552969671 shutdown_connections 2024-03-24T09:17:06.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.417+0000 7f003baf6700 1 -- 172.21.15.143:0/552969671 wait complete. 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 Processor -- start 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 -- start start 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003410cc20 0x7f00341a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f00341a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 0x7f00341aa050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00341aa620 con 0x7f003410de20 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f00341aa760 con 0x7f003410cc20 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003baf6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f00341aa8f0 con 0x7f003410e7c0 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003a093700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 0x7f00341aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003a093700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 0x7f00341aa050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:45340/0 (socket says 172.21.15.143:45340) 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003a093700 1 -- 172.21.15.143:0/1378705795 learned_addr learned my addr 172.21.15.143:0/1378705795 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f0039091700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f00341a2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:06.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f0039892700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003410cc20 0x7f00341a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.418+0000 7f003a093700 1 -- 172.21.15.143:0/1378705795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003410cc20 msgr2=0x7f00341a2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003a093700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003410cc20 0x7f00341a2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003a093700 1 -- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 msgr2=0x7f00341a2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003a093700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f00341a2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003a093700 1 -- 172.21.15.143:0/1378705795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002c009c40 con 0x7f003410e7c0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f0039091700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f00341a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003a093700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 0x7f00341aa050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c3a0 tx=0x7f002c00c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c017070 con 0x7f003410e7c0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f002c00f040 con 0x7f003410e7c0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00240095f0 con 0x7f003410e7c0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002c00e050 con 0x7f003410e7c0 2024-03-24T09:17:06.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.419+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00341aae90 con 0x7f003410e7c0 2024-03-24T09:17:06.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.420+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0034114060 con 0x7f003410e7c0 2024-03-24T09:17:06.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.421+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f002c004bd0 con 0x7f003410e7c0 2024-03-24T09:17:06.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.422+0000 7f002affd700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f00200a21d0 0x7f00200a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:06.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.422+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f002c0cd190 con 0x7f003410e7c0 2024-03-24T09:17:06.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.422+0000 7f0039892700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f00200a21d0 0x7f00200a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:06.424 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.422+0000 7f0039892700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f00200a21d0 0x7f00200a4690 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f003410dc80 tx=0x7f0030009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:17:06.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.425+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f002c096c00 con 0x7f003410e7c0 2024-03-24T09:17:06.628 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:06.628+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f003402dc00 con 0x7f00200a21d0 2024-03-24T09:17:06.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi117 2024-03-24T09:17:06.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi117 to 5696M 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T09:17:06.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:06 smithi117 ceph-mon[35788]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 259 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi117 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi117 to 5696M 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:06.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:06.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:06.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:06.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T09:17:06.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:06 smithi116 ceph-mon[32207]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 259 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:07.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi117 2024-03-24T09:17:07.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:07.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi117 to 5696M 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:07.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T09:17:07.026 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:06 smithi143 ceph-mon[35965]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 259 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:07.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:07 smithi143 ceph-mon[35965]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:17:07.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:07.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:07.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:07.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:07 smithi143 ceph-mon[35965]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T09:17:07.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:07 smithi117 ceph-mon[35788]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:17:07.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:07.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:07.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:07.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:07 smithi117 ceph-mon[35788]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T09:17:07.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:07 smithi116 ceph-mon[32207]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:17:07.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:07.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:07.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:07.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:07 smithi116 ceph-mon[32207]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T09:17:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:08 smithi143 ceph-mon[35965]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T09:17:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:08 smithi143 ceph-mon[35965]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 259 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:08 smithi117 ceph-mon[35788]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T09:17:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:08 smithi117 ceph-mon[35788]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 259 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:09.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:08 smithi116 ceph-mon[32207]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T09:17:09.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:08 smithi116 ceph-mon[32207]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 259 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:09 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/1936314131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]: dispatch 2024-03-24T09:17:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:09 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]: dispatch 2024-03-24T09:17:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:09 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]': finished 2024-03-24T09:17:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:09 smithi143 ceph-mon[35965]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T09:17:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:09 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:09 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/1936314131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]: dispatch 2024-03-24T09:17:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:09 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]: dispatch 2024-03-24T09:17:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:09 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]': finished 2024-03-24T09:17:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:09 smithi117 ceph-mon[35788]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T09:17:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:09 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:10.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:09 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/1936314131' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]: dispatch 2024-03-24T09:17:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:09 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]: dispatch 2024-03-24T09:17:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:09 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9ce166d-03cd-41ec-a11e-f93e301cfb42"}]': finished 2024-03-24T09:17:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:09 smithi116 ceph-mon[32207]: osdmap e56: 9 total, 8 up, 9 in 2024-03-24T09:17:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:10 smithi143 ceph-mon[35965]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:10 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/2293389862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:17:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:10 smithi117 ceph-mon[35788]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:10 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/2293389862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:17:11.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:10 smithi116 ceph-mon[32207]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:11.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:10 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/2293389862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:17:13.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:12 smithi143 ceph-mon[35965]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:12 smithi117 ceph-mon[35788]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:13.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:12 smithi116 ceph-mon[32207]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:15.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:14 smithi143 ceph-mon[35965]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:14 smithi117 ceph-mon[35788]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:15.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:14 smithi116 ceph-mon[32207]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:17.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:16 smithi143 ceph-mon[35965]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:17.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T09:17:17.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:16 smithi117 ceph-mon[35788]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T09:17:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:16 smithi116 ceph-mon[32207]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T09:17:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:18.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:17 smithi143 ceph-mon[35965]: Deploying daemon osd.8 on smithi143 2024-03-24T09:17:18.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:17 smithi117 ceph-mon[35788]: Deploying daemon osd.8 on smithi143 2024-03-24T09:17:18.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:17 smithi116 ceph-mon[32207]: Deploying daemon osd.8 on smithi143 2024-03-24T09:17:19.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:18 smithi143 ceph-mon[35965]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:19.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:18 smithi117 ceph-mon[35788]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:19.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:18 smithi116 ceph-mon[32207]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:21.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:20 smithi143 ceph-mon[35965]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:20 smithi117 ceph-mon[35788]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:21.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:20 smithi116 ceph-mon[32207]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:22 smithi143 ceph-mon[35965]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:22 smithi117 ceph-mon[35788]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:22 smithi116 ceph-mon[32207]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:25.196 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:24 smithi143 ceph-mon[35965]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:24 smithi117 ceph-mon[35788]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:24 smithi116 ceph-mon[32207]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:26.498 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 8 on host 'smithi143' 2024-03-24T09:17:26.499 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.497+0000 7f002affd700 1 -- 172.21.15.143:0/1378705795 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f003402dc00 con 0x7f00200a21d0 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f00200a21d0 msgr2=0x7f00200a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f00200a21d0 0x7f00200a4690 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f003410dc80 tx=0x7f0030009260 comp rx=0 tx=0).stop 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 msgr2=0x7f00341aa050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 0x7f00341aa050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f002c00c3a0 tx=0x7f002c00c6b0 comp rx=0 tx=0).stop 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 shutdown_connections 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f003410e7c0 0x7f00341aa050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003410cc20 0x7f00341a2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f00200a21d0 0x7f00200a4690 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 --2- 172.21.15.143:0/1378705795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f003410de20 0x7f00341a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:26.500 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 >> 172.21.15.143:0/1378705795 conn(0x7f003406ccd0 msgr2=0x7f00341111a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:17:26.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 shutdown_connections 2024-03-24T09:17:26.501 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:17:26.499+0000 7f003baf6700 1 -- 172.21.15.143:0/1378705795 wait complete. 2024-03-24T09:17:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:26.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T09:17:26.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T09:17:26.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:26.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:26 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:26 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T09:17:26.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T09:17:26.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:26.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:26.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:26 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:27.205 DEBUG:teuthology.orchestra.run.smithi143:osd.8> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.8.service 2024-03-24T09:17:27.208 INFO:tasks.cephadm:Deploying osd.9 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-24T09:17:27.208 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T09:17:27.458 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:17:27.757 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-mon[35965]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T09:17:27.757 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-mon[35965]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T09:17:27.757 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:27.757 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-mon[35965]: from='osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:17:27.758 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-mon[35965]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:17:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:27 smithi117 ceph-mon[35788]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T09:17:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:27 smithi117 ceph-mon[35788]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T09:17:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:27 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:27 smithi117 ceph-mon[35788]: from='osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:17:27.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:27 smithi117 ceph-mon[35788]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:17:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:27 smithi116 ceph-mon[32207]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T09:17:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:27 smithi116 ceph-mon[32207]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T09:17:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:27 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:27 smithi116 ceph-mon[32207]: from='osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:17:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:27 smithi116 ceph-mon[32207]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:17:28.024 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:17:27.933+0000 7fafac8f0700 -1 osd.8 0 waiting for initial osdmap 2024-03-24T09:17:28.024 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:17:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:17:27.942+0000 7fafa46db700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:17:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:28 smithi143 ceph-mon[35965]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:17:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:28 smithi143 ceph-mon[35965]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T09:17:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:28 smithi143 ceph-mon[35965]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:28 smithi143 ceph-mon[35965]: from='osd.8 ' entity='osd.8' 2024-03-24T09:17:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:28 smithi117 ceph-mon[35788]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:17:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:28 smithi117 ceph-mon[35788]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T09:17:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:28 smithi117 ceph-mon[35788]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:28 smithi117 ceph-mon[35788]: from='osd.8 ' entity='osd.8' 2024-03-24T09:17:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:28 smithi116 ceph-mon[32207]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:17:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:28 smithi116 ceph-mon[32207]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T09:17:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:28 smithi116 ceph-mon[32207]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 236 MiB used, 715 GiB / 715 GiB avail 2024-03-24T09:17:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:28 smithi116 ceph-mon[32207]: from='osd.8 ' entity='osd.8' 2024-03-24T09:17:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T09:17:29.621 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T09:17:29.622 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-24T09:17:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-24T09:17:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132725 s, 790 MB/s 2024-03-24T09:17:29.622 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-24T09:17:29.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:17:29.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:17:29.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:29.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] boot 2024-03-24T09:17:29.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T09:17:29.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:29.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:29 smithi117 ceph-mon[35788]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T09:17:29.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:17:29.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:17:29.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:29.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] boot 2024-03-24T09:17:29.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T09:17:29.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:29.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:29 smithi116 ceph-mon[32207]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T09:17:30.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:17:30.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:17:30.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:30.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: osd.8 [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] boot 2024-03-24T09:17:30.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: osdmap e59: 9 total, 9 up, 9 in 2024-03-24T09:17:30.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:17:30.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:29 smithi143 ceph-mon[35965]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T09:17:30.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-24T09:17:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:17:30.914 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:30 smithi116 ceph-mon[32207]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T09:17:30.915 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:30 smithi116 ceph-mon[32207]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:30.915 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:30.915 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:30.915 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:30 smithi143 ceph-mon[35965]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T09:17:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:30 smithi143 ceph-mon[35965]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:30 smithi117 ceph-mon[35788]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T09:17:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:30 smithi117 ceph-mon[35788]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:31.977 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi143 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi143 to 5696M 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:31.978 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 -- 172.21.15.116:0/501041070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 msgr2=0x7f12b0100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/501041070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0100930 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009990 tx=0x7f12a4009ca0 comp rx=0 tx=0).stop 2024-03-24T09:17:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 -- 172.21.15.116:0/501041070 shutdown_connections 2024-03-24T09:17:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/501041070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 0x7f12b010a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/501041070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0100930 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/501041070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 0x7f12b0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 -- 172.21.15.116:0/501041070 >> 172.21.15.116:0/501041070 conn(0x7f12b00fbe90 msgr2=0x7f12b00fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 -- 172.21.15.116:0/501041070 shutdown_connections 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 -- 172.21.15.116:0/501041070 wait complete. 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 Processor -- start 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.203+0000 7f12b6f6c700 1 -- start start 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0102e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 0x7f12b01014a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 0x7f12b01019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12b0102160 con 0x7f12b0100510 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f12b01022d0 con 0x7f12b01034f0 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f12b01a2240 con 0x7f12b0103e90 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0102e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48148/0 (socket says 172.21.15.116:48148) 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 -- 172.21.15.116:0/3428806104 learned_addr learned my addr 172.21.15.116:0/3428806104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12affff700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 0x7f12b01014a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 -- 172.21.15.116:0/3428806104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 msgr2=0x7f12b01019e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b5509700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 0x7f12b01019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 0x7f12b01019e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 -- 172.21.15.116:0/3428806104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 msgr2=0x7f12b01014a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 0x7f12b01014a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 -- 172.21.15.116:0/3428806104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a40095f0 con 0x7f12b0100510 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12affff700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 0x7f12b01014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b5509700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 0x7f12b01019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b4d08700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0102e50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f12a4003820 tx=0x7f12a4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a401d070 con 0x7f12b0100510 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b01a24c0 con 0x7f12b0100510 2024-03-24T09:17:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12a4004080 con 0x7f12b0100510 2024-03-24T09:17:32.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.204+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4005290 con 0x7f12b0100510 2024-03-24T09:17:32.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.205+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b01a28d0 con 0x7f12b0100510 2024-03-24T09:17:32.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.205+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b01097d0 con 0x7f12b0100510 2024-03-24T09:17:32.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.207+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f12a4004870 con 0x7f12b0100510 2024-03-24T09:17:32.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.207+0000 7f12adffb700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12a00a2150 0x7f12a00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:32.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.207+0000 7f12affff700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12a00a2150 0x7f12a00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:32.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.207+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f12a40ce890 con 0x7f12b0100510 2024-03-24T09:17:32.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.207+0000 7f12affff700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12a00a2150 0x7f12a00a4610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f129c005fd0 tx=0x7f129c005ee0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:17:32.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.211+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f12a40982a0 con 0x7f12b0100510 2024-03-24T09:17:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi143 2024-03-24T09:17:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:32.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi143 to 5696M 2024-03-24T09:17:32.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T09:17:32.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:32.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:32.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:32.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:32.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi143 2024-03-24T09:17:32.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:32.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi143 to 5696M 2024-03-24T09:17:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T09:17:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:32.415+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f12b002dca0 con 0x7f12a00a2150 2024-03-24T09:17:33.132 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:32 smithi116 ceph-mon[32207]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T09:17:33.133 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:32 smithi116 ceph-mon[32207]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:33.133 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:33.133 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:33.133 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:32 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:32 smithi143 ceph-mon[35965]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T09:17:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:32 smithi143 ceph-mon[35965]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:33.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:32 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:32 smithi117 ceph-mon[35788]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T09:17:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:32 smithi117 ceph-mon[35788]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:33.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:33.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:32 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:34.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:33 smithi116 ceph-mon[32207]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:17:34.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:33 smithi143 ceph-mon[35965]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:17:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:33 smithi117 ceph-mon[35788]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:17:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:34 smithi143 ceph-mon[35965]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:34 smithi117 ceph-mon[35788]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:34 smithi116 ceph-mon[32207]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:36.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:35 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1340772772' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c25c107-ff19-45fb-84e3-c994805faa36"}]: dispatch 2024-03-24T09:17:36.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:35 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1340772772' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c25c107-ff19-45fb-84e3-c994805faa36"}]': finished 2024-03-24T09:17:36.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:35 smithi143 ceph-mon[35965]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T09:17:36.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:36.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:35 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2696698545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:17:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:35 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1340772772' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c25c107-ff19-45fb-84e3-c994805faa36"}]: dispatch 2024-03-24T09:17:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:35 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1340772772' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c25c107-ff19-45fb-84e3-c994805faa36"}]': finished 2024-03-24T09:17:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:35 smithi117 ceph-mon[35788]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T09:17:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:36.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:35 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2696698545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:17:36.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:35 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1340772772' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c25c107-ff19-45fb-84e3-c994805faa36"}]: dispatch 2024-03-24T09:17:36.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:35 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1340772772' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c25c107-ff19-45fb-84e3-c994805faa36"}]': finished 2024-03-24T09:17:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:35 smithi116 ceph-mon[32207]: osdmap e64: 10 total, 9 up, 10 in 2024-03-24T09:17:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:35 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2696698545' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:17:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:36 smithi143 ceph-mon[35965]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:17:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:36 smithi117 ceph-mon[35788]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:17:37.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:36 smithi116 ceph-mon[32207]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T09:17:39.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:38 smithi116 ceph-mon[32207]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-24T09:17:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:38 smithi143 ceph-mon[35965]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-24T09:17:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:38 smithi117 ceph-mon[35788]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-24T09:17:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:40 smithi143 ceph-mon[35965]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:17:41.290 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:40 smithi116 ceph-mon[32207]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:17:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:40 smithi117 ceph-mon[35788]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:17:43.268 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:42 smithi116 ceph-mon[32207]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T09:17:43.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T09:17:43.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:42 smithi143 ceph-mon[35965]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T09:17:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T09:17:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:42 smithi117 ceph-mon[35788]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T09:17:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T09:17:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:44.271 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:43 smithi143 ceph-mon[35965]: Deploying daemon osd.9 on smithi116 2024-03-24T09:17:44.302 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:43 smithi116 ceph-mon[32207]: Deploying daemon osd.9 on smithi116 2024-03-24T09:17:44.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:43 smithi117 ceph-mon[35788]: Deploying daemon osd.9 on smithi116 2024-03-24T09:17:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:44 smithi143 ceph-mon[35965]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T09:17:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:44 smithi117 ceph-mon[35788]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T09:17:45.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:44 smithi116 ceph-mon[32207]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T09:17:47.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:46 smithi116 ceph-mon[32207]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:46 smithi143 ceph-mon[35965]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:46 smithi117 ceph-mon[35788]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:49.205 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:48 smithi116 ceph-mon[32207]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:49.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:48 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:49.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:48 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:49.206 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:48 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:48 smithi143 ceph-mon[35965]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:48 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:48 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:48 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:48 smithi117 ceph-mon[35788]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:48 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:48 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:48 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:51.221 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:50 smithi116 ceph-mon[32207]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:51.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:50 smithi143 ceph-mon[35965]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:50 smithi117 ceph-mon[35788]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:52.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.734+0000 7f12adffb700 1 -- 172.21.15.116:0/3428806104 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f12b002dca0 con 0x7f12a00a2150 2024-03-24T09:17:52.735 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 9 on host 'smithi116' 2024-03-24T09:17:52.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12a00a2150 msgr2=0x7f12a00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:52.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12a00a2150 0x7f12a00a4610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f129c005fd0 tx=0x7f129c005ee0 comp rx=0 tx=0).stop 2024-03-24T09:17:52.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 msgr2=0x7f12b0102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:52.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0102e50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f12a4003820 tx=0x7f12a4005e80 comp rx=0 tx=0).stop 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 shutdown_connections 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f12b0103e90 0x7f12b01019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12a00a2150 0x7f12a00a4610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12b0100510 0x7f12b0102e50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 --2- 172.21.15.116:0/3428806104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12b01034f0 0x7f12b01014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 >> 172.21.15.116:0/3428806104 conn(0x7f12b00fbe90 msgr2=0x7f12b0106910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 shutdown_connections 2024-03-24T09:17:52.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:17:52.736+0000 7f12b6f6c700 1 -- 172.21.15.116:0/3428806104 wait complete. 2024-03-24T09:17:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:52 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:53.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:52 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:53.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:52 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:17:53.468 DEBUG:teuthology.orchestra.run.smithi116:osd.9> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.9.service 2024-03-24T09:17:53.470 INFO:tasks.cephadm:Deploying osd.10 on smithi117 with /dev/vg_nvme/lv_1... 2024-03-24T09:17:53.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T09:17:53.716 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:17:54.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:53 smithi116 ceph-mon[32207]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T09:17:54.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:53 smithi116 ceph-mon[32207]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T09:17:54.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:53 smithi116 ceph-mon[32207]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:17:54.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:53 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:54.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:53 smithi117 ceph-mon[35788]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T09:17:54.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:53 smithi117 ceph-mon[35788]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T09:17:54.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:53 smithi117 ceph-mon[35788]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:17:54.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:53 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:53 smithi143 ceph-mon[35965]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T09:17:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:53 smithi143 ceph-mon[35965]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T09:17:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:53 smithi143 ceph-mon[35965]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-24T09:17:54.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:53 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:54.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:17:54.263+0000 7fa9fba98700 -1 osd.9 0 waiting for initial osdmap 2024-03-24T09:17:54.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:17:54.275+0000 7fa9f3883700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:17:54.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-mon[32207]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:17:54.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-mon[32207]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T09:17:54.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:54.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:54.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-mon[32207]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:54.994 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:54 smithi116 ceph-mon[32207]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' 2024-03-24T09:17:55.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:54 smithi117 ceph-mon[35788]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:17:55.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:54 smithi117 ceph-mon[35788]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T09:17:55.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:54 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:55.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:54 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:55.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:54 smithi117 ceph-mon[35788]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:55.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:54 smithi117 ceph-mon[35788]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' 2024-03-24T09:17:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:54 smithi143 ceph-mon[35965]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-24T09:17:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:54 smithi143 ceph-mon[35965]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T09:17:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:54 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:54 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:54 smithi143 ceph-mon[35965]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 268 MiB used, 804 GiB / 805 GiB avail 2024-03-24T09:17:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:54 smithi143 ceph-mon[35965]: from='osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084]' entity='osd.9' 2024-03-24T09:17:55.746 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T09:17:55.747 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T09:17:55.747 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-24T09:17:55.747 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-24T09:17:55.747 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113545 s, 923 MB/s 2024-03-24T09:17:55.747 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-24T09:17:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:55 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:17:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:55 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:17:56.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:55 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:56.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:55 smithi117 ceph-mon[35788]: osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] boot 2024-03-24T09:17:56.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:55 smithi117 ceph-mon[35788]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T09:17:56.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:55 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:55 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:17:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:55 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:17:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:55 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:55 smithi116 ceph-mon[32207]: osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] boot 2024-03-24T09:17:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:55 smithi116 ceph-mon[32207]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T09:17:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:55 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:55 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:17:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:55 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:17:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:55 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:55 smithi143 ceph-mon[35965]: osd.9 [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] boot 2024-03-24T09:17:56.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:55 smithi143 ceph-mon[35965]: osdmap e67: 10 total, 10 up, 10 in 2024-03-24T09:17:56.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:55 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:17:56.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_1 2024-03-24T09:17:56.695 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:17:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:17:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T09:17:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:57.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:56 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:57.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:56 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:17:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T09:17:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:17:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:56 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:17:57.900 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:57 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi116 2024-03-24T09:17:57.900 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:57 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-24T09:17:58.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:57 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi116 2024-03-24T09:17:58.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:57 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-24T09:17:58.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:57 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi116 2024-03-24T09:17:58.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:57 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-24T09:17:58.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 -- 172.21.15.117:0/35355161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c102330 msgr2=0x7f307c102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:58.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 --2- 172.21.15.117:0/35355161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c102330 0x7f307c102790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f30700099c0 tx=0x7f3070009cd0 comp rx=0 tx=0).stop 2024-03-24T09:17:58.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 -- 172.21.15.117:0/35355161 shutdown_connections 2024-03-24T09:17:58.462 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 --2- 172.21.15.117:0/35355161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f307c102cd0 0x7f307c109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 --2- 172.21.15.117:0/35355161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f307c101130 0x7f307c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 --2- 172.21.15.117:0/35355161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c102330 0x7f307c102790 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 -- 172.21.15.117:0/35355161 >> 172.21.15.117:0/35355161 conn(0x7f307c0fc6f0 msgr2=0x7f307c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 -- 172.21.15.117:0/35355161 shutdown_connections 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 -- 172.21.15.117:0/35355161 wait complete. 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.457+0000 7f3081923700 1 Processor -- start 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 -- start start 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 0x7f307c198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f307c102330 0x7f307c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f307c102cd0 0x7f307c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f307c19fcc0 con 0x7f307c102330 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f307c19fe00 con 0x7f307c101130 2024-03-24T09:17:58.463 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f3081923700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f307c19ff90 con 0x7f307c102cd0 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 0x7f307c198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 0x7f307c198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38650/0 (socket says 172.21.15.117:38650) 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 -- 172.21.15.117:0/2663282946 learned_addr learned my addr 172.21.15.117:0/2663282946 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307a7fc700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f307c102330 0x7f307c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307b7fe700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f307c102cd0 0x7f307c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 -- 172.21.15.117:0/2663282946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f307c102cd0 msgr2=0x7f307c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f307c102cd0 0x7f307c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 -- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f307c102330 msgr2=0x7f307c198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f307c102330 0x7f307c198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 -- 172.21.15.117:0/2663282946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30700095f0 con 0x7f307c101130 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.458+0000 7f307affd700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 0x7f307c198130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f306c00c970 tx=0x7f306c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.459+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c004220 con 0x7f307c101130 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.459+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f306c007d50 con 0x7f307c101130 2024-03-24T09:17:58.464 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.459+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307c1a0270 con 0x7f307c101130 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.459+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c003d00 con 0x7f307c101130 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.459+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307c1a0770 con 0x7f307c101130 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.459+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f307c068ec0 con 0x7f307c101130 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.461+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f306c0078a0 con 0x7f307c101130 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.461+0000 7f3080921700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f30640a2150 0x7f30640a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.461+0000 7f307a7fc700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f30640a2150 0x7f30640a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.461+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f306c0cd3d0 con 0x7f307c101130 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.462+0000 7f307a7fc700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f30640a2150 0x7f30640a4610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3070009990 tx=0x7f3070005ee0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:17:58.465 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.465+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f306c096ba0 con 0x7f307c101130 2024-03-24T09:17:58.669 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:17:58.669+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f307c0fe8f0 con 0x7f30640a2150 2024-03-24T09:17:59.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:58 smithi117 ceph-mon[35788]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:17:59.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:59.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:59.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:59.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:58 smithi116 ceph-mon[32207]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:17:59.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:59.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:59.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:17:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:58 smithi143 ceph-mon[35965]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:17:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:17:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:17:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:00.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:17:59 smithi117 ceph-mon[35788]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:00.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:17:59 smithi116 ceph-mon[32207]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:00.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:17:59 smithi143 ceph-mon[35965]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:00.873 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:00 smithi117 ceph-mon[35788]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:00 smithi143 ceph-mon[35965]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:01.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:00 smithi116 ceph-mon[32207]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:02.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:01 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/831702799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]: dispatch 2024-03-24T09:18:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:01 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]: dispatch 2024-03-24T09:18:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:01 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]': finished 2024-03-24T09:18:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:01 smithi143 ceph-mon[35965]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T09:18:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:02.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:01 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/831702799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]: dispatch 2024-03-24T09:18:02.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:01 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]: dispatch 2024-03-24T09:18:02.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:01 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]': finished 2024-03-24T09:18:02.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:01 smithi117 ceph-mon[35788]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T09:18:02.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:02.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:01 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/831702799' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]: dispatch 2024-03-24T09:18:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:01 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]: dispatch 2024-03-24T09:18:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:01 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33d2b62d-960d-458f-aa10-364ffb11ee53"}]': finished 2024-03-24T09:18:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:01 smithi116 ceph-mon[32207]: osdmap e69: 11 total, 10 up, 11 in 2024-03-24T09:18:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:02 smithi143 ceph-mon[35965]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:03.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:02 smithi143 ceph-mon[35965]: from='client.? 172.21.15.117:0/1376008049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:18:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:02 smithi117 ceph-mon[35788]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:02 smithi117 ceph-mon[35788]: from='client.? 172.21.15.117:0/1376008049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:18:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:02 smithi116 ceph-mon[32207]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:02 smithi116 ceph-mon[32207]: from='client.? 172.21.15.117:0/1376008049' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:18:05.220 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:04 smithi117 ceph-mon[35788]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:04 smithi143 ceph-mon[35965]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:05.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:04 smithi116 ceph-mon[32207]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:07.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:06 smithi117 ceph-mon[35788]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:06 smithi143 ceph-mon[35965]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:07.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:06 smithi116 ceph-mon[32207]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:09.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:08 smithi116 ceph-mon[32207]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:09.187 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:08 smithi117 ceph-mon[35788]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:08 smithi143 ceph-mon[35965]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:10.098 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:09 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T09:18:10.099 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:09 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:09 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T09:18:10.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:09 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T09:18:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:09 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:11.124 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:10 smithi117 ceph-mon[35788]: Deploying daemon osd.10 on smithi117 2024-03-24T09:18:11.124 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:10 smithi117 ceph-mon[35788]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:10 smithi143 ceph-mon[35965]: Deploying daemon osd.10 on smithi117 2024-03-24T09:18:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:10 smithi143 ceph-mon[35965]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:11.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:10 smithi116 ceph-mon[32207]: Deploying daemon osd.10 on smithi117 2024-03-24T09:18:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:10 smithi116 ceph-mon[32207]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:13.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:12 smithi143 ceph-mon[35965]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:13.315 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:12 smithi117 ceph-mon[35788]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:13.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:12 smithi116 ceph-mon[32207]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:15.165 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:14 smithi117 ceph-mon[35788]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:15.272 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:14 smithi143 ceph-mon[35965]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:15.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:14 smithi116 ceph-mon[32207]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:16 smithi117 ceph-mon[35788]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:16.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:16.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:16.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:16.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:16 smithi116 ceph-mon[32207]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:16 smithi143 ceph-mon[35965]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:18.885 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 10 on host 'smithi117' 2024-03-24T09:18:18.885 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.884+0000 7f3080921700 1 -- 172.21.15.117:0/2663282946 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f307c0fe8f0 con 0x7f30640a2150 2024-03-24T09:18:18.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.887+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f30640a2150 msgr2=0x7f30640a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:18.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.887+0000 7f3081923700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f30640a2150 0x7f30640a4610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3070009990 tx=0x7f3070005ee0 comp rx=0 tx=0).stop 2024-03-24T09:18:18.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.887+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 msgr2=0x7f307c198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:18.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.887+0000 7f3081923700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 0x7f307c198130 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f306c00c970 tx=0x7f306c00cc80 comp rx=0 tx=0).stop 2024-03-24T09:18:18.888 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 shutdown_connections 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f307c102cd0 0x7f307c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f307c101130 0x7f307c198130 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f30640a2150 0x7f30640a4610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 --2- 172.21.15.117:0/2663282946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f307c102330 0x7f307c198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 >> 172.21.15.117:0/2663282946 conn(0x7f307c0fc6f0 msgr2=0x7f307c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 shutdown_connections 2024-03-24T09:18:18.889 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-24T09:18:18.888+0000 7f3081923700 1 -- 172.21.15.117:0/2663282946 wait complete. 2024-03-24T09:18:19.189 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:18 smithi117 ceph-mon[35788]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:19.189 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:19.190 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:18 smithi143 ceph-mon[35965]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:19.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:18 smithi116 ceph-mon[32207]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:19.627 DEBUG:teuthology.orchestra.run.smithi117:osd.10> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.10.service 2024-03-24T09:18:19.630 INFO:tasks.cephadm:Deploying osd.11 on smithi143 with /dev/vg_nvme/lv_1... 2024-03-24T09:18:19.630 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T09:18:19.880 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:18:20.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:19 smithi143 ceph-mon[35965]: from='osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T09:18:20.214 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:19 smithi143 ceph-mon[35965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T09:18:20.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:19 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:20.215 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:19 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:19 smithi117 ceph-mon[35788]: from='osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T09:18:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:19 smithi117 ceph-mon[35788]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T09:18:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:19 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:19 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:19 smithi116 ceph-mon[32207]: from='osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T09:18:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:19 smithi116 ceph-mon[32207]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T09:18:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:19 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:19 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:21.263 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:20 smithi143 ceph-mon[35965]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:21.263 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:20 smithi143 ceph-mon[35965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T09:18:21.263 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:20 smithi143 ceph-mon[35965]: from='osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:18:21.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:20 smithi143 ceph-mon[35965]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T09:18:21.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:20 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:21.264 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:20 smithi143 ceph-mon[35965]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:18:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:20 smithi117 ceph-mon[35788]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:20 smithi117 ceph-mon[35788]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T09:18:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:20 smithi117 ceph-mon[35788]: from='osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:18:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:20 smithi117 ceph-mon[35788]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T09:18:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:20 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:21.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:20 smithi117 ceph-mon[35788]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:18:21.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:20 smithi116 ceph-mon[32207]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:20 smithi116 ceph-mon[32207]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T09:18:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:20 smithi116 ceph-mon[32207]: from='osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:18:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:20 smithi116 ceph-mon[32207]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T09:18:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:20 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:20 smithi116 ceph-mon[32207]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-24T09:18:21.732 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:18:21.402+0000 7f903d716700 -1 osd.10 0 waiting for initial osdmap 2024-03-24T09:18:21.733 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:18:21.415+0000 7f9035501700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:18:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T09:18:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T09:18:21.966 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-03-24T09:18:21.966 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-03-24T09:18:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113491 s, 924 MB/s 2024-03-24T09:18:21.967 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 2024-03-24T09:18:21.967 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-03-24T09:18:22.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:18:22.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T09:18:22.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: from='osd.10 ' entity='osd.10' 2024-03-24T09:18:22.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] boot 2024-03-24T09:18:22.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T09:18:22.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:21 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:18:22.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T09:18:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: from='osd.10 ' entity='osd.10' 2024-03-24T09:18:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] boot 2024-03-24T09:18:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T09:18:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:21 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: from='osd.10 ' entity='osd.10' 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: osd.10 [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] boot 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: osdmap e72: 11 total, 11 up, 11 in 2024-03-24T09:18:22.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:21 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:18:22.776 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi143:vg_nvme/lv_1 2024-03-24T09:18:23.018 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:18:23.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:22 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:18:23.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:22 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:18:23.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:22 smithi143 ceph-mon[35965]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:22 smithi143 ceph-mon[35965]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T09:18:23.249 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:22 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:18:23.249 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:22 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:18:23.250 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:22 smithi117 ceph-mon[35788]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:23.250 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:22 smithi117 ceph-mon[35788]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T09:18:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:22 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:18:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:22 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:18:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:22 smithi116 ceph-mon[32207]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-24T09:18:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:22 smithi116 ceph-mon[32207]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T09:18:24.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi117 2024-03-24T09:18:24.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi117 to 4272M 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T09:18:24.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:24.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:24.427 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.637 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 -- 172.21.15.143:0/2698614648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c102220 msgr2=0x7fc76c102640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:24.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2698614648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c102220 0x7fc76c102640 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc768009990 tx=0x7fc768009ca0 comp rx=0 tx=0).stop 2024-03-24T09:18:24.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 -- 172.21.15.143:0/2698614648 shutdown_connections 2024-03-24T09:18:24.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2698614648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c103dc0 0x7fc76c10a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:24.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2698614648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c102220 0x7fc76c102640 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:24.638 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2698614648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c103420 0x7fc76c103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.633+0000 7fc773ebc700 1 -- 172.21.15.143:0/2698614648 >> 172.21.15.143:0/2698614648 conn(0x7fc76c0fd7e0 msgr2=0x7fc76c0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 -- 172.21.15.143:0/2698614648 shutdown_connections 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 -- 172.21.15.143:0/2698614648 wait complete. 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 Processor -- start 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 -- start start 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c102220 0x7fc76c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c103420 0x7fc76c0745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 0x7fc76c074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:24.639 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c0752d0 con 0x7fc76c102220 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c075440 con 0x7fc76c103420 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.634+0000 7fc773ebc700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1a2240 con 0x7fc76c103dc0 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 0x7fc76c074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 0x7fc76c074b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:43806/0 (socket says 172.21.15.143:43806) 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 -- 172.21.15.143:0/2155647675 learned_addr learned my addr 172.21.15.143:0/2155647675 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc771c58700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c102220 0x7fc76c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc771457700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c103420 0x7fc76c0745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 -- 172.21.15.143:0/2155647675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c103420 msgr2=0x7fc76c0745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:24.640 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c103420 0x7fc76c0745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 -- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c102220 msgr2=0x7fc76c075f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c102220 0x7fc76c075f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 -- 172.21.15.143:0/2155647675 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7680095f0 con 0x7fc76c103dc0 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc772459700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 0x7fc76c074b30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc76400ef30 tx=0x7fc76400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764018070 con 0x7fc76c103dc0 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c1a2470 con 0x7fc76c103dc0 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc76400f040 con 0x7fc76c103dc0 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76400cdd0 con 0x7fc76c103dc0 2024-03-24T09:18:24.641 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.635+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c1a29a0 con 0x7fc76c103dc0 2024-03-24T09:18:24.642 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.636+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c109700 con 0x7fc76c103dc0 2024-03-24T09:18:24.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.637+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fc764003a00 con 0x7fc76c103dc0 2024-03-24T09:18:24.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.638+0000 7fc762ffd700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7580a2150 0x7fc7580a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:24.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.638+0000 7fc771c58700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7580a2150 0x7fc7580a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:24.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.638+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7fc7640cd230 con 0x7fc76c103dc0 2024-03-24T09:18:24.643 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.639+0000 7fc771c58700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7580a2150 0x7fc7580a4610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc76800b2e0 tx=0x7fc76800b720 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:24.644 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.643+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc7640968c0 con 0x7fc76c103dc0 2024-03-24T09:18:24.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi117 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi117 to 4272M 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:24.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi117 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi117 to 4272M 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:24.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:24.847 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:24.846+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc76c02dc00 con 0x7fc7580a2150 2024-03-24T09:18:25.694 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:25 smithi143 ceph-mon[35965]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:25 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:18:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:25 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:18:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:25 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:25.695 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:25 smithi143 ceph-mon[35965]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T09:18:25.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:25 smithi117 ceph-mon[35788]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:25.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:25 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:18:25.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:25 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:18:25.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:25 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:25.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:25 smithi117 ceph-mon[35788]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T09:18:25.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:25 smithi116 ceph-mon[32207]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:25.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:25 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T09:18:25.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:25 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T09:18:25.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:25 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:25.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:25 smithi116 ceph-mon[32207]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T09:18:26.744 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:26 smithi143 ceph-mon[35965]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 355 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:26.745 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:26 smithi143 ceph-mon[35965]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T09:18:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:26 smithi117 ceph-mon[35788]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 355 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:26 smithi117 ceph-mon[35788]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T09:18:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:26 smithi116 ceph-mon[32207]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 355 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:26 smithi116 ceph-mon[32207]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T09:18:28.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:27 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/3726956308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]: dispatch 2024-03-24T09:18:28.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:27 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]: dispatch 2024-03-24T09:18:28.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:27 smithi117 ceph-mon[35788]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]': finished 2024-03-24T09:18:28.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:27 smithi117 ceph-mon[35788]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T09:18:28.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:27 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:28.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:27 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/3726956308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]: dispatch 2024-03-24T09:18:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:27 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]: dispatch 2024-03-24T09:18:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:27 smithi116 ceph-mon[32207]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]': finished 2024-03-24T09:18:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:27 smithi116 ceph-mon[32207]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T09:18:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:27 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:28.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:27 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/3726956308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]: dispatch 2024-03-24T09:18:28.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:27 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]: dispatch 2024-03-24T09:18:28.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:27 smithi143 ceph-mon[35965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0397d0cd-1f13-475b-85e6-332bfd32d77d"}]': finished 2024-03-24T09:18:28.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:27 smithi143 ceph-mon[35965]: osdmap e77: 12 total, 11 up, 12 in 2024-03-24T09:18:28.105 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:27 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:29.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:28 smithi117 ceph-mon[35788]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:29.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:28 smithi117 ceph-mon[35788]: from='client.? 172.21.15.143:0/2984301244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:18:29.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:28 smithi116 ceph-mon[32207]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:29.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:28 smithi116 ceph-mon[32207]: from='client.? 172.21.15.143:0/2984301244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:18:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:28 smithi143 ceph-mon[35965]: pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:28 smithi143 ceph-mon[35965]: from='client.? 172.21.15.143:0/2984301244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T09:18:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:30 smithi143 ceph-mon[35965]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T09:18:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:30 smithi117 ceph-mon[35788]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T09:18:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:30 smithi116 ceph-mon[32207]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-24T09:18:33.126 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:32 smithi143 ceph-mon[35965]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T09:18:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:32 smithi117 ceph-mon[35788]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T09:18:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:32 smithi116 ceph-mon[32207]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T09:18:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:34 smithi143 ceph-mon[35965]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:18:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:34 smithi117 ceph-mon[35788]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:18:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:34 smithi116 ceph-mon[32207]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T09:18:36.263 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T09:18:36.263 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T09:18:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T09:18:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:36 smithi143 ceph-mon[35965]: Deploying daemon osd.11 on smithi143 2024-03-24T09:18:37.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:36 smithi143 ceph-mon[35965]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T09:18:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:36 smithi117 ceph-mon[35788]: Deploying daemon osd.11 on smithi143 2024-03-24T09:18:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:36 smithi117 ceph-mon[35788]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T09:18:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:36 smithi116 ceph-mon[32207]: Deploying daemon osd.11 on smithi143 2024-03-24T09:18:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:36 smithi116 ceph-mon[32207]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T09:18:39.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:38 smithi116 ceph-mon[32207]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T09:18:39.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:38 smithi143 ceph-mon[35965]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T09:18:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:38 smithi117 ceph-mon[35788]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-24T09:18:41.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:40 smithi143 ceph-mon[35965]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T09:18:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:40 smithi117 ceph-mon[35788]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T09:18:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:40 smithi116 ceph-mon[32207]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-24T09:18:42.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:42.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:42.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:42 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:42.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:42 smithi143 ceph-mon[35965]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:42.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:42.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:42.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:42 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:42.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:42 smithi117 ceph-mon[35788]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:42.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:42.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:42.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:42 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:42.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:42 smithi116 ceph-mon[32207]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:45.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:44 smithi143 ceph-mon[35965]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:45.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:45.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:45.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:44 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:44 smithi117 ceph-mon[35788]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:45.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:44 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:44 smithi116 ceph-mon[32207]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:44 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:45.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.563+0000 7fc762ffd700 1 -- 172.21.15.143:0/2155647675 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fc76c02dc00 con 0x7fc7580a2150 2024-03-24T09:18:45.564 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 11 on host 'smithi143' 2024-03-24T09:18:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7580a2150 msgr2=0x7fc7580a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:45.566 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7580a2150 0x7fc7580a4610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc76800b2e0 tx=0x7fc76800b720 comp rx=0 tx=0).stop 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 msgr2=0x7fc76c074b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 0x7fc76c074b30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc76400ef30 tx=0x7fc76400c3a0 comp rx=0 tx=0).stop 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 shutdown_connections 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c103dc0 0x7fc76c074b30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc7580a2150 0x7fc7580a4610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c102220 0x7fc76c075f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 --2- 172.21.15.143:0/2155647675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c103420 0x7fc76c0745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 >> 172.21.15.143:0/2155647675 conn(0x7fc76c0fd7e0 msgr2=0x7fc76c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 shutdown_connections 2024-03-24T09:18:45.567 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:45.566+0000 7fc773ebc700 1 -- 172.21.15.143:0/2155647675 wait complete. 2024-03-24T09:18:46.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:45 smithi143 ceph-mon[35965]: from='osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T09:18:46.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:45 smithi143 ceph-mon[35965]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T09:18:46.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:45 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:46.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:45 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:46.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:45 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:46.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:45 smithi117 ceph-mon[35788]: from='osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T09:18:46.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:45 smithi117 ceph-mon[35788]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T09:18:46.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:45 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:46.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:45 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:46.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:45 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:46.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:45 smithi116 ceph-mon[32207]: from='osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T09:18:46.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:45 smithi116 ceph-mon[32207]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T09:18:46.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:45 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:46.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:45 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:46.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:45 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:46.328 DEBUG:teuthology.orchestra.run.smithi143:osd.11> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.11.service 2024-03-24T09:18:46.331 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-24T09:18:46.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T09:18:46.582 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:18:47.235 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:46 smithi117 ceph-mon[35788]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:47.236 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:46 smithi117 ceph-mon[35788]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T09:18:47.236 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:46 smithi117 ceph-mon[35788]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T09:18:47.236 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:46 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:47.236 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:46 smithi117 ceph-mon[35788]: from='osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:18:47.236 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:46 smithi117 ceph-mon[35788]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:18:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:46 smithi143 ceph-mon[35965]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:46 smithi143 ceph-mon[35965]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T09:18:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:46 smithi143 ceph-mon[35965]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T09:18:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:46 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:46 smithi143 ceph-mon[35965]: from='osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:18:47.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:46 smithi143 ceph-mon[35965]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:18:47.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:46 smithi116 ceph-mon[32207]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:46 smithi116 ceph-mon[32207]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T09:18:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:46 smithi116 ceph-mon[32207]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T09:18:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:46 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:46 smithi116 ceph-mon[32207]: from='osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:18:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:46 smithi116 ceph-mon[32207]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-03-24T09:18:47.596 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:18:47.392+0000 7f191515a700 -1 osd.11 0 waiting for initial osdmap 2024-03-24T09:18:47.596 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:18:47.404+0000 7f190cf45700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T09:18:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:47 smithi116 ceph-mon[32207]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:18:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:47 smithi116 ceph-mon[32207]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T09:18:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:47 smithi116 ceph-mon[32207]: from='osd.11 ' entity='osd.11' 2024-03-24T09:18:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:47 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-mon[35965]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:18:48.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-mon[35965]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T09:18:48.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-mon[35965]: from='osd.11 ' entity='osd.11' 2024-03-24T09:18:48.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:47 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:47 smithi117 ceph-mon[35788]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-03-24T09:18:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:47 smithi117 ceph-mon[35788]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T09:18:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:47 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:47 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:47 smithi117 ceph-mon[35788]: from='osd.11 ' entity='osd.11' 2024-03-24T09:18:48.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:47 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:48.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 -- 172.21.15.116:0/2339216602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 msgr2=0x7f639810a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:48.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 --2- 172.21.15.116:0/2339216602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639810a720 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f639000b0a0 tx=0x7f639000b3b0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 -- 172.21.15.116:0/2339216602 shutdown_connections 2024-03-24T09:18:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 --2- 172.21.15.116:0/2339216602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639810a720 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 --2- 172.21.15.116:0/2339216602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63981004f0 0x7f6398100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 --2- 172.21.15.116:0/2339216602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981034d0 0x7f6398103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 -- 172.21.15.116:0/2339216602 >> 172.21.15.116:0/2339216602 conn(0x7f63980fbe10 msgr2=0x7f63980fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:48.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 -- 172.21.15.116:0/2339216602 shutdown_connections 2024-03-24T09:18:48.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.372+0000 7f639df7f700 1 -- 172.21.15.116:0/2339216602 wait complete. 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.373+0000 7f639df7f700 1 Processor -- start 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.373+0000 7f639df7f700 1 -- start start 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981004f0 0x7f6398198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63981034d0 0x7f6398198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f639819fdc0 con 0x7f6398103e70 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f639819ff00 con 0x7f63981004f0 2024-03-24T09:18:48.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f63981a0090 con 0x7f63981034d0 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f6397fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f6397fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639819f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51014/0 (socket says 172.21.15.116:51014) 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f6397fff700 1 -- 172.21.15.116:0/2085990121 learned_addr learned my addr 172.21.15.116:0/2085990121 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981004f0 0x7f6398198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f6396ffd700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63981034d0 0x7f6398198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 -- 172.21.15.116:0/2085990121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63981034d0 msgr2=0x7f6398198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63981034d0 0x7f6398198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 -- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 msgr2=0x7f639819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639819f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 -- 172.21.15.116:0/2085990121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6390009d00 con 0x7f63981004f0 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f6397fff700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f63977fe700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981004f0 0x7f6398198230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f638800e9c0 tx=0x7f638800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388004220 con 0x7f63981004f0 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.374+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63981a0370 con 0x7f63981004f0 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.375+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6388004380 con 0x7f63981004f0 2024-03-24T09:18:48.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.375+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388003c40 con 0x7f63981004f0 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.375+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63981a08c0 con 0x7f63981004f0 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.375+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6398109690 con 0x7f63981004f0 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.376+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f638800c7c0 con 0x7f63981004f0 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.377+0000 7f6394ff9700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63800a21a0 0x7f63800a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.377+0000 7f6396ffd700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63800a21a0 0x7f63800a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.377+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f6388014070 con 0x7f63981004f0 2024-03-24T09:18:48.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.377+0000 7f6396ffd700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63800a21a0 0x7f63800a4660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f638c009c60 tx=0x7f638c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:48.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.381+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6388096e80 con 0x7f63981004f0 2024-03-24T09:18:48.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.582+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f639810daa0 con 0x7f63981004f0 2024-03-24T09:18:48.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.582+0000 7f6394ff9700 1 -- 172.21.15.116:0/2085990121 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f6388085090 con 0x7f63981004f0 2024-03-24T09:18:48.583 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:18:48.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.584+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63800a21a0 msgr2=0x7f63800a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.584+0000 7f639df7f700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63800a21a0 0x7f63800a4660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f638c009c60 tx=0x7f638c009260 comp rx=0 tx=0).stop 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.584+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981004f0 msgr2=0x7f6398198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.584+0000 7f639df7f700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981004f0 0x7f6398198230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f638800e9c0 tx=0x7f638800ecd0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 shutdown_connections 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63800a21a0 0x7f63800a4660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6398103e70 0x7f639819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63981004f0 0x7f6398198230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 --2- 172.21.15.116:0/2085990121 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63981034d0 0x7f6398198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 >> 172.21.15.116:0/2085990121 conn(0x7f63980fbe10 msgr2=0x7f63981067d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 shutdown_connections 2024-03-24T09:18:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:48.585+0000 7f639df7f700 1 -- 172.21.15.116:0/2085990121 wait complete. 2024-03-24T09:18:49.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: purged_snaps scrub starts 2024-03-24T09:18:49.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: purged_snaps scrub ok 2024-03-24T09:18:49.270 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:49.270 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] boot 2024-03-24T09:18:49.270 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T09:18:49.270 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:49.270 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:48 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2085990121' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:18:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: purged_snaps scrub starts 2024-03-24T09:18:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: purged_snaps scrub ok 2024-03-24T09:18:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] boot 2024-03-24T09:18:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T09:18:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:48 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2085990121' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:18:49.296 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1711271928,"num_in_osds":12,"osd_in_since":1711271907,"num_remapped_pgs":0} 2024-03-24T09:18:49.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T09:18:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: purged_snaps scrub starts 2024-03-24T09:18:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: purged_snaps scrub ok 2024-03-24T09:18:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-24T09:18:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: osd.11 [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] boot 2024-03-24T09:18:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: osdmap e80: 12 total, 12 up, 12 in 2024-03-24T09:18:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:18:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:48 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2085990121' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T09:18:49.555 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:18:50.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: Detected new or changed devices on smithi143 2024-03-24T09:18:50.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.735 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi143 to 4272M 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.736 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:50.737 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:50 smithi116 ceph-mon[32207]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T09:18:50.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: Detected new or changed devices on smithi143 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi143 to 4272M 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:50.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:50.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:50 smithi143 ceph-mon[35965]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T09:18:50.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: Detected new or changed devices on smithi143 2024-03-24T09:18:50.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi143 to 4272M 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:50.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:50 smithi117 ceph-mon[35788]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 -- 172.21.15.116:0/1125894986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c106420 msgr2=0x7f1c7c10dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 --2- 172.21.15.116:0/1125894986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c106420 0x7f1c7c10dff0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400b0a0 tx=0x7f1c7400b3b0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 -- 172.21.15.116:0/1125894986 shutdown_connections 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 --2- 172.21.15.116:0/1125894986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c106420 0x7f1c7c10dff0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 --2- 172.21.15.116:0/1125894986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c101390 0x7f1c7c101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 --2- 172.21.15.116:0/1125894986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c101d40 0x7f1c7c105d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 -- 172.21.15.116:0/1125894986 >> 172.21.15.116:0/1125894986 conn(0x7f1c7c077120 msgr2=0x7f1c7c077530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 -- 172.21.15.116:0/1125894986 shutdown_connections 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.391+0000 7f1c81763700 1 -- 172.21.15.116:0/1125894986 wait complete. 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 Processor -- start 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 -- start start 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c101390 0x7f1c7c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 0x7f1c7c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c106420 0x7f1c7c1a47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c19a510 con 0x7f1c7c101d40 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c1a4cf0 con 0x7f1c7c106420 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c81763700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1c7c1a4e30 con 0x7f1c7c101390 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c7a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 0x7f1c7c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c7a7fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 0x7f1c7c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51040/0 (socket says 172.21.15.116:51040) 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c7a7fc700 1 -- 172.21.15.116:0/307287267 learned_addr learned my addr 172.21.15.116:0/307287267 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c7b7fe700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c106420 0x7f1c7c1a47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.392+0000 7f1c7affd700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c101390 0x7f1c7c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7a7fc700 1 -- 172.21.15.116:0/307287267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c101390 msgr2=0x7f1c7c199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7a7fc700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c101390 0x7f1c7c199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7a7fc700 1 -- 172.21.15.116:0/307287267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c106420 msgr2=0x7f1c7c1a47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7a7fc700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c106420 0x7f1c7c1a47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7a7fc700 1 -- 172.21.15.116:0/307287267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c74009d00 con 0x7f1c7c101d40 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7b7fe700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c106420 0x7f1c7c1a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c7a7fc700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 0x7f1c7c199c70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00b530 tx=0x7f1c6c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c011830 con 0x7f1c7c101d40 2024-03-24T09:18:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c7c1a50e0 con 0x7f1c7c101d40 2024-03-24T09:18:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c6c011e40 con 0x7f1c7c101d40 2024-03-24T09:18:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c00f2e0 con 0x7f1c7c101d40 2024-03-24T09:18:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.393+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c7c1a5630 con 0x7f1c7c101d40 2024-03-24T09:18:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.394+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c7c1096d0 con 0x7f1c7c101d40 2024-03-24T09:18:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.395+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f1c6c011990 con 0x7f1c7c101d40 2024-03-24T09:18:51.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.396+0000 7f1c63fff700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1c640a2150 0x7f1c640a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:51.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.396+0000 7f1c7affd700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1c640a2150 0x7f1c640a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:51.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.396+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1c6c0cd720 con 0x7f1c7c101d40 2024-03-24T09:18:51.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.396+0000 7f1c7affd700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1c640a2150 0x7f1c640a4610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1c70006fd0 tx=0x7f1c70008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:51.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.399+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1c6c096b90 con 0x7f1c7c101d40 2024-03-24T09:18:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.597+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c7c102820 con 0x7f1c7c101d40 2024-03-24T09:18:51.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.598+0000 7f1c63fff700 1 -- 172.21.15.116:0/307287267 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18525 (secure 0 0 0) 0x7f1c6c09a250 con 0x7f1c7c101d40 2024-03-24T09:18:51.599 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:18:51.599 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:18:49.985097+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:18:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1c640a2150 msgr2=0x7f1c640a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1c640a2150 0x7f1c640a4610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1c70006fd0 tx=0x7f1c70008040 comp rx=0 tx=0).stop 2024-03-24T09:18:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 msgr2=0x7f1c7c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 0x7f1c7c199c70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00b530 tx=0x7f1c6c00b8f0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 shutdown_connections 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c7c106420 0x7f1c7c1a47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c7c101390 0x7f1c7c199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1c640a2150 0x7f1c640a4610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 --2- 172.21.15.116:0/307287267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c7c101d40 0x7f1c7c199c70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 >> 172.21.15.116:0/307287267 conn(0x7f1c7c077120 msgr2=0x7f1c7c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 shutdown_connections 2024-03-24T09:18:51.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:51.600+0000 7f1c81763700 1 -- 172.21.15.116:0/307287267 wait complete. 2024-03-24T09:18:51.969 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:51 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/307287267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:18:52.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:51 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/307287267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:18:52.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:51 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/307287267' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:18:52.326 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T09:18:52.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T09:18:52.563 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:18:52.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:52 smithi116 ceph-mon[32207]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:53.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:52 smithi143 ceph-mon[35965]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:53.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:52 smithi117 ceph-mon[35788]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 -- 172.21.15.116:0/4056401641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac102e10 msgr2=0x7fe8ac106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 --2- 172.21.15.116:0/4056401641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac102e10 0x7fe8ac106e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009990 tx=0x7fe8a0009ca0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 -- 172.21.15.116:0/4056401641 shutdown_connections 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 --2- 172.21.15.116:0/4056401641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac107540 0x7fe8ac10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 --2- 172.21.15.116:0/4056401641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8ac102460 0x7fe8ac102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 --2- 172.21.15.116:0/4056401641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac102e10 0x7fe8ac106e60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 -- 172.21.15.116:0/4056401641 >> 172.21.15.116:0/4056401641 conn(0x7fe8ac0fdcd0 msgr2=0x7fe8ac1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 -- 172.21.15.116:0/4056401641 shutdown_connections 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 -- 172.21.15.116:0/4056401641 wait complete. 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 Processor -- start 2024-03-24T09:18:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.201+0000 7fe8b406a700 1 -- start start 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b406a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8ac102460 0x7fe8ac198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b406a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac102e10 0x7fe8ac198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b406a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 0x7fe8ac19d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b406a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac19d610 con 0x7fe8ac107540 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b406a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac19d780 con 0x7fe8ac102e10 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b406a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ac19d940 con 0x7fe8ac102460 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 0x7fe8ac19d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 0x7fe8ac19d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51060/0 (socket says 172.21.15.116:51060) 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 -- 172.21.15.116:0/3411509625 learned_addr learned my addr 172.21.15.116:0/3411509625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b1e06700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8ac102460 0x7fe8ac198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b1605700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac102e10 0x7fe8ac198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 -- 172.21.15.116:0/3411509625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8ac102460 msgr2=0x7fe8ac198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8ac102460 0x7fe8ac198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 -- 172.21.15.116:0/3411509625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac102e10 msgr2=0x7fe8ac198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac102e10 0x7fe8ac198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 -- 172.21.15.116:0/3411509625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a00095f0 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b1605700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac102e10 0x7fe8ac198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8b2607700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 0x7fe8ac19d000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00b550 tx=0x7fe89c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89c011830 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe89c011e40 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.202+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89c00f2e0 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.203+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ac19dc20 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.203+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ac1a5480 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.203+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ac10a860 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.204+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fe89c011990 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.205+0000 7fe8aaffd700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8940a21d0 0x7fe8940a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.205+0000 7fe8b1e06700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8940a21d0 0x7fe8940a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.205+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe89c0cd210 con 0x7fe8ac107540 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.205+0000 7fe8b1e06700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8940a21d0 0x7fe8940a4690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8a400a400 tx=0x7fe8a400d040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:54.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.210+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe89c096680 con 0x7fe8ac107540 2024-03-24T09:18:54.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.404+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8ac112220 con 0x7fe8ac107540 2024-03-24T09:18:54.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.405+0000 7fe8aaffd700 1 -- 172.21.15.116:0/3411509625 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fe89c099d40 con 0x7fe8ac107540 2024-03-24T09:18:54.405 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:18:54.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8940a21d0 msgr2=0x7fe8940a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8940a21d0 0x7fe8940a4690 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8a400a400 tx=0x7fe8a400d040 comp rx=0 tx=0).stop 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 msgr2=0x7fe8ac19d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 0x7fe8ac19d000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe89c00b550 tx=0x7fe89c00b860 comp rx=0 tx=0).stop 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 shutdown_connections 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8940a21d0 0x7fe8940a4690 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8ac107540 0x7fe8ac19d000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8ac102460 0x7fe8ac198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 --2- 172.21.15.116:0/3411509625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8ac102e10 0x7fe8ac198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 >> 172.21.15.116:0/3411509625 conn(0x7fe8ac0fdcd0 msgr2=0x7fe8ac0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 shutdown_connections 2024-03-24T09:18:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:54.407+0000 7fe8b406a700 1 -- 172.21.15.116:0/3411509625 wait complete. 2024-03-24T09:18:55.139 INFO:tasks.cephadm:Adding mds.a on smithi116 2024-03-24T09:18:55.139 INFO:tasks.cephadm:Adding mds.d on smithi116 2024-03-24T09:18:55.139 INFO:tasks.cephadm:Adding mds.g on smithi116 2024-03-24T09:18:55.139 INFO:tasks.cephadm:Adding mds.j on smithi116 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.b on smithi117 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.e on smithi117 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.h on smithi117 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.k on smithi117 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.c on smithi143 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.f on smithi143 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.i on smithi143 2024-03-24T09:18:55.140 INFO:tasks.cephadm:Adding mds.l on smithi143 2024-03-24T09:18:55.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l' 2024-03-24T09:18:55.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:54 smithi143 ceph-mon[35965]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:55.170 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3411509625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:18:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:54 smithi117 ceph-mon[35788]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3411509625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:18:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:54 smithi116 ceph-mon[32207]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3411509625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:18:55.391 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:18:57.117 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:56 smithi143 ceph-mon[35965]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/3891809981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0103450 msgr2=0x7fbbe01038b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/3891809981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0103450 0x7fbbe01038b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc009990 tx=0x7fbbdc009ca0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/3891809981 shutdown_connections 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/3891809981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbe0103df0 0x7fbbe010a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/3891809981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbe0102250 0x7fbbe0102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/3891809981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0103450 0x7fbbe01038b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/3891809981 >> 172.21.15.143:0/3891809981 conn(0x7fbbe00fd810 msgr2=0x7fbbe00ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/3891809981 shutdown_connections 2024-03-24T09:18:57.228 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.225+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/3891809981 wait complete. 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 Processor -- start 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 -- start start 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 0x7fbbe0075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbe0103450 0x7fbbe00745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbe0103df0 0x7fbbe0074b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe019bf90 con 0x7fbbe0103450 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe0075070 con 0x7fbbe0103df0 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe7a2a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe0075230 con 0x7fbbe0102250 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe57c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 0x7fbbe0075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe57c6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 0x7fbbe0075fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:39560/0 (socket says 172.21.15.143:39560) 2024-03-24T09:18:57.229 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe57c6700 1 -- 172.21.15.143:0/255482321 learned_addr learned my addr 172.21.15.143:0/255482321 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.226+0000 7fbbe4fc5700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbe0103450 0x7fbbe00745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe5fc7700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbe0103df0 0x7fbbe0074b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe57c6700 1 -- 172.21.15.143:0/255482321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbe0103df0 msgr2=0x7fbbe0074b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe57c6700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbe0103df0 0x7fbbe0074b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe57c6700 1 -- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbe0103450 msgr2=0x7fbbe00745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe57c6700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbe0103450 0x7fbbe00745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe57c6700 1 -- 172.21.15.143:0/255482321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbdc0095f0 con 0x7fbbe0102250 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe57c6700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 0x7fbbe0075fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000d980 tx=0x7fbbd000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd0004020 con 0x7fbbe0102250 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe01a6960 con 0x7fbbe0102250 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbbd000f690 con 0x7fbbe0102250 2024-03-24T09:18:57.230 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd00039e0 con 0x7fbbe0102250 2024-03-24T09:18:57.231 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.227+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe01a6dd0 con 0x7fbbe0102250 2024-03-24T09:18:57.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.228+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbe01096b0 con 0x7fbbe0102250 2024-03-24T09:18:57.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.230+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fbbd000f800 con 0x7fbbe0102250 2024-03-24T09:18:57.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.230+0000 7fbbd67fc700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbbcc0a2150 0x7fbbcc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:57.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.230+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbbd00cd350 con 0x7fbbe0102250 2024-03-24T09:18:57.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.230+0000 7fbbe4fc5700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbbcc0a2150 0x7fbbcc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:57.232 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.231+0000 7fbbe4fc5700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbbcc0a2150 0x7fbbcc0a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc003780 tx=0x7fbbdc005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:57.235 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.235+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbbd00967c0 con 0x7fbbe0102250 2024-03-24T09:18:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:56 smithi117 ceph-mon[35788]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:57.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:56 smithi116 ceph-mon[32207]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:57.449 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.448+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l", "target": ["mon-mgr", ""]}) -- 0x7fbbe002dd00 con 0x7fbbcc0a2150 2024-03-24T09:18:57.459 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.459+0000 7fbbd67fc700 1 -- 172.21.15.143:0/255482321 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fbbe002dd00 con 0x7fbbcc0a2150 2024-03-24T09:18:57.459 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled mds.all update... 2024-03-24T09:18:57.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbbcc0a2150 msgr2=0x7fbbcc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:57.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbbcc0a2150 0x7fbbcc0a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc003780 tx=0x7fbbdc005fd0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 msgr2=0x7fbbe0075fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:57.461 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 0x7fbbe0075fa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbbd000d980 tx=0x7fbbd000dc90 comp rx=0 tx=0).stop 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 shutdown_connections 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbe0103df0 0x7fbbe0074b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbbe0102250 0x7fbbe0075fa0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbbcc0a2150 0x7fbbcc0a4610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 --2- 172.21.15.143:0/255482321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbe0103450 0x7fbbe00745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 >> 172.21.15.143:0/255482321 conn(0x7fbbe00fd810 msgr2=0x7fbbe01067f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 shutdown_connections 2024-03-24T09:18:57.462 INFO:teuthology.orchestra.run.smithi143.stderr:2024-03-24T09:18:57.461+0000 7fbbe7a2a700 1 -- 172.21.15.143:0/255482321 wait complete. 2024-03-24T09:18:58.125 DEBUG:teuthology.orchestra.run.smithi116:mds.a> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.a.service 2024-03-24T09:18:58.128 DEBUG:teuthology.orchestra.run.smithi116:mds.d> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.d.service 2024-03-24T09:18:58.171 DEBUG:teuthology.orchestra.run.smithi116:mds.g> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.g.service 2024-03-24T09:18:58.214 DEBUG:teuthology.orchestra.run.smithi116:mds.j> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.j.service 2024-03-24T09:18:58.257 DEBUG:teuthology.orchestra.run.smithi117:mds.b> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.b.service 2024-03-24T09:18:58.259 DEBUG:teuthology.orchestra.run.smithi117:mds.e> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.e.service 2024-03-24T09:18:58.302 DEBUG:teuthology.orchestra.run.smithi117:mds.h> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.h.service 2024-03-24T09:18:58.346 DEBUG:teuthology.orchestra.run.smithi117:mds.k> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.k.service 2024-03-24T09:18:58.389 DEBUG:teuthology.orchestra.run.smithi143:mds.c> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.c.service 2024-03-24T09:18:58.391 DEBUG:teuthology.orchestra.run.smithi143:mds.f> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.f.service 2024-03-24T09:18:58.435 DEBUG:teuthology.orchestra.run.smithi143:mds.i> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.i.service 2024-03-24T09:18:58.478 DEBUG:teuthology.orchestra.run.smithi143:mds.l> sudo journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.l.service 2024-03-24T09:18:58.520 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-24T09:18:58.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l;count:12 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:58.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:18:58.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:18:58.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:58 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 -- 172.21.15.116:0/1621034769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 msgr2=0x7f22e4101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 --2- 172.21.15.116:0/1621034769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e4101ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009900 tx=0x7f22d402bc30 comp rx=0 tx=0).stop 2024-03-24T09:18:58.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 -- 172.21.15.116:0/1621034769 shutdown_connections 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 --2- 172.21.15.116:0/1621034769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41021e0 0x7f22e4108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 --2- 172.21.15.116:0/1621034769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e4101ca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 --2- 172.21.15.116:0/1621034769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100630 0x7f22e4100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 -- 172.21.15.116:0/1621034769 >> 172.21.15.116:0/1621034769 conn(0x7f22e40fbba0 msgr2=0x7f22e40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 -- 172.21.15.116:0/1621034769 shutdown_connections 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 -- 172.21.15.116:0/1621034769 wait complete. 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 Processor -- start 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.634+0000 7f22ea807640 1 -- start start 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e4100630 0x7f22e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e41021e0 0x7f22e419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22e419f210 con 0x7f22e4101820 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f22e419f380 con 0x7f22e4100630 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f22e419f540 con 0x7f22e41021e0 2024-03-24T09:18:58.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54454/0 (socket says 172.21.15.116:54454) 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 -- 172.21.15.116:0/1974221555 learned_addr learned my addr 172.21.15.116:0/1974221555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e8da4640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e41021e0 0x7f22e419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e3fff640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e4100630 0x7f22e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 -- 172.21.15.116:0/1974221555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e41021e0 msgr2=0x7f22e419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e41021e0 0x7f22e419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 -- 172.21.15.116:0/1974221555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e4100630 msgr2=0x7f22e40693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e4100630 0x7f22e40693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 -- 172.21.15.116:0/1974221555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22d4009560 con 0x7f22e4101820 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e3fff640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e4100630 0x7f22e40693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e8da4640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e41021e0 0x7f22e419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e37fe640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e419c410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22d4006010 tx=0x7f22d4004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:58.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4004a50 con 0x7f22e4101820 2024-03-24T09:18:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22d402de90 con 0x7f22e4101820 2024-03-24T09:18:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e419f760 con 0x7f22e4101820 2024-03-24T09:18:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4005410 con 0x7f22e4101820 2024-03-24T09:18:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.635+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e419fc70 con 0x7f22e4101820 2024-03-24T09:18:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.636+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e4107a70 con 0x7f22e4101820 2024-03-24T09:18:58.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.637+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f22d4005570 con 0x7f22e4101820 2024-03-24T09:18:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.638+0000 7f22e17fa640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f22cc0a20c0 0x7f22cc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.638+0000 7f22e3fff640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f22cc0a20c0 0x7f22cc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.638+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f22d40effa0 con 0x7f22e4101820 2024-03-24T09:18:58.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.638+0000 7f22e3fff640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f22cc0a20c0 0x7f22cc0a4580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f22e4101680 tx=0x7f22d8005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:58.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.641+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f22d40b9410 con 0x7f22e4101820 2024-03-24T09:18:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l;count:12 2024-03-24T09:18:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:18:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:18:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:58 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:58.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:18:58.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l;count:12 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:18:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:58 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:18:58.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.818+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22e410c140 con 0x7f22e4101820 2024-03-24T09:18:58.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.819+0000 7f22e17fa640 1 -- 172.21.15.116:0/1974221555 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18525 (secure 0 0 0) 0x7f22d40bcad0 con 0x7f22e4101820 2024-03-24T09:18:58.820 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:18:58.820 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:18:49.985097+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:18:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f22cc0a20c0 msgr2=0x7f22cc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f22cc0a20c0 0x7f22cc0a4580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f22e4101680 tx=0x7f22d8005f90 comp rx=0 tx=0).stop 2024-03-24T09:18:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 msgr2=0x7f22e419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e419c410 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f22d4006010 tx=0x7f22d4004370 comp rx=0 tx=0).stop 2024-03-24T09:18:58.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 shutdown_connections 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f22cc0a20c0 0x7f22cc0a4580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e41021e0 0x7f22e419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4101820 0x7f22e419c410 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.821+0000 7f22ea807640 1 --2- 172.21.15.116:0/1974221555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e4100630 0x7f22e40693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.822+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 >> 172.21.15.116:0/1974221555 conn(0x7f22e40fbba0 msgr2=0x7f22e4104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.822+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 shutdown_connections 2024-03-24T09:18:58.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.822+0000 7f22ea807640 1 -- 172.21.15.116:0/1974221555 wait complete. 2024-03-24T09:18:58.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T09:18:58.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 -- 172.21.15.116:0/2777246553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f81021c0 msgr2=0x7f02f8108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 --2- 172.21.15.116:0/2777246553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f81021c0 0x7f02f8108a50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f02f0009ca0 tx=0x7f02f002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 -- 172.21.15.116:0/2777246553 shutdown_connections 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 --2- 172.21.15.116:0/2777246553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f81021c0 0x7f02f8108a50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 --2- 172.21.15.116:0/2777246553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02f8101800 0x7f02f8101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 --2- 172.21.15.116:0/2777246553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f8100610 0x7f02f8100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 -- 172.21.15.116:0/2777246553 >> 172.21.15.116:0/2777246553 conn(0x7f02f80fbb80 msgr2=0x7f02f80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 -- 172.21.15.116:0/2777246553 shutdown_connections 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.937+0000 7f02fee2e640 1 -- 172.21.15.116:0/2777246553 wait complete. 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 Processor -- start 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 -- start start 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 0x7f02f8196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02f8101800 0x7f02f8196ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f81021c0 0x7f02f819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02f819e060 con 0x7f02f8100610 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f02f819e1d0 con 0x7f02f8101800 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fee2e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f02f819e390 con 0x7f02f81021c0 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 0x7f02f8196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 0x7f02f8196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54466/0 (socket says 172.21.15.116:54466) 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 -- 172.21.15.116:0/1967386894 learned_addr learned my addr 172.21.15.116:0/1967386894 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02f7fff640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02f8101800 0x7f02f8196ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fd3cb640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f81021c0 0x7f02f819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 -- 172.21.15.116:0/1967386894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f81021c0 msgr2=0x7f02f819db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f81021c0 0x7f02f819db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 -- 172.21.15.116:0/1967386894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02f8101800 msgr2=0x7f02f8196ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:58.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02f8101800 0x7f02f8196ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fcbca640 1 -- 172.21.15.116:0/1967386894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f0009900 con 0x7f02f8100610 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.938+0000 7f02fd3cb640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f81021c0 0x7f02f819db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.939+0000 7f02fcbca640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 0x7f02f8196570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02e800d950 tx=0x7f02e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.939+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8009c30 con 0x7f02f8100610 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.939+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f02e8009d90 con 0x7f02f8100610 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.939+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f819e670 con 0x7f02f8100610 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.939+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e800f6b0 con 0x7f02f8100610 2024-03-24T09:18:58.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.939+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f819eb80 con 0x7f02f8100610 2024-03-24T09:18:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.940+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f8107980 con 0x7f02f8100610 2024-03-24T09:18:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.941+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f02e8010a30 con 0x7f02f8100610 2024-03-24T09:18:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.941+0000 7f02f5ffb640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f02e00a2140 0x7f02e00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.942+0000 7f02f7fff640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f02e00a2140 0x7f02e00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.942+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f02e80cd520 con 0x7f02f8100610 2024-03-24T09:18:58.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.942+0000 7f02f7fff640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f02e00a2140 0x7f02e00a4600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f02f8197a80 tx=0x7f02ec005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:58.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:58.946+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f02e8096990 con 0x7f02f8100610 2024-03-24T09:18:59.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.120+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02f802d1d0 con 0x7f02f8100610 2024-03-24T09:18:59.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.120+0000 7f02f5ffb640 1 -- 172.21.15.116:0/1967386894 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f02e80b8e60 con 0x7f02f8100610 2024-03-24T09:18:59.121 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f02e00a2140 msgr2=0x7f02e00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f02e00a2140 0x7f02e00a4600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f02f8197a80 tx=0x7f02ec005f90 comp rx=0 tx=0).stop 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 msgr2=0x7f02f8196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 0x7f02f8196570 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f02e800d950 tx=0x7f02e800de10 comp rx=0 tx=0).stop 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 shutdown_connections 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f02e00a2140 0x7f02e00a4600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02f81021c0 0x7f02f819db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02f8101800 0x7f02f8196ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 --2- 172.21.15.116:0/1967386894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02f8100610 0x7f02f8196570 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 >> 172.21.15.116:0/1967386894 conn(0x7f02f80fbb80 msgr2=0x7f02f8104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 shutdown_connections 2024-03-24T09:18:59.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.122+0000 7f02fee2e640 1 -- 172.21.15.116:0/1967386894 wait complete. 2024-03-24T09:18:59.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 -- 172.21.15.116:0/2989725953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534101820 msgr2=0x7fc534101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 --2- 172.21.15.116:0/2989725953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534101820 0x7fc534101ca0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009900 tx=0x7fc51c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 -- 172.21.15.116:0/2989725953 shutdown_connections 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 --2- 172.21.15.116:0/2989725953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5341021e0 0x7fc534108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 --2- 172.21.15.116:0/2989725953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534101820 0x7fc534101ca0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 --2- 172.21.15.116:0/2989725953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc534100630 0x7fc534100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 -- 172.21.15.116:0/2989725953 >> 172.21.15.116:0/2989725953 conn(0x7fc5340fbba0 msgr2=0x7fc5340fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 -- 172.21.15.116:0/2989725953 shutdown_connections 2024-03-24T09:18:59.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.233+0000 7fc538925640 1 -- 172.21.15.116:0/2989725953 wait complete. 2024-03-24T09:18:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 Processor -- start 2024-03-24T09:18:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 -- start start 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 0x7fc5341976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc534101820 0x7fc534197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5341021e0 0x7fc53419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc53419f1d0 con 0x7fc534100630 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc53419f340 con 0x7fc5341021e0 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.234+0000 7fc538925640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc53419f500 con 0x7fc534101820 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 0x7fc5341976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 0x7fc5341976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54484/0 (socket says 172.21.15.116:54484) 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 -- 172.21.15.116:0/301793509 learned_addr learned my addr 172.21.15.116:0/301793509 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc532d9d640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5341021e0 0x7fc53419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc529d9b640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc534101820 0x7fc534197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 -- 172.21.15.116:0/301793509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc534101820 msgr2=0x7fc534197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc534101820 0x7fc534197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 -- 172.21.15.116:0/301793509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5341021e0 msgr2=0x7fc53419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5341021e0 0x7fc53419ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 -- 172.21.15.116:0/301793509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51c009560 con 0x7fc534100630 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc53259c640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 0x7fc5341976e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc52400d770 tx=0x7fc52400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524009c30 con 0x7fc534100630 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53419f7e0 con 0x7fc534100630 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5240044d0 con 0x7fc534100630 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc524003de0 con 0x7fc534100630 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.235+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc53419fcc0 con 0x7fc534100630 2024-03-24T09:18:59.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.236+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53410bef0 con 0x7fc534100630 2024-03-24T09:18:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.237+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fc524003f40 con 0x7fc534100630 2024-03-24T09:18:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.238+0000 7fc52bfff640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc5200a20c0 0x7fc5200a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:18:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.238+0000 7fc529d9b640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc5200a20c0 0x7fc5200a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:18:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.238+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc5240cd0c0 con 0x7fc534100630 2024-03-24T09:18:59.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.238+0000 7fc529d9b640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc5200a20c0 0x7fc5200a4580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009230 tx=0x7fc51c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:18:59.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.242+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5240965e0 con 0x7fc534100630 2024-03-24T09:18:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.450+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc53410c0a0 con 0x7fc534100630 2024-03-24T09:18:59.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.451+0000 7fc52bfff640 1 -- 172.21.15.116:0/301793509 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fc524099ca0 con 0x7fc534100630 2024-03-24T09:18:59.451 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:18:59.452 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T09:18:59.452 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 1 2024-03-24T09:18:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc5200a20c0 msgr2=0x7fc5200a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc5200a20c0 0x7fc5200a4580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009230 tx=0x7fc51c005fd0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 msgr2=0x7fc5341976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 0x7fc5341976e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc52400d770 tx=0x7fc52400dc30 comp rx=0 tx=0).stop 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 shutdown_connections 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5341021e0 0x7fc53419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc5200a20c0 0x7fc5200a4580 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc534101820 0x7fc534197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 --2- 172.21.15.116:0/301793509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc534100630 0x7fc5341976e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 >> 172.21.15.116:0/301793509 conn(0x7fc5340fbba0 msgr2=0x7fc534068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 shutdown_connections 2024-03-24T09:18:59.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:18:59.453+0000 7fc538925640 1 -- 172.21.15.116:0/301793509 wait complete. 2024-03-24T09:18:59.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:59 smithi143 ceph-mon[35965]: Deploying daemon mds.l on smithi143 2024-03-24T09:18:59.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:59 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1974221555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:18:59.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:59 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1967386894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:18:59.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:18:59 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/301793509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:18:59.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:59 smithi117 ceph-mon[35788]: Deploying daemon mds.l on smithi143 2024-03-24T09:18:59.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:59 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1974221555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:18:59.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:59 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1967386894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:18:59.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:18:59 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/301793509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:18:59.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:59 smithi116 ceph-mon[32207]: Deploying daemon mds.l on smithi143 2024-03-24T09:18:59.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1974221555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:18:59.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1967386894' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:18:59.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:18:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/301793509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:00 smithi143 ceph-mon[35965]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:00.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:00 smithi117 ceph-mon[35788]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:00.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:00 smithi116 ceph-mon[32207]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:01.275 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:19:01 smithi143 systemd[1]: Started Ceph mds.l for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:01.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:01.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 -- 172.21.15.116:0/3639915623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc1021f0 msgr2=0x7faecc108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:01.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 --2- 172.21.15.116:0/3639915623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc1021f0 0x7faecc108a80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7faec4009ca0 tx=0x7faec402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 -- 172.21.15.116:0/3639915623 shutdown_connections 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 --2- 172.21.15.116:0/3639915623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc1021f0 0x7faecc108a80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 --2- 172.21.15.116:0/3639915623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faecc101830 0x7faecc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 --2- 172.21.15.116:0/3639915623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc100640 0x7faecc100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 -- 172.21.15.116:0/3639915623 >> 172.21.15.116:0/3639915623 conn(0x7faecc0fbbf0 msgr2=0x7faecc0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 -- 172.21.15.116:0/3639915623 shutdown_connections 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.579+0000 7faed43c2640 1 -- 172.21.15.116:0/3639915623 wait complete. 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 Processor -- start 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 -- start start 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc100640 0x7faecc197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:01.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 0x7faecc197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faecc1021f0 0x7faecc19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faecc19f200 con 0x7faecc101830 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faecc19f370 con 0x7faecc1021f0 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed43c2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7faecc19f530 con 0x7faecc100640 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 0x7faecc197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 0x7faecc197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54490/0 (socket says 172.21.15.116:54490) 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 -- 172.21.15.116:0/2469035348 learned_addr learned my addr 172.21.15.116:0/2469035348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed215e640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc100640 0x7faecc197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed295f640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faecc1021f0 0x7faecc19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 -- 172.21.15.116:0/2469035348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc100640 msgr2=0x7faecc197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc100640 0x7faecc197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 -- 172.21.15.116:0/2469035348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faecc1021f0 msgr2=0x7faecc19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:01.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faecc1021f0 0x7faecc19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.580+0000 7faed195d640 1 -- 172.21.15.116:0/2469035348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faec4009900 con 0x7faecc101830 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.581+0000 7faed195d640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 0x7faecc197c50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faebc00e9f0 tx=0x7faebc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.581+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc009700 con 0x7faecc101830 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.581+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faebc00cd80 con 0x7faecc101830 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.581+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faecc19f810 con 0x7faecc101830 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.581+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faebc018840 con 0x7faecc101830 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.581+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faecc19fd20 con 0x7faecc101830 2024-03-24T09:19:01.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.582+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faecc10bf00 con 0x7faecc101830 2024-03-24T09:19:01.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.583+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7faebc0189a0 con 0x7faecc101830 2024-03-24T09:19:01.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.583+0000 7faec37fe640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faeb80a20c0 0x7faeb80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:01.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.583+0000 7faed215e640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faeb80a20c0 0x7faeb80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:01.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.583+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faebc014070 con 0x7faecc101830 2024-03-24T09:19:01.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.584+0000 7faed215e640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faeb80a20c0 0x7faeb80a4580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faec8005fd0 tx=0x7faec800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:01.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.588+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faebc053280 con 0x7faecc101830 2024-03-24T09:19:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.797+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faecc19ff40 con 0x7faecc101830 2024-03-24T09:19:01.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.797+0000 7faec37fe640 1 -- 172.21.15.116:0/2469035348 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7faebc097050 con 0x7faecc101830 2024-03-24T09:19:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:01.799 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T09:19:01.800 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faeb80a20c0 msgr2=0x7faeb80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faeb80a20c0 0x7faeb80a4580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faec8005fd0 tx=0x7faec800b390 comp rx=0 tx=0).stop 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 msgr2=0x7faecc197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 0x7faecc197c50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faebc00e9f0 tx=0x7faebc00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 shutdown_connections 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faecc1021f0 0x7faecc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faecc101830 0x7faecc197c50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faeb80a20c0 0x7faeb80a4580 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 --2- 172.21.15.116:0/2469035348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faecc100640 0x7faecc197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 >> 172.21.15.116:0/2469035348 conn(0x7faecc0fbbf0 msgr2=0x7faecc0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 shutdown_connections 2024-03-24T09:19:01.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:01.799+0000 7faed43c2640 1 -- 172.21.15.116:0/2469035348 wait complete. 2024-03-24T09:19:02.014 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6832/2011428167,v1:172.21.15.143:6833/2011428167] up:boot 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: fsmap 1 up:standby 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:02.015 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:01 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2469035348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6832/2011428167,v1:172.21.15.143:6833/2011428167] up:boot 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: fsmap 1 up:standby 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:02.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:02.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:01 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2469035348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:02.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6832/2011428167,v1:172.21.15.143:6833/2011428167] up:boot 2024-03-24T09:19:02.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: fsmap 1 up:standby 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:02.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:01 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2469035348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:03.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:02 smithi117 ceph-mon[35788]: Deploying daemon mds.k on smithi117 2024-03-24T09:19:03.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:02 smithi117 ceph-mon[35788]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:02 smithi143 ceph-mon[35965]: Deploying daemon mds.k on smithi117 2024-03-24T09:19:03.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:02 smithi143 ceph-mon[35965]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:02 smithi116 ceph-mon[32207]: Deploying daemon mds.k on smithi117 2024-03-24T09:19:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:02 smithi116 ceph-mon[32207]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:03.566 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:03 smithi117 systemd[1]: Starting Ceph mds.k for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:03.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:03.869 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:03 smithi117 podman[64689]: 2024-03-24 09:19:03.482129571 +0000 UTC m=+0.018058445 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:03.869 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:03 smithi117 podman[64689]: 2024-03-24 09:19:03.667088035 +0000 UTC m=+0.203016907 container create 6b2a3e1261f6ec019d36398b926c9d0e0f4afd6a656e4c61b84db8fdad0ee019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True) 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 -- 172.21.15.116:0/3032996776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90381021e0 msgr2=0x7f9038108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 --2- 172.21.15.116:0/3032996776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90381021e0 0x7f9038108a70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9030009ca0 tx=0x7f903002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 -- 172.21.15.116:0/3032996776 shutdown_connections 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 --2- 172.21.15.116:0/3032996776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90381021e0 0x7f9038108a70 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 --2- 172.21.15.116:0/3032996776 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9038101820 0x7f9038101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 --2- 172.21.15.116:0/3032996776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9038100630 0x7f9038100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 -- 172.21.15.116:0/3032996776 >> 172.21.15.116:0/3032996776 conn(0x7f90380fbbc0 msgr2=0x7f90380fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 -- 172.21.15.116:0/3032996776 shutdown_connections 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.926+0000 7f903d75e640 1 -- 172.21.15.116:0/3032996776 wait complete. 2024-03-24T09:19:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 Processor -- start 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 -- start start 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 0x7f9038196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9038101820 0x7f9038196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90381021e0 0x7f903819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f903819e0a0 con 0x7f9038100630 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f903819e210 con 0x7f9038101820 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f903d75e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f903819e3d0 con 0x7f90381021e0 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 0x7f9038196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 0x7f9038196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54508/0 (socket says 172.21.15.116:54508) 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 -- 172.21.15.116:0/1222124823 learned_addr learned my addr 172.21.15.116:0/1222124823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f902ffff640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9038101820 0x7f9038196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f90377fe640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90381021e0 0x7f903819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 -- 172.21.15.116:0/1222124823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90381021e0 msgr2=0x7f903819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90381021e0 0x7f903819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 -- 172.21.15.116:0/1222124823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9038101820 msgr2=0x7f9038196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9038101820 0x7f9038196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.927+0000 7f9036ffd640 1 -- 172.21.15.116:0/1222124823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9030009900 con 0x7f9038100630 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.928+0000 7f9036ffd640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 0x7f9038196600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f902800d950 tx=0x7f902800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.928+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9028009c30 con 0x7f9038100630 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.928+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9028009d90 con 0x7f9038100630 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.928+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903819e6b0 con 0x7f9038100630 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.928+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902800f6b0 con 0x7f9038100630 2024-03-24T09:19:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.928+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f903819ebc0 con 0x7f9038100630 2024-03-24T09:19:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.929+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9038107950 con 0x7f9038100630 2024-03-24T09:19:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.930+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f9028010a30 con 0x7f9038100630 2024-03-24T09:19:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.930+0000 7f9034ff9640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f90240a2140 0x7f90240a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.930+0000 7f902ffff640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f90240a2140 0x7f90240a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.930+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f90280cd520 con 0x7f9038100630 2024-03-24T09:19:03.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.931+0000 7f902ffff640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f90240a2140 0x7f90240a4600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9038197ac0 tx=0x7f9020005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:03.935+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9028096990 con 0x7f9038100630 2024-03-24T09:19:04.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.153+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f903810bef0 con 0x7f9038100630 2024-03-24T09:19:04.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.153+0000 7f9034ff9640 1 -- 172.21.15.116:0/1222124823 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f90280b8e60 con 0x7f9038100630 2024-03-24T09:19:04.154 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:04.154 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T09:19:04.154 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-03-24T09:19:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f90240a2140 msgr2=0x7f90240a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f90240a2140 0x7f90240a4600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9038197ac0 tx=0x7f9020005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 msgr2=0x7f9038196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 0x7f9038196600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f902800d950 tx=0x7f902800de10 comp rx=0 tx=0).stop 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 shutdown_connections 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f90240a2140 0x7f90240a4600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f90381021e0 0x7f903819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9038101820 0x7f9038196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 --2- 172.21.15.116:0/1222124823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9038100630 0x7f9038196600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 >> 172.21.15.116:0/1222124823 conn(0x7f90380fbbc0 msgr2=0x7f9038104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 shutdown_connections 2024-03-24T09:19:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:04.155+0000 7f903d75e640 1 -- 172.21.15.116:0/1222124823 wait complete. 2024-03-24T09:19:04.316 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:03 smithi117 podman[64689]: 2024-03-24 09:19:03.991057552 +0000 UTC m=+0.526986427 container init 6b2a3e1261f6ec019d36398b926c9d0e0f4afd6a656e4c61b84db8fdad0ee019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T09:19:04.316 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:03 smithi117 podman[64689]: 2024-03-24 09:19:03.9936794 +0000 UTC m=+0.529608271 container start 6b2a3e1261f6ec019d36398b926c9d0e0f4afd6a656e4c61b84db8fdad0ee019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T09:19:04.316 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mds[64708]: -- 172.21.15.117:0/2482737984 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56235dda41a0 con 0x56235dd7a800 2024-03-24T09:19:04.317 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k[64704]: starting mds.k at 2024-03-24T09:19:04.317 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:04 smithi117 bash[64689]: 6b2a3e1261f6ec019d36398b926c9d0e0f4afd6a656e4c61b84db8fdad0ee019 2024-03-24T09:19:04.317 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:19:04 smithi117 systemd[1]: Started Ceph mds.k for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:05.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1222124823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:05.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:04 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1222124823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:04 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1222124823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:04 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:06.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:05 smithi116 ceph-mon[32207]: Deploying daemon mds.j on smithi116 2024-03-24T09:19:06.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:05 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] up:boot 2024-03-24T09:19:06.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:05 smithi116 ceph-mon[32207]: fsmap 2 up:standby 2024-03-24T09:19:06.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:05 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:19:06.167 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:05 smithi143 ceph-mon[35965]: Deploying daemon mds.j on smithi116 2024-03-24T09:19:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:05 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] up:boot 2024-03-24T09:19:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:05 smithi143 ceph-mon[35965]: fsmap 2 up:standby 2024-03-24T09:19:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:05 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:19:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:05 smithi117 ceph-mon[35788]: Deploying daemon mds.j on smithi116 2024-03-24T09:19:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:05 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] up:boot 2024-03-24T09:19:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:05 smithi117 ceph-mon[35788]: fsmap 2 up:standby 2024-03-24T09:19:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:05 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:19:06.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 -- 172.21.15.116:0/2791893058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 msgr2=0x7f4d54101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 --2- 172.21.15.116:0/2791893058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d54101b60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009900 tx=0x7f4d4402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 -- 172.21.15.116:0/2791893058 shutdown_connections 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 --2- 172.21.15.116:0/2791893058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d541032f0 0x7f4d54109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 --2- 172.21.15.116:0/2791893058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54102930 0x7f4d54102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 --2- 172.21.15.116:0/2791893058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d54101b60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 -- 172.21.15.116:0/2791893058 >> 172.21.15.116:0/2791893058 conn(0x7f4d540fccb0 msgr2=0x7f4d540ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 -- 172.21.15.116:0/2791893058 shutdown_connections 2024-03-24T09:19:06.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.402+0000 7f4d5b447640 1 -- 172.21.15.116:0/2791893058 wait complete. 2024-03-24T09:19:06.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 Processor -- start 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 -- start start 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d541976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d54102930 0x7f4d54197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d541032f0 0x7f4d5419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5419f1b0 con 0x7f4d54101740 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5419f320 con 0x7f4d541032f0 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d5b447640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5419f4e0 con 0x7f4d54102930 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d541976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d541976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54520/0 (socket says 172.21.15.116:54520) 2024-03-24T09:19:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 -- 172.21.15.116:0/1646848154 learned_addr learned my addr 172.21.15.116:0/1646848154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d599e4640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d541032f0 0x7f4d5419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d589e2640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d54102930 0x7f4d54197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 -- 172.21.15.116:0/1646848154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d54102930 msgr2=0x7f4d54197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d54102930 0x7f4d54197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 -- 172.21.15.116:0/1646848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d541032f0 msgr2=0x7f4d5419ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d541032f0 0x7f4d5419ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.403+0000 7f4d591e3640 1 -- 172.21.15.116:0/1646848154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d44009560 con 0x7f4d54101740 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.404+0000 7f4d591e3640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d541976c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4d44000c00 tx=0x7f4d44004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.404+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d440035f0 con 0x7f4d54101740 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.404+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d44003c00 con 0x7f4d54101740 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.404+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5419f760 con 0x7f4d54101740 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.404+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d440055f0 con 0x7f4d54101740 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.404+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5419fcc0 con 0x7f4d54101740 2024-03-24T09:19:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.405+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d54108a60 con 0x7f4d54101740 2024-03-24T09:19:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.406+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f4d44004dc0 con 0x7f4d54101740 2024-03-24T09:19:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.406+0000 7f4d4a7fc640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d400a2140 0x7f4d400a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.407+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4d4402de40 con 0x7f4d54101740 2024-03-24T09:19:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.407+0000 7f4d589e2640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d400a2140 0x7f4d400a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.407+0000 7f4d589e2640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d400a2140 0x7f4d400a4600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4d54198bd0 tx=0x7f4d5000a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.411+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4d440b9670 con 0x7f4d54101740 2024-03-24T09:19:06.549 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 systemd[1]: Starting Ceph mds.j for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.632+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d5402d130 con 0x7f4d54101740 2024-03-24T09:19:06.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.633+0000 7f4d4a7fc640 1 -- 172.21.15.116:0/1646848154 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f4d440bcd30 con 0x7f4d54101740 2024-03-24T09:19:06.634 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:06.634 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T09:19:06.634 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d400a2140 msgr2=0x7f4d400a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d400a2140 0x7f4d400a4600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4d54198bd0 tx=0x7f4d5000a2d0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 msgr2=0x7f4d541976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d541976c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4d44000c00 tx=0x7f4d44004be0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 shutdown_connections 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d541032f0 0x7f4d5419ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d400a2140 0x7f4d400a4600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d54102930 0x7f4d54197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 --2- 172.21.15.116:0/1646848154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d54101740 0x7f4d541976c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 >> 172.21.15.116:0/1646848154 conn(0x7f4d540fccb0 msgr2=0x7f4d54105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 shutdown_connections 2024-03-24T09:19:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:06.635+0000 7f4d5b447640 1 -- 172.21.15.116:0/1646848154 wait complete. 2024-03-24T09:19:06.818 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 podman[72673]: 2024-03-24 09:19:06.452121575 +0000 UTC m=+0.017781223 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:06.818 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 podman[72673]: 2024-03-24 09:19:06.585401354 +0000 UTC m=+0.151060988 container create e47f948a299cf2a39f0358b43a58ca438d7ad689773d50b4bb3508d498bd8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:19:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:07 smithi143 ceph-mon[35965]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:07 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1646848154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:07 smithi117 ceph-mon[35788]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:07 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1646848154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:07 smithi116 ceph-mon[32207]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:07 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1646848154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:07.319 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 podman[72673]: 2024-03-24 09:19:06.82689394 +0000 UTC m=+0.392553592 container init e47f948a299cf2a39f0358b43a58ca438d7ad689773d50b4bb3508d498bd8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T09:19:07.319 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 podman[72673]: 2024-03-24 09:19:06.82957184 +0000 UTC m=+0.395231481 container start e47f948a299cf2a39f0358b43a58ca438d7ad689773d50b4bb3508d498bd8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T09:19:07.319 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 ceph-mds[72693]: -- 172.21.15.116:0/2360468480 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd511001a0 con 0x55bd510d6800 2024-03-24T09:19:07.319 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j[72689]: starting mds.j at 2024-03-24T09:19:07.319 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 bash[72673]: e47f948a299cf2a39f0358b43a58ca438d7ad689773d50b4bb3508d498bd8f7d 2024-03-24T09:19:07.319 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:19:06 smithi116 systemd[1]: Started Ceph mds.j for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:08.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.116:6834/203590866,v1:172.21.15.116:6835/203590866] up:boot 2024-03-24T09:19:08.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: fsmap 3 up:standby 2024-03-24T09:19:08.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T09:19:08.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:08.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:08.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:08 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.116:6834/203590866,v1:172.21.15.116:6835/203590866] up:boot 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: fsmap 3 up:standby 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:08.279 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:08.280 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:08 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.116:6834/203590866,v1:172.21.15.116:6835/203590866] up:boot 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: fsmap 3 up:standby 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:08 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:08.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- 172.21.15.116:0/4189925893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 msgr2=0x7fca30108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- 172.21.15.116:0/4189925893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca30108a70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fca28009ca0 tx=0x7fca2802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- 172.21.15.116:0/4189925893 shutdown_connections 2024-03-24T09:19:08.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- 172.21.15.116:0/4189925893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca30108a70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- 172.21.15.116:0/4189925893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30101820 0x7fca30101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- 172.21.15.116:0/4189925893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 0x7fca30100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- 172.21.15.116:0/4189925893 >> 172.21.15.116:0/4189925893 conn(0x7fca300fbba0 msgr2=0x7fca300fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- 172.21.15.116:0/4189925893 shutdown_connections 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- 172.21.15.116:0/4189925893 wait complete. 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 Processor -- start 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- start start 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 0x7fca30196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30101820 0x7fca30196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca3019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca3019e100 con 0x7fca301021e0 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca3019e270 con 0x7fca30100630 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca36e9d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fca3019e430 con 0x7fca30101820 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.753+0000 7fca3543a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca3019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:08.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca3019dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54074/0 (socket says 172.21.15.116:54074) 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 -- 172.21.15.116:0/566704107 learned_addr learned my addr 172.21.15.116:0/566704107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca27fff640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30101820 0x7fca30196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca34c39640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 0x7fca30196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 -- 172.21.15.116:0/566704107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30101820 msgr2=0x7fca30196b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30101820 0x7fca30196b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 -- 172.21.15.116:0/566704107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 msgr2=0x7fca30196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 0x7fca30196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 -- 172.21.15.116:0/566704107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca28009900 con 0x7fca301021e0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca34c39640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 0x7fca30196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca3543a640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca3019dbc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fca28037040 tx=0x7fca28032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28003970 con 0x7fca301021e0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca28004040 con 0x7fca301021e0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca30068870 con 0x7fca301021e0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2803d630 con 0x7fca301021e0 2024-03-24T09:19:08.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.754+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca30068da0 con 0x7fca301021e0 2024-03-24T09:19:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.755+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca30068500 con 0x7fca301021e0 2024-03-24T09:19:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.756+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fca2803b040 con 0x7fca301021e0 2024-03-24T09:19:08.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.757+0000 7fca25ffb640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fca180a20c0 0x7fca180a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:08.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.757+0000 7fca34c39640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fca180a20c0 0x7fca180a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:08.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.757+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fca280f05f0 con 0x7fca301021e0 2024-03-24T09:19:08.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.758+0000 7fca34c39640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fca180a20c0 0x7fca180a4580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fca30101680 tx=0x7fca2c00a560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:08.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.761+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fca280b9a40 con 0x7fca301021e0 2024-03-24T09:19:08.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.990+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca301074e0 con 0x7fca301021e0 2024-03-24T09:19:08.991 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:08.991 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T09:19:08.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.991+0000 7fca25ffb640 1 -- 172.21.15.116:0/566704107 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fca280bd100 con 0x7fca301021e0 2024-03-24T09:19:08.992 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 4 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fca180a20c0 msgr2=0x7fca180a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fca180a20c0 0x7fca180a4580 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fca30101680 tx=0x7fca2c00a560 comp rx=0 tx=0).stop 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 msgr2=0x7fca3019dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca3019dbc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fca28037040 tx=0x7fca28032980 comp rx=0 tx=0).stop 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 shutdown_connections 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca301021e0 0x7fca3019dbc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fca180a20c0 0x7fca180a4580 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca30101820 0x7fca30196b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 --2- 172.21.15.116:0/566704107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca30100630 0x7fca30196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 >> 172.21.15.116:0/566704107 conn(0x7fca300fbba0 msgr2=0x7fca30104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 shutdown_connections 2024-03-24T09:19:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:08.993+0000 7fca36e9d640 1 -- 172.21.15.116:0/566704107 wait complete. 2024-03-24T09:19:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:09 smithi143 ceph-mon[35965]: Deploying daemon mds.d on smithi116 2024-03-24T09:19:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:09 smithi143 ceph-mon[35965]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:09 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/566704107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:09 smithi117 ceph-mon[35788]: Deploying daemon mds.d on smithi116 2024-03-24T09:19:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:09 smithi117 ceph-mon[35788]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:09 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/566704107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:09 smithi116 ceph-mon[32207]: Deploying daemon mds.d on smithi116 2024-03-24T09:19:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:09 smithi116 ceph-mon[32207]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:09 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/566704107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:09.318 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 systemd[1]: Starting Ceph mds.d for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:09.715 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 podman[73027]: 2024-03-24 09:19:09.325473829 +0000 UTC m=+0.018216580 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:09.716 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 podman[73027]: 2024-03-24 09:19:09.480122437 +0000 UTC m=+0.172865214 container create e02a1bbc976816117595a9b871ae6191dbf780f505306a6edc8ad1fa7f45b949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T09:19:09.717 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 podman[73027]: 2024-03-24 09:19:09.716075759 +0000 UTC m=+0.408818506 container init e02a1bbc976816117595a9b871ae6191dbf780f505306a6edc8ad1fa7f45b949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T09:19:10.068 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 podman[73027]: 2024-03-24 09:19:09.718654592 +0000 UTC m=+0.411397335 container start e02a1bbc976816117595a9b871ae6191dbf780f505306a6edc8ad1fa7f45b949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T09:19:10.068 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 ceph-mds[73046]: -- 172.21.15.116:0/94269415 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564bb45b61a0 con 0x564bb458c800 2024-03-24T09:19:10.068 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d[73042]: starting mds.d at 2024-03-24T09:19:10.068 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 bash[73027]: e02a1bbc976816117595a9b871ae6191dbf780f505306a6edc8ad1fa7f45b949 2024-03-24T09:19:10.068 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:19:09 smithi116 systemd[1]: Started Ceph mds.d for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:11.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:11.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: Deploying daemon mds.e on smithi117 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.116:6836/1967493794,v1:172.21.15.116:6837/1967493794] up:boot 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: fsmap 4 up:standby 2024-03-24T09:19:11.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:10 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T09:19:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 -- 172.21.15.116:0/2996116191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0100630 msgr2=0x7f1bd0100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/2996116191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0100630 0x7f1bd0100a50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc009900 tx=0x7f1bcc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 -- 172.21.15.116:0/2996116191 shutdown_connections 2024-03-24T09:19:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/2996116191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd01021e0 0x7f1bd0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/2996116191 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd0101820 0x7f1bd0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/2996116191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0100630 0x7f1bd0100a50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 -- 172.21.15.116:0/2996116191 >> 172.21.15.116:0/2996116191 conn(0x7f1bd00fbba0 msgr2=0x7f1bd00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 -- 172.21.15.116:0/2996116191 shutdown_connections 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.123+0000 7f1bd7d84640 1 -- 172.21.15.116:0/2996116191 wait complete. 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 Processor -- start 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 -- start start 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd0100630 0x7f1bd0197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 0x7f1bd0197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:11.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd01021e0 0x7f1bd019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd019f200 con 0x7f1bd0101820 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd019f370 con 0x7f1bd01021e0 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd7d84640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd019f530 con 0x7f1bd0100630 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 0x7f1bd0197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 0x7f1bd0197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54118/0 (socket says 172.21.15.116:54118) 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 -- 172.21.15.116:0/693852125 learned_addr learned my addr 172.21.15.116:0/693852125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:11.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd6321640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd01021e0 0x7f1bd019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd5b20640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd0100630 0x7f1bd0197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 -- 172.21.15.116:0/693852125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd0100630 msgr2=0x7f1bd0197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd0100630 0x7f1bd0197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 -- 172.21.15.116:0/693852125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd01021e0 msgr2=0x7f1bd019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd01021e0 0x7f1bd019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 -- 172.21.15.116:0/693852125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bcc009560 con 0x7f1bd0101820 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.124+0000 7f1bd531f640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 0x7f1bd0197ca0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c890 tx=0x7f1bc000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:11.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.125+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0004220 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.125+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bc0007e60 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.125+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd019f810 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.125+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0004a00 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.125+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd019fd20 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.126+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd010bef0 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.127+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f1bc00079b0 con 0x7f1bd0101820 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.127+0000 7f1bc6ffd640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1bbc0a20c0 0x7f1bbc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.127+0000 7f1bd5b20640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1bbc0a20c0 0x7f1bbc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:11.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.127+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1bc00cd500 con 0x7f1bd0101820 2024-03-24T09:19:11.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.128+0000 7f1bd5b20640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1bbc0a20c0 0x7f1bbc0a4580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc006010 tx=0x7f1bcc000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:11.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.132+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1bc0096970 con 0x7f1bd0101820 2024-03-24T09:19:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: Deploying daemon mds.e on smithi117 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.116:6836/1967493794,v1:172.21.15.116:6837/1967493794] up:boot 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: fsmap 4 up:standby 2024-03-24T09:19:11.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:10 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T09:19:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:11.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:11.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: Deploying daemon mds.e on smithi117 2024-03-24T09:19:11.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.116:6836/1967493794,v1:172.21.15.116:6837/1967493794] up:boot 2024-03-24T09:19:11.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: fsmap 4 up:standby 2024-03-24T09:19:11.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:10 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T09:19:11.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.343+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bd01a0150 con 0x7f1bd0101820 2024-03-24T09:19:11.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.344+0000 7f1bc6ffd640 1 -- 172.21.15.116:0/693852125 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3877 (secure 0 0 0) 0x7f1bc009a030 con 0x7f1bd0101820 2024-03-24T09:19:11.345 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:11.345 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T09:19:11.345 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 5 2024-03-24T09:19:11.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1bbc0a20c0 msgr2=0x7f1bbc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:11.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1bbc0a20c0 0x7f1bbc0a4580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc006010 tx=0x7f1bcc000bc0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 msgr2=0x7f1bd0197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:11.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 0x7f1bd0197ca0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1bc000c890 tx=0x7f1bc000cd50 comp rx=0 tx=0).stop 2024-03-24T09:19:11.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 shutdown_connections 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd01021e0 0x7f1bd019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1bd0101820 0x7f1bd0197ca0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1bbc0a20c0 0x7f1bbc0a4580 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 --2- 172.21.15.116:0/693852125 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1bd0100630 0x7f1bd0197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 >> 172.21.15.116:0/693852125 conn(0x7f1bd00fbba0 msgr2=0x7f1bd0068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 shutdown_connections 2024-03-24T09:19:11.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:11.346+0000 7f1bd7d84640 1 -- 172.21.15.116:0/693852125 wait complete. 2024-03-24T09:19:12.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:11 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/693852125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:12.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:11 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/693852125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:12.317 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 systemd[1]: Starting Ceph mds.e for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:12.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:11 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/693852125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:12.679 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 podman[65007]: 2024-03-24 09:19:12.279101368 +0000 UTC m=+0.017585016 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:12.679 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 podman[65007]: 2024-03-24 09:19:12.40997425 +0000 UTC m=+0.148457946 container create b6d6ed44dcd8b9b732d2e41ccd9aba686338ceb095ea3346f07090340d52826c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T09:19:12.679 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 podman[65007]: 2024-03-24 09:19:12.679375261 +0000 UTC m=+0.417858921 container init b6d6ed44dcd8b9b732d2e41ccd9aba686338ceb095ea3346f07090340d52826c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:19:13.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:13.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 podman[65007]: 2024-03-24 09:19:12.682362961 +0000 UTC m=+0.420846609 container start b6d6ed44dcd8b9b732d2e41ccd9aba686338ceb095ea3346f07090340d52826c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-mds[65026]: -- 172.21.15.117:0/2709123393 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d7a9481a0 con 0x558d7a91ec00 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e[65022]: starting mds.e at 2024-03-24T09:19:13.067 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 bash[65007]: b6d6ed44dcd8b9b732d2e41ccd9aba686338ceb095ea3346f07090340d52826c 2024-03-24T09:19:13.068 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:19:12 smithi117 systemd[1]: Started Ceph mds.e for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:13.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:13.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.177 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:13.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:13.178 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:12 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:12 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:13.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:13.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 -- 172.21.15.116:0/3054745220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341018a0 msgr2=0x7f4634101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 --2- 172.21.15.116:0/3054745220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341018a0 0x7f4634101d20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4628009900 tx=0x7f462802bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 -- 172.21.15.116:0/3054745220 shutdown_connections 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 --2- 172.21.15.116:0/3054745220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4634102260 0x7f4634108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 --2- 172.21.15.116:0/3054745220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341018a0 0x7f4634101d20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 --2- 172.21.15.116:0/3054745220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46341006b0 0x7f4634100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 -- 172.21.15.116:0/3054745220 >> 172.21.15.116:0/3054745220 conn(0x7f46340fbc40 msgr2=0x7f46340fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 -- 172.21.15.116:0/3054745220 shutdown_connections 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 -- 172.21.15.116:0/3054745220 wait complete. 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.472+0000 7f4639dd7640 1 Processor -- start 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 -- start start 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 0x7f46341965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46341018a0 0x7f4634196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4634102260 0x7f463419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f463419e0e0 con 0x7f46341006b0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f463419e250 con 0x7f4634102260 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4639dd7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f463419e410 con 0x7f46341018a0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 0x7f46341965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 0x7f46341965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54126/0 (socket says 172.21.15.116:54126) 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 -- 172.21.15.116:0/1820546078 learned_addr learned my addr 172.21.15.116:0/1820546078 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4632ffd640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46341018a0 0x7f4634196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 -- 172.21.15.116:0/1820546078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46341018a0 msgr2=0x7f4634196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46341018a0 0x7f4634196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 -- 172.21.15.116:0/1820546078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4634102260 msgr2=0x7f463419dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f4633fff640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4634102260 0x7f463419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4634102260 0x7f463419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 -- 172.21.15.116:0/1820546078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4628009560 con 0x7f46341006b0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.473+0000 7f46337fe640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 0x7f46341965f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f462400d770 tx=0x7f462400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.474+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4624009c30 con 0x7f46341006b0 2024-03-24T09:19:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.474+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f46240044d0 con 0x7f46341006b0 2024-03-24T09:19:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.474+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4624003de0 con 0x7f46341006b0 2024-03-24T09:19:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.474+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463419e6f0 con 0x7f46341006b0 2024-03-24T09:19:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.474+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463419ec20 con 0x7f46341006b0 2024-03-24T09:19:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.475+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4634107a40 con 0x7f46341006b0 2024-03-24T09:19:13.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.476+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f4624003f40 con 0x7f46341006b0 2024-03-24T09:19:13.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.476+0000 7f4630ff9640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f461c0a2140 0x7f461c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:13.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.476+0000 7f4632ffd640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f461c0a2140 0x7f461c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:13.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.476+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f46240cd100 con 0x7f46341006b0 2024-03-24T09:19:13.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.477+0000 7f4632ffd640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f461c0a2140 0x7f461c0a4600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4628009230 tx=0x7f4628005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:13.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.481+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4624096570 con 0x7f46341006b0 2024-03-24T09:19:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.692+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f463410bf00 con 0x7f46341006b0 2024-03-24T09:19:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.692+0000 7f4630ff9640 1 -- 172.21.15.116:0/1820546078 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4705 (secure 0 0 0) 0x7f463410bf00 con 0x7f46341006b0 2024-03-24T09:19:13.693 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:13.693 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-24T09:19:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 6 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.694+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f461c0a2140 msgr2=0x7f461c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.694+0000 7f4639dd7640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f461c0a2140 0x7f461c0a4600 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4628009230 tx=0x7f4628005fd0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.694+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 msgr2=0x7f46341965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.694+0000 7f4639dd7640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 0x7f46341965f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f462400d770 tx=0x7f462400dc30 comp rx=0 tx=0).stop 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 shutdown_connections 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4634102260 0x7f463419dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f461c0a2140 0x7f461c0a4600 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f46341018a0 0x7f4634196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 --2- 172.21.15.116:0/1820546078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46341006b0 0x7f46341965f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 >> 172.21.15.116:0/1820546078 conn(0x7f46340fbc40 msgr2=0x7f4634104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 shutdown_connections 2024-03-24T09:19:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:13.695+0000 7f4639dd7640 1 -- 172.21.15.116:0/1820546078 wait complete. 2024-03-24T09:19:14.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:13 smithi143 ceph-mon[35965]: Deploying daemon mds.f on smithi143 2024-03-24T09:19:14.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:13 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] up:boot 2024-03-24T09:19:14.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:13 smithi143 ceph-mon[35965]: fsmap 5 up:standby 2024-03-24T09:19:14.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:13 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:19:14.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1820546078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:14.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:13 smithi116 ceph-mon[32207]: Deploying daemon mds.f on smithi143 2024-03-24T09:19:14.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:13 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] up:boot 2024-03-24T09:19:14.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:13 smithi116 ceph-mon[32207]: fsmap 5 up:standby 2024-03-24T09:19:14.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:13 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:19:14.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1820546078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:13 smithi117 ceph-mon[35788]: Deploying daemon mds.f on smithi143 2024-03-24T09:19:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:13 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] up:boot 2024-03-24T09:19:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:13 smithi117 ceph-mon[35788]: fsmap 5 up:standby 2024-03-24T09:19:14.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:13 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:19:14.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1820546078' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:15.238 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:14 smithi143 ceph-mon[35965]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:15.238 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 systemd[1]: Starting Ceph mds.f for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:14 smithi117 ceph-mon[35788]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:14 smithi116 ceph-mon[32207]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:15.512 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 podman[66222]: 2024-03-24 09:19:15.140189481 +0000 UTC m=+0.017912495 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:15.512 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 podman[66222]: 2024-03-24 09:19:15.267214842 +0000 UTC m=+0.144937852 container create f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T09:19:15.512 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 podman[66222]: 2024-03-24 09:19:15.512113953 +0000 UTC m=+0.389836968 container init f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T09:19:15.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:15.774 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 podman[66222]: 2024-03-24 09:19:15.514794019 +0000 UTC m=+0.392517038 container start f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:19:15.774 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 ceph-mds[66241]: -- 172.21.15.143:0/2385066612 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d9bceea1a0 con 0x55d9bcec0c00 2024-03-24T09:19:15.774 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f[66237]: starting mds.f at 2024-03-24T09:19:15.774 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 bash[66222]: f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 2024-03-24T09:19:15.775 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:19:15 smithi143 systemd[1]: Started Ceph mds.f for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 -- 172.21.15.116:0/1312668064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 msgr2=0x7f17c0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 --2- 172.21.15.116:0/1312668064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0101ca0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009900 tx=0x7f17b402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 -- 172.21.15.116:0/1312668064 shutdown_connections 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 --2- 172.21.15.116:0/1312668064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c01021e0 0x7f17c0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 --2- 172.21.15.116:0/1312668064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0101ca0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 --2- 172.21.15.116:0/1312668064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c0100630 0x7f17c0100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:15.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 -- 172.21.15.116:0/1312668064 >> 172.21.15.116:0/1312668064 conn(0x7f17c00fbba0 msgr2=0x7f17c00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:15.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 -- 172.21.15.116:0/1312668064 shutdown_connections 2024-03-24T09:19:15.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 -- 172.21.15.116:0/1312668064 wait complete. 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 Processor -- start 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.823+0000 7f17c4884640 1 -- start start 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c0100630 0x7f17c0197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c01021e0 0x7f17c019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17c019f270 con 0x7f17c0101820 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f17c019f3e0 con 0x7f17c01021e0 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f17c019f5a0 con 0x7f17c0100630 2024-03-24T09:19:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54148/0 (socket says 172.21.15.116:54148) 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 -- 172.21.15.116:0/4032538947 learned_addr learned my addr 172.21.15.116:0/4032538947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17be59c640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c0100630 0x7f17c0197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bed9d640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c01021e0 0x7f17c019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 -- 172.21.15.116:0/4032538947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c0100630 msgr2=0x7f17c0197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c0100630 0x7f17c0197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 -- 172.21.15.116:0/4032538947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c01021e0 msgr2=0x7f17c019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c01021e0 0x7f17c019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 -- 172.21.15.116:0/4032538947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b4009560 con 0x7f17c0101820 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bed9d640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c01021e0 0x7f17c019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17bdd9b640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0197cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f17b402c7d0 tx=0x7f17b4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4004500 con 0x7f17c0101820 2024-03-24T09:19:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c019f820 con 0x7f17c0101820 2024-03-24T09:19:15.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17b402de90 con 0x7f17c0101820 2024-03-24T09:19:15.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.824+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4005350 con 0x7f17c0101820 2024-03-24T09:19:15.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.825+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c019fd80 con 0x7f17c0101820 2024-03-24T09:19:15.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.826+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c010bef0 con 0x7f17c0101820 2024-03-24T09:19:15.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.827+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f17b40054b0 con 0x7f17c0101820 2024-03-24T09:19:15.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.827+0000 7f17af7fe640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17a80a20c0 0x7f17a80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:15.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.827+0000 7f17be59c640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17a80a20c0 0x7f17a80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:15.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.827+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f17b40ef950 con 0x7f17c0101820 2024-03-24T09:19:15.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.828+0000 7f17be59c640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17a80a20c0 0x7f17a80a4580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f17c0101680 tx=0x7f17b0009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:15.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:15.832+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f17b40b8dc0 con 0x7f17c0101820 2024-03-24T09:19:16.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.044+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17c01a01f0 con 0x7f17c0101820 2024-03-24T09:19:16.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.045+0000 7f17af7fe640 1 -- 172.21.15.116:0/4032538947 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5533 (secure 0 0 0) 0x7f17b40bc480 con 0x7f17c0101820 2024-03-24T09:19:16.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:16.045 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-24T09:19:16.046 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 7 2024-03-24T09:19:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17a80a20c0 msgr2=0x7f17a80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17a80a20c0 0x7f17a80a4580 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f17c0101680 tx=0x7f17b0009500 comp rx=0 tx=0).stop 2024-03-24T09:19:16.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 msgr2=0x7f17c0197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0197cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f17b402c7d0 tx=0x7f17b4004040 comp rx=0 tx=0).stop 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 shutdown_connections 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c01021e0 0x7f17c019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c0101820 0x7f17c0197cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17a80a20c0 0x7f17a80a4580 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 --2- 172.21.15.116:0/4032538947 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f17c0100630 0x7f17c0197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 >> 172.21.15.116:0/4032538947 conn(0x7f17c00fbba0 msgr2=0x7f17c00fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 shutdown_connections 2024-03-24T09:19:16.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:16.047+0000 7f17c4884640 1 -- 172.21.15.116:0/4032538947 wait complete. 2024-03-24T09:19:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: Deploying daemon mds.c on smithi143 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] up:boot 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: fsmap 6 up:standby 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:19:16.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:16 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4032538947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: Deploying daemon mds.c on smithi143 2024-03-24T09:19:17.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:17.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] up:boot 2024-03-24T09:19:17.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: fsmap 6 up:standby 2024-03-24T09:19:17.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:19:17.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:16 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4032538947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: Deploying daemon mds.c on smithi143 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] up:boot 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: fsmap 6 up:standby 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:19:17.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:16 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4032538947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:18.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 -- 172.21.15.116:0/2581371929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69001018a0 msgr2=0x7f6900101d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 --2- 172.21.15.116:0/2581371929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69001018a0 0x7f6900101d20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f68f4009900 tx=0x7f68f402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 -- 172.21.15.116:0/2581371929 shutdown_connections 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 --2- 172.21.15.116:0/2581371929 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6900102260 0x7f6900108af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 --2- 172.21.15.116:0/2581371929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69001018a0 0x7f6900101d20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 --2- 172.21.15.116:0/2581371929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69001006b0 0x7f6900100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 -- 172.21.15.116:0/2581371929 >> 172.21.15.116:0/2581371929 conn(0x7f69000fbc40 msgr2=0x7f69000fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 -- 172.21.15.116:0/2581371929 shutdown_connections 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.174+0000 7f6905d6b640 1 -- 172.21.15.116:0/2581371929 wait complete. 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 Processor -- start 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 -- start start 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69001006b0 0x7f690010af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69001018a0 0x7f69001095b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:18.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 0x7f6900109af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f690010a250 con 0x7f6900102260 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f690010a3c0 con 0x7f69001006b0 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f6905d6b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f69001a1740 con 0x7f69001018a0 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f68fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 0x7f6900109af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f68fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 0x7f6900109af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48374/0 (socket says 172.21.15.116:48374) 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f68fffff640 1 -- 172.21.15.116:0/3352664826 learned_addr learned my addr 172.21.15.116:0/3352664826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.175+0000 7f68feffd640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69001018a0 0x7f69001095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fffff640 1 -- 172.21.15.116:0/3352664826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69001018a0 msgr2=0x7f69001095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68ff7fe640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69001006b0 0x7f690010af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fffff640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69001018a0 0x7f69001095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fffff640 1 -- 172.21.15.116:0/3352664826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69001006b0 msgr2=0x7f690010af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fffff640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69001006b0 0x7f690010af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fffff640 1 -- 172.21.15.116:0/3352664826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f4009560 con 0x7f6900102260 2024-03-24T09:19:18.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fffff640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 0x7f6900109af0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f68f800ead0 tx=0x7f68f800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f800cde0 con 0x7f6900102260 2024-03-24T09:19:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69001a19f0 con 0x7f6900102260 2024-03-24T09:19:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68f8005020 con 0x7f6900102260 2024-03-24T09:19:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f80116d0 con 0x7f6900102260 2024-03-24T09:19:18.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.176+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69001a1f20 con 0x7f6900102260 2024-03-24T09:19:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.177+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69001079d0 con 0x7f6900102260 2024-03-24T09:19:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.178+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f68f8005190 con 0x7f6900102260 2024-03-24T09:19:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.179+0000 7f68fcff9640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f68e80a20c0 0x7f68e80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.179+0000 7f68ff7fe640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f68e80a20c0 0x7f68e80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.179+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f68f80cdce0 con 0x7f6900102260 2024-03-24T09:19:18.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.179+0000 7f68ff7fe640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f68e80a20c0 0x7f68e80a4580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f68f0005fd0 tx=0x7f68f0005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:18.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.182+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f68f8097150 con 0x7f6900102260 2024-03-24T09:19:18.275 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 podman[66536]: 2024-03-24 09:19:18.148985117 +0000 UTC m=+0.138738047 container create 8b385a11dbd3fca8708a3e6a1b2747e55b2377f1c3dfc38effd6bbfc448921d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-24T09:19:18.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.393+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f690010d2a0 con 0x7f6900102260 2024-03-24T09:19:18.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.394+0000 7f68fcff9640 1 -- 172.21.15.116:0/3352664826 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5533 (secure 0 0 0) 0x7f68f801f090 con 0x7f6900102260 2024-03-24T09:19:18.394 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:18.394 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-24T09:19:18.395 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 7 2024-03-24T09:19:18.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.395+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f68e80a20c0 msgr2=0x7f68e80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:18.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.395+0000 7f6905d6b640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f68e80a20c0 0x7f68e80a4580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f68f0005fd0 tx=0x7f68f0005c70 comp rx=0 tx=0).stop 2024-03-24T09:19:18.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.395+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 msgr2=0x7f6900109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:18.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 0x7f6900109af0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f68f800ead0 tx=0x7f68f800ef90 comp rx=0 tx=0).stop 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 shutdown_connections 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6900102260 0x7f6900109af0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f68e80a20c0 0x7f68e80a4580 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f69001018a0 0x7f69001095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 --2- 172.21.15.116:0/3352664826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69001006b0 0x7f690010af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 >> 172.21.15.116:0/3352664826 conn(0x7f69000fbc40 msgr2=0x7f6900104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 shutdown_connections 2024-03-24T09:19:18.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:18.396+0000 7f6905d6b640 1 -- 172.21.15.116:0/3352664826 wait complete. 2024-03-24T09:19:18.544 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 podman[66536]: 2024-03-24 09:19:18.444060084 +0000 UTC m=+0.433813018 container init 8b385a11dbd3fca8708a3e6a1b2747e55b2377f1c3dfc38effd6bbfc448921d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True) 2024-03-24T09:19:18.544 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 podman[66536]: 2024-03-24 09:19:18.446698862 +0000 UTC m=+0.436451795 container start 8b385a11dbd3fca8708a3e6a1b2747e55b2377f1c3dfc38effd6bbfc448921d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:19:18.544 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mds[66555]: -- 172.21.15.143:0/1888429778 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d8ce9341a0 con 0x55d8ce90ac00 2024-03-24T09:19:18.545 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c[66551]: starting mds.c at 2024-03-24T09:19:18.545 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 bash[66536]: 8b385a11dbd3fca8708a3e6a1b2747e55b2377f1c3dfc38effd6bbfc448921d6 2024-03-24T09:19:18.970 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:19:18 smithi143 systemd[1]: Started Ceph mds.c for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3352664826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:18 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3352664826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:19.280 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:19.281 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:18 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3352664826' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:19.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:18 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:19 smithi143 ceph-mon[35965]: Deploying daemon mds.h on smithi117 2024-03-24T09:19:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:19 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6836/3300095096,v1:172.21.15.143:6837/3300095096] up:boot 2024-03-24T09:19:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:19 smithi143 ceph-mon[35965]: fsmap 7 up:standby 2024-03-24T09:19:20.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:19 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:19:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:19 smithi117 ceph-mon[35788]: Deploying daemon mds.h on smithi117 2024-03-24T09:19:20.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:19 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6836/3300095096,v1:172.21.15.143:6837/3300095096] up:boot 2024-03-24T09:19:20.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:19 smithi117 ceph-mon[35788]: fsmap 7 up:standby 2024-03-24T09:19:20.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:19 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:19:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:19 smithi116 ceph-mon[32207]: Deploying daemon mds.h on smithi117 2024-03-24T09:19:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:19 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6836/3300095096,v1:172.21.15.143:6837/3300095096] up:boot 2024-03-24T09:19:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:19 smithi116 ceph-mon[32207]: fsmap 7 up:standby 2024-03-24T09:19:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:19 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:19:20.408 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 -- 172.21.15.116:0/4262323120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b64100040 msgr2=0x7f6b64100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:20.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 --2- 172.21.15.116:0/4262323120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b64100040 0x7f6b64100460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009900 tx=0x7f6b5402bcd0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 -- 172.21.15.116:0/4262323120 shutdown_connections 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 --2- 172.21.15.116:0/4262323120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b64101340 0x7f6b641017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 --2- 172.21.15.116:0/4262323120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b641009a0 0x7f6b64100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 --2- 172.21.15.116:0/4262323120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b64100040 0x7f6b64100460 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 -- 172.21.15.116:0/4262323120 >> 172.21.15.116:0/4262323120 conn(0x7f6b640fbbc0 msgr2=0x7f6b640fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 -- 172.21.15.116:0/4262323120 shutdown_connections 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.519+0000 7f6b694f4640 1 -- 172.21.15.116:0/4262323120 wait complete. 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 Processor -- start 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 -- start start 2024-03-24T09:19:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b64100040 0x7f6b64199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 0x7f6b64199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b64101340 0x7f6b641a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b641a1430 con 0x7f6b641009a0 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6b641a15a0 con 0x7f6b64100040 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b694f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6b641a1760 con 0x7f6b64101340 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b627fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 0x7f6b64199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b627fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 0x7f6b64199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48398/0 (socket says 172.21.15.116:48398) 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.520+0000 7f6b627fc640 1 -- 172.21.15.116:0/3931797866 learned_addr learned my addr 172.21.15.116:0/3931797866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b62ffd640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b64100040 0x7f6b64199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b637fe640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b64101340 0x7f6b641a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b627fc640 1 -- 172.21.15.116:0/3931797866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b64101340 msgr2=0x7f6b641a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b627fc640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b64101340 0x7f6b641a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b627fc640 1 -- 172.21.15.116:0/3931797866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b64100040 msgr2=0x7f6b64199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b627fc640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b64100040 0x7f6b64199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b627fc640 1 -- 172.21.15.116:0/3931797866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b54009560 con 0x7f6b641009a0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b627fc640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 0x7f6b64199e80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800d770 tx=0x7f6b5800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58009c30 con 0x7f6b641009a0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b5800de80 con 0x7f6b641009a0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b641a1a40 con 0x7f6b641009a0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58004e00 con 0x7f6b641009a0 2024-03-24T09:19:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.521+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b641a1f00 con 0x7f6b641009a0 2024-03-24T09:19:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.522+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6410b010 con 0x7f6b641009a0 2024-03-24T09:19:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.523+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f6b58004f60 con 0x7f6b641009a0 2024-03-24T09:19:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.524+0000 7f6b4bfff640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6b4c0a20c0 0x7f6b4c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.524+0000 7f6b62ffd640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6b4c0a20c0 0x7f6b4c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.524+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6b580cd290 con 0x7f6b641009a0 2024-03-24T09:19:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.524+0000 7f6b62ffd640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6b4c0a20c0 0x7f6b4c0a4580 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009230 tx=0x7f6b54005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.528+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6b58096700 con 0x7f6b641009a0 2024-03-24T09:19:20.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.741+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b6410f200 con 0x7f6b641009a0 2024-03-24T09:19:20.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.742+0000 7f6b4bfff640 1 -- 172.21.15.116:0/3931797866 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6361 (secure 0 0 0) 0x7f6b58099dc0 con 0x7f6b641009a0 2024-03-24T09:19:20.743 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:20.743 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-24T09:19:20.744 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2024-03-24T09:19:20.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6b4c0a20c0 msgr2=0x7f6b4c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6b4c0a20c0 0x7f6b4c0a4580 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009230 tx=0x7f6b54005fd0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 msgr2=0x7f6b64199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 0x7f6b64199e80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800d770 tx=0x7f6b5800dc30 comp rx=0 tx=0).stop 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 shutdown_connections 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f6b4c0a20c0 0x7f6b4c0a4580 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b64101340 0x7f6b641a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b641009a0 0x7f6b64199e80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 --2- 172.21.15.116:0/3931797866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b64100040 0x7f6b64199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 >> 172.21.15.116:0/3931797866 conn(0x7f6b640fbbc0 msgr2=0x7f6b640fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 shutdown_connections 2024-03-24T09:19:20.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:20.744+0000 7f6b694f4640 1 -- 172.21.15.116:0/3931797866 wait complete. 2024-03-24T09:19:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:20 smithi117 ceph-mon[35788]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:21.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:20 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3931797866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:21.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:20 smithi116 ceph-mon[32207]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:21.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:20 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3931797866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:20 smithi143 ceph-mon[35965]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:20 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3931797866' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:21.816 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:19:21 smithi117 systemd[1]: Started Ceph mds.h for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:22.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 -- 172.21.15.116:0/132005559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec101be0 msgr2=0x7f63ec102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 --2- 172.21.15.116:0/132005559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec101be0 0x7f63ec102000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f63dc009930 tx=0x7f63dc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 -- 172.21.15.116:0/132005559 shutdown_connections 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 --2- 172.21.15.116:0/132005559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec1012d0 0x7f63ec108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 --2- 172.21.15.116:0/132005559 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ec100910 0x7f63ec100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 --2- 172.21.15.116:0/132005559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec101be0 0x7f63ec102000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 -- 172.21.15.116:0/132005559 >> 172.21.15.116:0/132005559 conn(0x7f63ec0fb340 msgr2=0x7f63ec0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:22.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 -- 172.21.15.116:0/132005559 shutdown_connections 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.871+0000 7f63f1e15640 1 -- 172.21.15.116:0/132005559 wait complete. 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 Processor -- start 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 -- start start 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec100910 0x7f63ec100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 0x7f63ec19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ec101be0 0x7f63ec19ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec19f1f0 con 0x7f63ec1012d0 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec19f360 con 0x7f63ec100910 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63f1e15640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec19f520 con 0x7f63ec101be0 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 0x7f63ec19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:22.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 0x7f63ec19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48416/0 (socket says 172.21.15.116:48416) 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 -- 172.21.15.116:0/3019573626 learned_addr learned my addr 172.21.15.116:0/3019573626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eb7fe640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec100910 0x7f63ec100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 -- 172.21.15.116:0/3019573626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ec101be0 msgr2=0x7f63ec19ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63ebfff640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ec101be0 0x7f63ec19ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ec101be0 0x7f63ec19ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 -- 172.21.15.116:0/3019573626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec100910 msgr2=0x7f63ec100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec100910 0x7f63ec100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eaffd640 1 -- 172.21.15.116:0/3019573626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc009560 con 0x7f63ec1012d0 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.872+0000 7f63eb7fe640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec100910 0x7f63ec100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.873+0000 7f63eaffd640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 0x7f63ec19c420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c920 tx=0x7f63e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:22.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.873+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0007c70 con 0x7f63ec1012d0 2024-03-24T09:19:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.873+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f63e0007dd0 con 0x7f63ec1012d0 2024-03-24T09:19:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.873+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e00187c0 con 0x7f63ec1012d0 2024-03-24T09:19:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.873+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ec19f800 con 0x7f63ec1012d0 2024-03-24T09:19:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.873+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ec19fd60 con 0x7f63ec1012d0 2024-03-24T09:19:22.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.874+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ec107ac0 con 0x7f63ec1012d0 2024-03-24T09:19:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.875+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f63e000f450 con 0x7f63ec1012d0 2024-03-24T09:19:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.875+0000 7f63e8ff9640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63d40a2140 0x7f63d40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.876+0000 7f63eb7fe640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63d40a2140 0x7f63d40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.876+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f63e00cdc00 con 0x7f63ec1012d0 2024-03-24T09:19:22.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.876+0000 7f63eb7fe640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63d40a2140 0x7f63d40a4600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63dc036040 tx=0x7f63dc000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:22.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:22.879+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f63e0097180 con 0x7f63ec1012d0 2024-03-24T09:19:23.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: Deploying daemon mds.i on smithi143 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6836/309438710,v1:172.21.15.117:6837/309438710] up:boot 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: fsmap 8 up:standby 2024-03-24T09:19:23.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:22 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:19:23.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: Deploying daemon mds.i on smithi143 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6836/309438710,v1:172.21.15.117:6837/309438710] up:boot 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: fsmap 8 up:standby 2024-03-24T09:19:23.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:22 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:23.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: Deploying daemon mds.i on smithi143 2024-03-24T09:19:23.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:23.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6836/309438710,v1:172.21.15.117:6837/309438710] up:boot 2024-03-24T09:19:23.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: fsmap 8 up:standby 2024-03-24T09:19:23.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:22 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:19:23.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.090+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63ec10bf30 con 0x7f63ec1012d0 2024-03-24T09:19:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.090+0000 7f63e8ff9640 1 -- 172.21.15.116:0/3019573626 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7186 (secure 0 0 0) 0x7f63e009a840 con 0x7f63ec1012d0 2024-03-24T09:19:23.092 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:23.092 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-24T09:19:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 9 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63d40a2140 msgr2=0x7f63d40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63d40a2140 0x7f63d40a4600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f63dc036040 tx=0x7f63dc000bc0 comp rx=0 tx=0).stop 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 msgr2=0x7f63ec19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 0x7f63ec19c420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f63e000c920 tx=0x7f63e000cde0 comp rx=0 tx=0).stop 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 shutdown_connections 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f63d40a2140 0x7f63d40a4600 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63ec101be0 0x7f63ec19ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63ec1012d0 0x7f63ec19c420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 --2- 172.21.15.116:0/3019573626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63ec100910 0x7f63ec100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 >> 172.21.15.116:0/3019573626 conn(0x7f63ec0fb340 msgr2=0x7f63ec104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 shutdown_connections 2024-03-24T09:19:23.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:23.092+0000 7f63f1e15640 1 -- 172.21.15.116:0/3019573626 wait complete. 2024-03-24T09:19:24.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:23 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3019573626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:24.025 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:23 smithi143 systemd[1]: Starting Ceph mds.i for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:24.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:23 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3019573626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:24.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:23 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3019573626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:24.334 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 podman[66844]: 2024-03-24 09:19:23.978774551 +0000 UTC m=+0.017546462 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:24.334 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 podman[66844]: 2024-03-24 09:19:24.105886867 +0000 UTC m=+0.144658781 container create 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T09:19:24.334 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 podman[66844]: 2024-03-24 09:19:24.334266653 +0000 UTC m=+0.373038569 container init 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:19:24.700 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 podman[66844]: 2024-03-24 09:19:24.337312743 +0000 UTC m=+0.376084657 container start 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T09:19:24.700 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mds[66863]: -- 172.21.15.143:0/67182808 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f08d7641a0 con 0x55f08d73a800 2024-03-24T09:19:24.700 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i[66859]: starting mds.i at 2024-03-24T09:19:24.700 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 bash[66844]: 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 2024-03-24T09:19:24.700 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:19:24 smithi143 systemd[1]: Started Ceph mds.i for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:24.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:24.957 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:24.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:24.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:24.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:24.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:24.958 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:24 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:25.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:25.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:25.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:24 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:24 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:25.106 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:25.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 -- 172.21.15.116:0/1715442163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 msgr2=0x7f7864101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:25.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 --2- 172.21.15.116:0/1715442163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864101ca0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7854009900 tx=0x7f785402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 -- 172.21.15.116:0/1715442163 shutdown_connections 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 --2- 172.21.15.116:0/1715442163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78641021e0 0x7f7864108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 --2- 172.21.15.116:0/1715442163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864101ca0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 --2- 172.21.15.116:0/1715442163 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7864100630 0x7f7864100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.209+0000 7f786bbd7640 1 -- 172.21.15.116:0/1715442163 >> 172.21.15.116:0/1715442163 conn(0x7f78640fbba0 msgr2=0x7f78640fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 -- 172.21.15.116:0/1715442163 shutdown_connections 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 -- 172.21.15.116:0/1715442163 wait complete. 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 Processor -- start 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 -- start start 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f78641976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:25.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78641021e0 0x7f786419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f786419f190 con 0x7f7864101820 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f786419f300 con 0x7f7864100630 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f786bbd7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f786419f4c0 con 0x7f78641021e0 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f7869172640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f7869172640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48434/0 (socket says 172.21.15.116:48434) 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.210+0000 7f7869172640 1 -- 172.21.15.116:0/3751498842 learned_addr learned my addr 172.21.15.116:0/3751498842 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869973640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f78641976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f786a174640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78641021e0 0x7f786419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:25.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869172640 1 -- 172.21.15.116:0/3751498842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78641021e0 msgr2=0x7f786419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:25.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869172640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78641021e0 0x7f786419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869172640 1 -- 172.21.15.116:0/3751498842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 msgr2=0x7f78641976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869172640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f78641976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869172640 1 -- 172.21.15.116:0/3751498842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7854009560 con 0x7f7864101820 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f786a174640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78641021e0 0x7f786419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869973640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f78641976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f7869172640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864197be0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7854036040 tx=0x7f78540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78540044e0 con 0x7f7864101820 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786419f740 con 0x7f7864101820 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f785402de90 con 0x7f7864101820 2024-03-24T09:19:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7854005470 con 0x7f7864101820 2024-03-24T09:19:25.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.211+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786419fc50 con 0x7f7864101820 2024-03-24T09:19:25.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.212+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786410bef0 con 0x7f7864101820 2024-03-24T09:19:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.213+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f7854003800 con 0x7f7864101820 2024-03-24T09:19:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.213+0000 7f785affd640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f78500a2140 0x7f78500a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.214+0000 7f7869973640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f78500a2140 0x7f78500a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.214+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f78540efbd0 con 0x7f7864101820 2024-03-24T09:19:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.214+0000 7f7869973640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f78500a2140 0x7f78500a4600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7864101680 tx=0x7f786000a5e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.218+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78540b8fa0 con 0x7f7864101820 2024-03-24T09:19:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.436+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f786419fe70 con 0x7f7864101820 2024-03-24T09:19:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.437+0000 7f785affd640 1 -- 172.21.15.116:0/3751498842 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8016 (secure 0 0 0) 0x7f78540bc660 con 0x7f7864101820 2024-03-24T09:19:25.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:25.438 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T09:19:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 10 2024-03-24T09:19:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.439+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f78500a2140 msgr2=0x7f78500a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.439+0000 7f786bbd7640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f78500a2140 0x7f78500a4600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7864101680 tx=0x7f786000a5e0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.439+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 msgr2=0x7f7864197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.439+0000 7f786bbd7640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864197be0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7854036040 tx=0x7f78540042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 shutdown_connections 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f78500a2140 0x7f78500a4600 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78641021e0 0x7f786419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7864101820 0x7f7864197be0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 --2- 172.21.15.116:0/3751498842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7864100630 0x7f78641976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 >> 172.21.15.116:0/3751498842 conn(0x7f78640fbba0 msgr2=0x7f78640fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 shutdown_connections 2024-03-24T09:19:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:25.440+0000 7f786bbd7640 1 -- 172.21.15.116:0/3751498842 wait complete. 2024-03-24T09:19:26.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:25 smithi143 ceph-mon[35965]: Deploying daemon mds.g on smithi116 2024-03-24T09:19:26.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] up:boot 2024-03-24T09:19:26.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:25 smithi143 ceph-mon[35965]: fsmap 9 up:standby 2024-03-24T09:19:26.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:25 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:19:26.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:25 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3751498842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:26.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:25 smithi117 ceph-mon[35788]: Deploying daemon mds.g on smithi116 2024-03-24T09:19:26.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] up:boot 2024-03-24T09:19:26.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:25 smithi117 ceph-mon[35788]: fsmap 9 up:standby 2024-03-24T09:19:26.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:25 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:19:26.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:25 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3751498842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:26.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:25 smithi116 ceph-mon[32207]: Deploying daemon mds.g on smithi116 2024-03-24T09:19:26.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] up:boot 2024-03-24T09:19:26.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:25 smithi116 ceph-mon[32207]: fsmap 9 up:standby 2024-03-24T09:19:26.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:25 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:19:26.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:25 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3751498842' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:26.972 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:26 smithi116 ceph-mon[32207]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:26.972 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:26 smithi116 systemd[1]: Starting Ceph mds.g for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:27.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:26 smithi143 ceph-mon[35965]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:27.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:26 smithi117 ceph-mon[35788]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:27.248 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:26 smithi116 podman[73625]: 2024-03-24 09:19:26.873847386 +0000 UTC m=+0.017135949 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:27.248 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:26 smithi116 podman[73625]: 2024-03-24 09:19:26.987314437 +0000 UTC m=+0.130603031 container create fc2441da97ec17ab58abb893e5f322d533b42d7458d68d60b0d878200e2b33c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:19:27.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:27.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 -- 172.21.15.116:0/760528695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c101820 msgr2=0x7fe86c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:27.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 --2- 172.21.15.116:0/760528695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c101820 0x7fe86c101ca0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009900 tx=0x7fe85c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:27.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 -- 172.21.15.116:0/760528695 shutdown_connections 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 --2- 172.21.15.116:0/760528695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c1021e0 0x7fe86c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 --2- 172.21.15.116:0/760528695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c101820 0x7fe86c101ca0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 --2- 172.21.15.116:0/760528695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c100630 0x7fe86c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 -- 172.21.15.116:0/760528695 >> 172.21.15.116:0/760528695 conn(0x7fe86c0fbba0 msgr2=0x7fe86c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 -- 172.21.15.116:0/760528695 shutdown_connections 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.560+0000 7fe873be9640 1 -- 172.21.15.116:0/760528695 wait complete. 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 Processor -- start 2024-03-24T09:19:27.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 -- start start 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 0x7fe86c199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c101820 0x7fe86c199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c1021e0 0x7fe86c1a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c1a1450 con 0x7fe86c100630 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c1a15c0 con 0x7fe86c101820 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe873be9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c1a1780 con 0x7fe86c1021e0 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe871985640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 0x7fe86c199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe871985640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 0x7fe86c199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40034/0 (socket says 172.21.15.116:40034) 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe871985640 1 -- 172.21.15.116:0/3964290354 learned_addr learned my addr 172.21.15.116:0/3964290354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:27.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.561+0000 7fe872186640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c1021e0 0x7fe86c1a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871184640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c101820 0x7fe86c199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871985640 1 -- 172.21.15.116:0/3964290354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c1021e0 msgr2=0x7fe86c1a0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871985640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c1021e0 0x7fe86c1a0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871985640 1 -- 172.21.15.116:0/3964290354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c101820 msgr2=0x7fe86c199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871985640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c101820 0x7fe86c199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871985640 1 -- 172.21.15.116:0/3964290354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c009560 con 0x7fe86c100630 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871184640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c101820 0x7fe86c199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe871985640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 0x7fe86c199960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e9c0 tx=0x7fe86800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868009700 con 0x7fe86c100630 2024-03-24T09:19:27.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c1a1a60 con 0x7fe86c100630 2024-03-24T09:19:27.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8680044d0 con 0x7fe86c100630 2024-03-24T09:19:27.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868010430 con 0x7fe86c100630 2024-03-24T09:19:27.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.562+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c1a1f70 con 0x7fe86c100630 2024-03-24T09:19:27.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.563+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c10bef0 con 0x7fe86c100630 2024-03-24T09:19:27.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.564+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fe868003f30 con 0x7fe86c100630 2024-03-24T09:19:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.565+0000 7fe862ffd640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8580a20c0 0x7fe8580a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.565+0000 7fe871184640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8580a20c0 0x7fe8580a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.565+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe868014070 con 0x7fe86c100630 2024-03-24T09:19:27.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.565+0000 7fe871184640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8580a20c0 0x7fe8580a4580 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe85c036040 tx=0x7fe85c000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:27.570 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:27 smithi116 podman[73625]: 2024-03-24 09:19:27.248408628 +0000 UTC m=+0.391697180 container init fc2441da97ec17ab58abb893e5f322d533b42d7458d68d60b0d878200e2b33c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T09:19:27.570 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:27 smithi116 podman[73625]: 2024-03-24 09:19:27.251232435 +0000 UTC m=+0.394520989 container start fc2441da97ec17ab58abb893e5f322d533b42d7458d68d60b0d878200e2b33c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T09:19:27.570 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:27 smithi116 ceph-mds[73644]: -- 172.21.15.116:0/614309943 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564298da01a0 con 0x564298d76c00 2024-03-24T09:19:27.570 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g[73640]: starting mds.g at 2024-03-24T09:19:27.570 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:27 smithi116 bash[73625]: fc2441da97ec17ab58abb893e5f322d533b42d7458d68d60b0d878200e2b33c0 2024-03-24T09:19:27.570 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:19:27 smithi116 systemd[1]: Started Ceph mds.g for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:27.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.569+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe868097100 con 0x7fe86c100630 2024-03-24T09:19:27.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.807+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe86c10c080 con 0x7fe86c100630 2024-03-24T09:19:27.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.807+0000 7fe862ffd640 1 -- 172.21.15.116:0/3964290354 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8845 (secure 0 0 0) 0x7fe86809a7c0 con 0x7fe86c100630 2024-03-24T09:19:27.808 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:27.808 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T09:19:27.809 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-24T09:19:27.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8580a20c0 msgr2=0x7fe8580a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8580a20c0 0x7fe8580a4580 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe85c036040 tx=0x7fe85c000bc0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 msgr2=0x7fe86c199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 0x7fe86c199960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e9c0 tx=0x7fe86800ee80 comp rx=0 tx=0).stop 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 shutdown_connections 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8580a20c0 0x7fe8580a4580 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c1021e0 0x7fe86c1a0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c101820 0x7fe86c199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 --2- 172.21.15.116:0/3964290354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c100630 0x7fe86c199960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:27.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 >> 172.21.15.116:0/3964290354 conn(0x7fe86c0fbba0 msgr2=0x7fe86c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 shutdown_connections 2024-03-24T09:19:27.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:27.810+0000 7fe873be9640 1 -- 172.21.15.116:0/3964290354 wait complete. 2024-03-24T09:19:28.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:28.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: Deploying daemon mds.a on smithi116 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.116:6838/1877964932,v1:172.21.15.116:6839/1877964932] up:boot 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: fsmap 10 up:standby 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3964290354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:28.449 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:28 smithi116 ceph-mon[32207]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: Deploying daemon mds.a on smithi116 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.116:6838/1877964932,v1:172.21.15.116:6839/1877964932] up:boot 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: fsmap 10 up:standby 2024-03-24T09:19:28.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:19:28.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3964290354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:28.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:28 smithi143 ceph-mon[35965]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: Deploying daemon mds.a on smithi116 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.116:6838/1877964932,v1:172.21.15.116:6839/1877964932] up:boot 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: fsmap 10 up:standby 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3964290354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:28.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:28 smithi117 ceph-mon[35788]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:29.767 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:29 smithi116 systemd[1]: Starting Ceph mds.a for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:29.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.936+0000 7f15513fd640 1 -- 172.21.15.116:0/1131450776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c1021e0 msgr2=0x7f154c108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.936+0000 7f15513fd640 1 --2- 172.21.15.116:0/1131450776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c1021e0 0x7f154c108a70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1544009ca0 tx=0x7f154402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.936+0000 7f15513fd640 1 -- 172.21.15.116:0/1131450776 shutdown_connections 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.936+0000 7f15513fd640 1 --2- 172.21.15.116:0/1131450776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c1021e0 0x7f154c108a70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.936+0000 7f15513fd640 1 --2- 172.21.15.116:0/1131450776 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c101820 0x7f154c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.936+0000 7f15513fd640 1 --2- 172.21.15.116:0/1131450776 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f154c100630 0x7f154c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- 172.21.15.116:0/1131450776 >> 172.21.15.116:0/1131450776 conn(0x7f154c0fbba0 msgr2=0x7f154c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:29.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- 172.21.15.116:0/1131450776 shutdown_connections 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- 172.21.15.116:0/1131450776 wait complete. 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 Processor -- start 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- start start 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f154c100630 0x7f154c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 0x7f154c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c1021e0 0x7f154c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f154c19f180 con 0x7f154c101820 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f154c19f2f0 con 0x7f154c1021e0 2024-03-24T09:19:29.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f15513fd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f154c19f4b0 con 0x7f154c100630 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f1543fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 0x7f154c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f1543fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 0x7f154c197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40046/0 (socket says 172.21.15.116:40046) 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.937+0000 7f1543fff640 1 -- 172.21.15.116:0/1842676514 learned_addr learned my addr 172.21.15.116:0/1842676514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f154affd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f154c100630 0x7f154c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f154b7fe640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c1021e0 0x7f154c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1543fff640 1 -- 172.21.15.116:0/1842676514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f154c100630 msgr2=0x7f154c197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1543fff640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f154c100630 0x7f154c197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1543fff640 1 -- 172.21.15.116:0/1842676514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c1021e0 msgr2=0x7f154c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1543fff640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c1021e0 0x7f154c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1543fff640 1 -- 172.21.15.116:0/1842676514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1544009900 con 0x7f154c101820 2024-03-24T09:19:29.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f154b7fe640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c1021e0 0x7f154c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1543fff640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 0x7f154c197bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f153400d950 tx=0x7f153400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1534009c30 con 0x7f154c101820 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154c19f790 con 0x7f154c101820 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1534009d90 con 0x7f154c101820 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153400f6b0 con 0x7f154c101820 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.938+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f154c19fca0 con 0x7f154c101820 2024-03-24T09:19:29.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.939+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f154c10bef0 con 0x7f154c101820 2024-03-24T09:19:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.940+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f1534010aa0 con 0x7f154c101820 2024-03-24T09:19:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.940+0000 7f1548ff9640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f15380a20c0 0x7f15380a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.941+0000 7f154affd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f15380a20c0 0x7f15380a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.941+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f15340cd520 con 0x7f154c101820 2024-03-24T09:19:29.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.941+0000 7f154affd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f15380a20c0 0x7f15380a4580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f153c005fd0 tx=0x7f153c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:29.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:29.945+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1534096990 con 0x7f154c101820 2024-03-24T09:19:30.028 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:29 smithi116 podman[73986]: 2024-03-24 09:19:29.669586962 +0000 UTC m=+0.017394836 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:30.028 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:29 smithi116 podman[73986]: 2024-03-24 09:19:29.78337889 +0000 UTC m=+0.131186804 container create 5c1a4d77530075d640807738cc48e4e35a4ec4bd47889e8724a3f1d644d0c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T09:19:30.028 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:30 smithi116 podman[73986]: 2024-03-24 09:19:30.02833638 +0000 UTC m=+0.376144258 container init 5c1a4d77530075d640807738cc48e4e35a4ec4bd47889e8724a3f1d644d0c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T09:19:30.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.174+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f154c10c080 con 0x7f154c101820 2024-03-24T09:19:30.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.175+0000 7f1548ff9640 1 -- 172.21.15.116:0/1842676514 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8845 (secure 0 0 0) 0x7f15340b8e60 con 0x7f154c101820 2024-03-24T09:19:30.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:30.176 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T09:19:30.176 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-24T09:19:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f15380a20c0 msgr2=0x7f15380a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f15380a20c0 0x7f15380a4580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f153c005fd0 tx=0x7f153c00b390 comp rx=0 tx=0).stop 2024-03-24T09:19:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 msgr2=0x7f154c197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 0x7f154c197bd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f153400d950 tx=0x7f153400de10 comp rx=0 tx=0).stop 2024-03-24T09:19:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 shutdown_connections 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f154c1021e0 0x7f154c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f154c101820 0x7f154c197bd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f15380a20c0 0x7f15380a4580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 --2- 172.21.15.116:0/1842676514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f154c100630 0x7f154c197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 >> 172.21.15.116:0/1842676514 conn(0x7f154c0fbba0 msgr2=0x7f154c068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 shutdown_connections 2024-03-24T09:19:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:30.177+0000 7f15513fd640 1 -- 172.21.15.116:0/1842676514 wait complete. 2024-03-24T09:19:30.318 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:30 smithi116 podman[73986]: 2024-03-24 09:19:30.031505269 +0000 UTC m=+0.379313147 container start 5c1a4d77530075d640807738cc48e4e35a4ec4bd47889e8724a3f1d644d0c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T09:19:30.318 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mds[74044]: -- 172.21.15.116:0/2101052592 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ac9c7fa1a0 con 0x55ac9c7d0c00 2024-03-24T09:19:30.318 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a[74021]: starting mds.a at 2024-03-24T09:19:30.318 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:30 smithi116 bash[73986]: 5c1a4d77530075d640807738cc48e4e35a4ec4bd47889e8724a3f1d644d0c892 2024-03-24T09:19:30.318 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:19:30 smithi116 systemd[1]: Started Ceph mds.a for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1842676514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:31.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:30 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1842676514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:30 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1842676514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T09:19:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:30 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:32.190 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:32.264 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:31 smithi117 ceph-mon[35788]: Deploying daemon mds.b on smithi117 2024-03-24T09:19:32.265 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:31 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.116:6840/891659164,v1:172.21.15.116:6841/891659164] up:boot 2024-03-24T09:19:32.265 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:31 smithi117 ceph-mon[35788]: fsmap 11 up:standby 2024-03-24T09:19:32.265 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:31 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:19:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:31 smithi143 ceph-mon[35965]: Deploying daemon mds.b on smithi117 2024-03-24T09:19:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:31 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.116:6840/891659164,v1:172.21.15.116:6841/891659164] up:boot 2024-03-24T09:19:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:31 smithi143 ceph-mon[35965]: fsmap 11 up:standby 2024-03-24T09:19:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:31 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 -- 172.21.15.116:0/3223938291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 msgr2=0x7fe184100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/3223938291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe184100a50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe174009900 tx=0x7fe17402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 -- 172.21.15.116:0/3223938291 shutdown_connections 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/3223938291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1841021e0 0x7fe184108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/3223938291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe184101820 0x7fe184101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/3223938291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe184100a50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 -- 172.21.15.116:0/3223938291 >> 172.21.15.116:0/3223938291 conn(0x7fe1840fbba0 msgr2=0x7fe1840fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 -- 172.21.15.116:0/3223938291 shutdown_connections 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.304+0000 7fe18a1d7640 1 -- 172.21.15.116:0/3223938291 wait complete. 2024-03-24T09:19:32.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 Processor -- start 2024-03-24T09:19:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 -- start start 2024-03-24T09:19:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe1840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe184101820 0x7fe18419e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1841021e0 0x7fe1841a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1841a14a0 con 0x7fe184100630 2024-03-24T09:19:32.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe1841a1610 con 0x7fe184101820 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe18a1d7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe1841a17d0 con 0x7fe1841021e0 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe1840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe1840693e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40080/0 (socket says 172.21.15.116:40080) 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 -- 172.21.15.116:0/2234877486 learned_addr learned my addr 172.21.15.116:0/2234877486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe182ffd640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe184101820 0x7fe18419e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe183fff640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1841021e0 0x7fe1841a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 -- 172.21.15.116:0/2234877486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1841021e0 msgr2=0x7fe1841a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1841021e0 0x7fe1841a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 -- 172.21.15.116:0/2234877486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe184101820 msgr2=0x7fe18419e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe184101820 0x7fe18419e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe1837fe640 1 -- 172.21.15.116:0/2234877486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe174009560 con 0x7fe184100630 2024-03-24T09:19:32.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.305+0000 7fe182ffd640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe184101820 0x7fe18419e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.306+0000 7fe1837fe640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe1840693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe174000c00 tx=0x7fe1740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.306+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1740044e0 con 0x7fe184100630 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.306+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe17402de90 con 0x7fe184100630 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.306+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1841a19f0 con 0x7fe184100630 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.306+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe174005410 con 0x7fe184100630 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.306+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1841a1f00 con 0x7fe184100630 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.307+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1841079c0 con 0x7fe184100630 2024-03-24T09:19:32.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.308+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fe174005570 con 0x7fe184100630 2024-03-24T09:19:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.309+0000 7fe180ff9640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe16c0a20f0 0x7fe16c0a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.309+0000 7fe182ffd640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe16c0a20f0 0x7fe16c0a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.309+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe1740efb00 con 0x7fe184100630 2024-03-24T09:19:32.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.309+0000 7fe182ffd640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe16c0a20f0 0x7fe16c0a45b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1780058c0 tx=0x7fe178005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:32.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.313+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe1740b8f70 con 0x7fe184100630 2024-03-24T09:19:32.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:31 smithi116 ceph-mon[32207]: Deploying daemon mds.b on smithi117 2024-03-24T09:19:32.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:31 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.116:6840/891659164,v1:172.21.15.116:6841/891659164] up:boot 2024-03-24T09:19:32.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:31 smithi116 ceph-mon[32207]: fsmap 11 up:standby 2024-03-24T09:19:32.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:31 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:19:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.523+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe18410c140 con 0x7fe184100630 2024-03-24T09:19:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.524+0000 7fe180ff9640 1 -- 172.21.15.116:0/2234877486 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9671 (secure 0 0 0) 0x7fe1740bc630 con 0x7fe184100630 2024-03-24T09:19:32.524 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:32.525 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-24T09:19:32.525 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 12 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe16c0a20f0 msgr2=0x7fe16c0a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe16c0a20f0 0x7fe16c0a45b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1780058c0 tx=0x7fe178005850 comp rx=0 tx=0).stop 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 msgr2=0x7fe1840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe1840693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe174000c00 tx=0x7fe1740042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 shutdown_connections 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe16c0a20f0 0x7fe16c0a45b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1841021e0 0x7fe1841a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe184101820 0x7fe18419e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 --2- 172.21.15.116:0/2234877486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe184100630 0x7fe1840693e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 >> 172.21.15.116:0/2234877486 conn(0x7fe1840fbba0 msgr2=0x7fe184104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 shutdown_connections 2024-03-24T09:19:32.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:32.526+0000 7fe18a1d7640 1 -- 172.21.15.116:0/2234877486 wait complete. 2024-03-24T09:19:32.566 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 systemd[1]: Starting Ceph mds.b for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:19:32.930 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 podman[65631]: 2024-03-24 09:19:32.562577933 +0000 UTC m=+0.017732604 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:19:32.930 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 podman[65631]: 2024-03-24 09:19:32.685798494 +0000 UTC m=+0.140953161 container create 551dc8fe39be56da4b15f6ccdc4b410a0da7744a00c94cf826e8ce7f8b629a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:19:32.930 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 podman[65631]: 2024-03-24 09:19:32.930214731 +0000 UTC m=+0.385369400 container init 551dc8fe39be56da4b15f6ccdc4b410a0da7744a00c94cf826e8ce7f8b629a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:19:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:32 smithi143 ceph-mon[35965]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:32 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2234877486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:32 smithi117 ceph-mon[35788]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:32 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2234877486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:33.316 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 podman[65631]: 2024-03-24 09:19:32.933042773 +0000 UTC m=+0.388197458 container start 551dc8fe39be56da4b15f6ccdc4b410a0da7744a00c94cf826e8ce7f8b629a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T09:19:33.316 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 ceph-mds[65651]: -- 172.21.15.117:0/3278713916 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563786d281a0 con 0x563786cff000 2024-03-24T09:19:33.317 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: starting mds.b at 2024-03-24T09:19:33.317 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:33 smithi117 bash[65631]: 551dc8fe39be56da4b15f6ccdc4b410a0da7744a00c94cf826e8ce7f8b629a2f 2024-03-24T09:19:33.317 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:33 smithi117 systemd[1]: Started Ceph mds.b for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:19:33.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:32 smithi116 ceph-mon[32207]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:33.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:32 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2234877486' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] up:boot 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: fsmap 12 up:standby 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:19:34.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:33 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] up:boot 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: fsmap 12 up:standby 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:19:34.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:33 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] up:boot 2024-03-24T09:19:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: fsmap 12 up:standby 2024-03-24T09:19:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:19:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:19:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:33 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:34.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:34.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 -- 172.21.15.116:0/3883649806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc1021e0 msgr2=0x7f3fbc108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:34.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 --2- 172.21.15.116:0/3883649806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc108a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4009ca0 tx=0x7f3fb402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 -- 172.21.15.116:0/3883649806 shutdown_connections 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 --2- 172.21.15.116:0/3883649806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc108a70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 --2- 172.21.15.116:0/3883649806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc101820 0x7f3fbc101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 --2- 172.21.15.116:0/3883649806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fbc100630 0x7f3fbc100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 -- 172.21.15.116:0/3883649806 >> 172.21.15.116:0/3883649806 conn(0x7f3fbc0fbbc0 msgr2=0x7f3fbc0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 -- 172.21.15.116:0/3883649806 shutdown_connections 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.648+0000 7f3fc324d640 1 -- 172.21.15.116:0/3883649806 wait complete. 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 Processor -- start 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 -- start start 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 0x7f3fbc197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fbc101820 0x7f3fbc197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:34.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc19f200 con 0x7f3fbc100630 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc19f370 con 0x7f3fbc1021e0 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc324d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc19f530 con 0x7f3fbc101820 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc0fe9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 0x7f3fbc197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc0fe9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 0x7f3fbc197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40112/0 (socket says 172.21.15.116:40112) 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fc0fe9640 1 -- 172.21.15.116:0/1430376654 learned_addr learned my addr 172.21.15.116:0/1430376654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.649+0000 7f3fbbfff640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fbc101820 0x7f3fbc197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:34.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc0fe9640 1 -- 172.21.15.116:0/1430376654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fbc101820 msgr2=0x7f3fbc197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc17ea640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc0fe9640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fbc101820 0x7f3fbc197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc0fe9640 1 -- 172.21.15.116:0/1430376654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc1021e0 msgr2=0x7f3fbc19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc0fe9640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc0fe9640 1 -- 172.21.15.116:0/1430376654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb4009900 con 0x7f3fbc100630 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc17ea640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc0fe9640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 0x7f3fbc197710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00e9c0 tx=0x7f3fac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac0096e0 con 0x7f3fbc100630 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fbc19f810 con 0x7f3fbc100630 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fac0044e0 con 0x7f3fbc100630 2024-03-24T09:19:34.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fac010430 con 0x7f3fbc100630 2024-03-24T09:19:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.650+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fbc19fcd0 con 0x7f3fbc100630 2024-03-24T09:19:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.651+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fbc10bef0 con 0x7f3fbc100630 2024-03-24T09:19:34.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.652+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f3fac003f80 con 0x7f3fbc100630 2024-03-24T09:19:34.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.653+0000 7f3fb9ffb640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3fa40a20c0 0x7f3fa40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:34.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.653+0000 7f3fbbfff640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3fa40a20c0 0x7f3fa40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:34.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.653+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3fac014070 con 0x7f3fbc100630 2024-03-24T09:19:34.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.653+0000 7f3fbbfff640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3fa40a20c0 0x7f3fa40a4580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc198c20 tx=0x7f3fb000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.657+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3fac0970a0 con 0x7f3fbc100630 2024-03-24T09:19:34.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.926+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fbc10c080 con 0x7f3fbc100630 2024-03-24T09:19:34.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.926+0000 7f3fb9ffb640 1 -- 172.21.15.116:0/1430376654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f3fac09a760 con 0x7f3fbc100630 2024-03-24T09:19:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:34.927 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-24T09:19:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 13 2024-03-24T09:19:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3fa40a20c0 msgr2=0x7f3fa40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3fa40a20c0 0x7f3fa40a4580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc198c20 tx=0x7f3fb000b390 comp rx=0 tx=0).stop 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 msgr2=0x7f3fbc197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 0x7f3fbc197710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3fac00e9c0 tx=0x7f3fac00ee80 comp rx=0 tx=0).stop 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 shutdown_connections 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fbc1021e0 0x7f3fbc19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3fa40a20c0 0x7f3fa40a4580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fbc101820 0x7f3fbc197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 --2- 172.21.15.116:0/1430376654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100630 0x7f3fbc197710 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.928+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 >> 172.21.15.116:0/1430376654 conn(0x7f3fbc0fbbc0 msgr2=0x7f3fbc069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.929+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 shutdown_connections 2024-03-24T09:19:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:34.929+0000 7f3fc324d640 1 -- 172.21.15.116:0/1430376654 wait complete. 2024-03-24T09:19:34.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 -- 172.21.15.116:0/694279303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 msgr2=0x7f9118108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 --2- 172.21.15.116:0/694279303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 0x7f9118108a70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9110009be0 tx=0x7f911002bef0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 -- 172.21.15.116:0/694279303 shutdown_connections 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 --2- 172.21.15.116:0/694279303 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 0x7f9118108a70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 --2- 172.21.15.116:0/694279303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118101820 0x7f9118101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 --2- 172.21.15.116:0/694279303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118100630 0x7f9118100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 -- 172.21.15.116:0/694279303 >> 172.21.15.116:0/694279303 conn(0x7f91180fbba0 msgr2=0x7f91180fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 -- 172.21.15.116:0/694279303 shutdown_connections 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.048+0000 7f911d08e640 1 -- 172.21.15.116:0/694279303 wait complete. 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 Processor -- start 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 -- start start 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 0x7f91180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118101820 0x7f911819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 0x7f911819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f911819f1e0 con 0x7f9118100630 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f911819f350 con 0x7f9118101820 2024-03-24T09:19:35.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911d08e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f911819f510 con 0x7f91181021e0 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f9116d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 0x7f91180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f9116d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 0x7f91180693e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40134/0 (socket says 172.21.15.116:40134) 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f9116d9d640 1 -- 172.21.15.116:0/4086717579 learned_addr learned my addr 172.21.15.116:0/4086717579 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911659c640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118101820 0x7f911819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.049+0000 7f911759e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 0x7f911819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f9116d9d640 1 -- 172.21.15.116:0/4086717579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 msgr2=0x7f911819ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f9116d9d640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 0x7f911819ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f9116d9d640 1 -- 172.21.15.116:0/4086717579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118101820 msgr2=0x7f911819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f9116d9d640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118101820 0x7f911819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f9116d9d640 1 -- 172.21.15.116:0/4086717579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f910c009560 con 0x7f9118100630 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f9116d9d640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 0x7f91180693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f910800e9c0 tx=0x7f910800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91080096e0 con 0x7f9118100630 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9110009840 con 0x7f9118100630 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f91080044e0 con 0x7f9118100630 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9108010430 con 0x7f9118100630 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.050+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911819fb10 con 0x7f9118100630 2024-03-24T09:19:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.051+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91181079a0 con 0x7f9118100630 2024-03-24T09:19:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.053+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f9108003f80 con 0x7f9118100630 2024-03-24T09:19:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.053+0000 7f90fffff640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f91000a20c0 0x7f91000a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.054+0000 7f911659c640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f91000a20c0 0x7f91000a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.054+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9108014070 con 0x7f9118100630 2024-03-24T09:19:35.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.054+0000 7f911659c640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f91000a20c0 0x7f91000a4580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9118069150 tx=0x7f910c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:35.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.058+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9108097090 con 0x7f9118100630 2024-03-24T09:19:35.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.243+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f911810c0c0 con 0x7f9118100630 2024-03-24T09:19:35.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.244+0000 7f90fffff640 1 -- 172.21.15.116:0/4086717579 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18525 (secure 0 0 0) 0x7f910809a750 con 0x7f9118100630 2024-03-24T09:19:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:35.246 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:18:49.985097+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:19:35.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f91000a20c0 msgr2=0x7f91000a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f91000a20c0 0x7f91000a4580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9118069150 tx=0x7f910c009260 comp rx=0 tx=0).stop 2024-03-24T09:19:35.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 msgr2=0x7f91180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 0x7f91180693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f910800e9c0 tx=0x7f910800ee80 comp rx=0 tx=0).stop 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 shutdown_connections 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f91000a20c0 0x7f91000a4580 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91181021e0 0x7f911819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9118101820 0x7f911819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 --2- 172.21.15.116:0/4086717579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9118100630 0x7f91180693e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.246+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 >> 172.21.15.116:0/4086717579 conn(0x7f91180fbba0 msgr2=0x7f9118104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.247+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 shutdown_connections 2024-03-24T09:19:35.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.247+0000 7f911d08e640 1 -- 172.21.15.116:0/4086717579 wait complete. 2024-03-24T09:19:35.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:34 smithi116 ceph-mon[32207]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:35.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:34 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1430376654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:35.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T09:19:35.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:19:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:34 smithi143 ceph-mon[35965]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:34 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1430376654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:34 smithi117 ceph-mon[35788]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:34 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1430376654' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 -- 172.21.15.116:0/1572464225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44100840 msgr2=0x7faa44100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 --2- 172.21.15.116:0/1572464225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44100840 0x7faa44100cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faa30009900 tx=0x7faa3002bc80 comp rx=0 tx=0).stop 2024-03-24T09:19:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 -- 172.21.15.116:0/1572464225 shutdown_connections 2024-03-24T09:19:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 --2- 172.21.15.116:0/1572464225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa44101200 0x7faa44108aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 --2- 172.21.15.116:0/1572464225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44100840 0x7faa44100cc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 --2- 172.21.15.116:0/1572464225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa44101c80 0x7faa441020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 -- 172.21.15.116:0/1572464225 >> 172.21.15.116:0/1572464225 conn(0x7faa440fb2f0 msgr2=0x7faa440fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 -- 172.21.15.116:0/1572464225 shutdown_connections 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.370+0000 7faa4ad19640 1 -- 172.21.15.116:0/1572464225 wait complete. 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 Processor -- start 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 -- start start 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa44100840 0x7faa44196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 0x7faa44196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa44101c80 0x7faa4419dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa4419e100 con 0x7faa44101200 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faa4419e270 con 0x7faa44100840 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa4ad19640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7faa4419e430 con 0x7faa44101c80 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.371+0000 7faa43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 0x7faa44196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 0x7faa44196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40146/0 (socket says 172.21.15.116:40146) 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 -- 172.21.15.116:0/3946336838 learned_addr learned my addr 172.21.15.116:0/3946336838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa48ab5640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa44100840 0x7faa44196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa492b6640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa44101c80 0x7faa4419dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 -- 172.21.15.116:0/3946336838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa44101c80 msgr2=0x7faa4419dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa44101c80 0x7faa4419dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 -- 172.21.15.116:0/3946336838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa44100840 msgr2=0x7faa44196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa44100840 0x7faa44196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 -- 172.21.15.116:0/3946336838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa30009560 con 0x7faa44101200 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa43fff640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 0x7faa44196b50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa30000c00 tx=0x7faa300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300044e0 con 0x7faa44101200 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa30003cb0 con 0x7faa44101200 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30005660 con 0x7faa44101200 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4419e6b0 con 0x7faa44101200 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.372+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4419ebc0 con 0x7faa44101200 2024-03-24T09:19:35.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.373+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa44107a50 con 0x7faa44101200 2024-03-24T09:19:35.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.375+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7faa30004dc0 con 0x7faa44101200 2024-03-24T09:19:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.375+0000 7faa41ffb640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faa340a2140 0x7faa340a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.375+0000 7faa48ab5640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faa340a2140 0x7faa340a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.375+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faa300f0260 con 0x7faa44101200 2024-03-24T09:19:35.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.376+0000 7faa48ab5640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faa340a2140 0x7faa340a4600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa3800bb60 tx=0x7faa3800b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:35.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.383+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faa300b96d0 con 0x7faa44101200 2024-03-24T09:19:35.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.584+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa4402d130 con 0x7faa44101200 2024-03-24T09:19:35.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.585+0000 7faa41ffb640 1 -- 172.21.15.116:0/3946336838 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7faa300bcd90 con 0x7faa44101200 2024-03-24T09:19:35.585 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:19:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faa340a2140 msgr2=0x7faa340a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faa340a2140 0x7faa340a4600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faa3800bb60 tx=0x7faa3800b390 comp rx=0 tx=0).stop 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 msgr2=0x7faa44196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 0x7faa44196b50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faa30000c00 tx=0x7faa300042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 shutdown_connections 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7faa340a2140 0x7faa340a4600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa44101c80 0x7faa4419dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa44101200 0x7faa44196b50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 --2- 172.21.15.116:0/3946336838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa44100840 0x7faa44196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 >> 172.21.15.116:0/3946336838 conn(0x7faa440fb2f0 msgr2=0x7faa44104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 shutdown_connections 2024-03-24T09:19:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.587+0000 7faa4ad19640 1 -- 172.21.15.116:0/3946336838 wait complete. 2024-03-24T09:19:35.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T09:19:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 -- 172.21.15.116:0/2720898380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9081021b0 msgr2=0x7ff908108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/2720898380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9081021b0 0x7ff908108a40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff900009ca0 tx=0x7ff90002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 -- 172.21.15.116:0/2720898380 shutdown_connections 2024-03-24T09:19:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/2720898380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9081021b0 0x7ff908108a40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/2720898380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9081017f0 0x7ff908101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/2720898380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff908100600 0x7ff908100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.701+0000 7ff90dfd0640 1 -- 172.21.15.116:0/2720898380 >> 172.21.15.116:0/2720898380 conn(0x7ff9080fbb70 msgr2=0x7ff9080fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 -- 172.21.15.116:0/2720898380 shutdown_connections 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 -- 172.21.15.116:0/2720898380 wait complete. 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 Processor -- start 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 -- start start 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 0x7ff908197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9081017f0 0x7ff908197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9081021b0 0x7ff90819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff90819f180 con 0x7ff908100600 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff90819f2f0 con 0x7ff9081021b0 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff90dfd0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff90819f4b0 con 0x7ff9081017f0 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff9077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 0x7ff908197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff9077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 0x7ff908197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40152/0 (socket says 172.21.15.116:40152) 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.702+0000 7ff9077fe640 1 -- 172.21.15.116:0/3140643220 learned_addr learned my addr 172.21.15.116:0/3140643220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff907fff640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9081021b0 0x7ff90819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff8feffd640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9081017f0 0x7ff908197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9077fe640 1 -- 172.21.15.116:0/3140643220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9081017f0 msgr2=0x7ff908197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9077fe640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9081017f0 0x7ff908197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9077fe640 1 -- 172.21.15.116:0/3140643220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9081021b0 msgr2=0x7ff90819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9077fe640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9081021b0 0x7ff90819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9077fe640 1 -- 172.21.15.116:0/3140643220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8f8009490 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9077fe640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 0x7ff908197700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800e9c0 tx=0x7ff8f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f800cde0 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8f80108e0 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80188e0 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff900009900 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.703+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff90819fb10 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.704+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff90810bec0 con 0x7ff908100600 2024-03-24T09:19:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.705+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7ff8f8010a50 con 0x7ff908100600 2024-03-24T09:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.705+0000 7ff9057fa640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff8f40a20c0 0x7ff8f40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.706+0000 7ff8feffd640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff8f40a20c0 0x7ff8f40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.706+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff8f8014070 con 0x7ff908100600 2024-03-24T09:19:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.706+0000 7ff8feffd640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff8f40a20c0 0x7ff8f40a4580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff908198ba0 tx=0x7ff8f0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:35.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.710+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff8f80973e0 con 0x7ff908100600 2024-03-24T09:19:35.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.893+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff90810c0a0 con 0x7ff908100600 2024-03-24T09:19:35.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.893+0000 7ff9057fa640 1 -- 172.21.15.116:0/3140643220 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7ff8f809aaa0 con 0x7ff908100600 2024-03-24T09:19:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:35.894 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-03-24T09:19:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:19:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-03-24T09:19:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-03-24T09:19:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:19:35.895 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-03-24T09:19:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.895+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff8f40a20c0 msgr2=0x7ff8f40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.895+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff8f40a20c0 0x7ff8f40a4580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff908198ba0 tx=0x7ff8f0005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.895+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 msgr2=0x7ff908197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.895+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 0x7ff908197700 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800e9c0 tx=0x7ff8f800ee80 comp rx=0 tx=0).stop 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 shutdown_connections 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9081021b0 0x7ff90819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff8f40a20c0 0x7ff8f40a4580 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff9081017f0 0x7ff908197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 --2- 172.21.15.116:0/3140643220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff908100600 0x7ff908197700 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 >> 172.21.15.116:0/3140643220 conn(0x7ff9080fbb70 msgr2=0x7ff9080fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 shutdown_connections 2024-03-24T09:19:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:35.896+0000 7ff90dfd0640 1 -- 172.21.15.116:0/3140643220 wait complete. 2024-03-24T09:19:35.907 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T09:19:35.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T09:19:36.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- 172.21.15.116:0/111428616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 msgr2=0x7fe2941045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:36.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- 172.21.15.116:0/111428616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941045e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe290009900 tx=0x7fe29002bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- 172.21.15.116:0/111428616 shutdown_connections 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- 172.21.15.116:0/111428616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 0x7fe294109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- 172.21.15.116:0/111428616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 0x7fe294106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- 172.21.15.116:0/111428616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941045e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- 172.21.15.116:0/111428616 >> 172.21.15.116:0/111428616 conn(0x7fe2940fbba0 msgr2=0x7fe2940fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- 172.21.15.116:0/111428616 shutdown_connections 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- 172.21.15.116:0/111428616 wait complete. 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 Processor -- start 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- start start 2024-03-24T09:19:36.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 0x7fe294197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 0x7fe29419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe29419f2b0 con 0x7fe2941021f0 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe29419f420 con 0x7fe294107450 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.016+0000 7fe29be2e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe29419f5e0 con 0x7fe294104b20 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40180/0 (socket says 172.21.15.116:40180) 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 -- 172.21.15.116:0/3829427194 learned_addr learned my addr 172.21.15.116:0/3829427194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:36.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe2993c9640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 0x7fe294197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe29a3cb640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 0x7fe29419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 -- 172.21.15.116:0/3829427194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 msgr2=0x7fe294197d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 0x7fe294197d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 -- 172.21.15.116:0/3829427194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 msgr2=0x7fe29419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 0x7fe29419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 -- 172.21.15.116:0/3829427194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe290009560 con 0x7fe2941021f0 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe2993c9640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 0x7fe294197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe29a3cb640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 0x7fe29419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe299bca640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941977c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe290036040 tx=0x7fe2900042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2900044e0 con 0x7fe2941021f0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29419f860 con 0x7fe2941021f0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe29002de90 con 0x7fe2941021f0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe290005470 con 0x7fe2941021f0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.017+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29419fd70 con 0x7fe2941021f0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.018+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2940fd690 con 0x7fe2941021f0 2024-03-24T09:19:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.019+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fe290003800 con 0x7fe2941021f0 2024-03-24T09:19:36.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.020+0000 7fe28affd640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe2800a20c0 0x7fe2800a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:36.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.020+0000 7fe2993c9640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe2800a20c0 0x7fe2800a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:36.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.020+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe2900efb30 con 0x7fe2941021f0 2024-03-24T09:19:36.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.020+0000 7fe2993c9640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe2800a20c0 0x7fe2800a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2840058c0 tx=0x7fe284005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:36.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.024+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe2900b8fa0 con 0x7fe2941021f0 2024-03-24T09:19:36.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:36.207+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe29410d220 con 0x7fe2941021f0 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4086717579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3946336838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:19:36.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3140643220' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T09:19:36.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:36 smithi116 ceph-mon[32207]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4086717579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3946336838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3140643220' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T09:19:36.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:36 smithi143 ceph-mon[35965]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:36.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4086717579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3946336838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3140643220' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T09:19:36.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:36 smithi117 ceph-mon[35788]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.079+0000 7fe28affd640 1 -- 172.21.15.116:0/3829427194 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fe2900bc660 con 0x7fe2941021f0 2024-03-24T09:19:37.080 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2024-03-24T09:19:37.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe2800a20c0 msgr2=0x7fe2800a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:37.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe2800a20c0 0x7fe2800a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2840058c0 tx=0x7fe284005850 comp rx=0 tx=0).stop 2024-03-24T09:19:37.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 msgr2=0x7fe2941977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941977c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe290036040 tx=0x7fe2900042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 shutdown_connections 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe294107450 0x7fe29419ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe2800a20c0 0x7fe2800a4580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe294104b20 0x7fe294197d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 --2- 172.21.15.116:0/3829427194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2941021f0 0x7fe2941977c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 >> 172.21.15.116:0/3829427194 conn(0x7fe2940fbba0 msgr2=0x7fe294102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 shutdown_connections 2024-03-24T09:19:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.081+0000 7fe29be2e640 1 -- 172.21.15.116:0/3829427194 wait complete. 2024-03-24T09:19:37.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 -- 172.21.15.116:0/1642167793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54102200 msgr2=0x7fac541045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 --2- 172.21.15.116:0/1642167793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54102200 0x7fac541045f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fac44009900 tx=0x7fac4402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 -- 172.21.15.116:0/1642167793 shutdown_connections 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 --2- 172.21.15.116:0/1642167793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac54107460 0x7fac54109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 --2- 172.21.15.116:0/1642167793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac54104b30 0x7fac54106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 --2- 172.21.15.116:0/1642167793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54102200 0x7fac541045f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 -- 172.21.15.116:0/1642167793 >> 172.21.15.116:0/1642167793 conn(0x7fac540fbbb0 msgr2=0x7fac540fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 -- 172.21.15.116:0/1642167793 shutdown_connections 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.215+0000 7fac5928f640 1 -- 172.21.15.116:0/1642167793 wait complete. 2024-03-24T09:19:37.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 Processor -- start 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 -- start start 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac54102200 0x7fac54101b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac54104b30 0x7fac541001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 0x7fac541006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fac54100e50 con 0x7fac54107460 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac54100fc0 con 0x7fac54102200 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac5928f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fac541a1820 con 0x7fac54104b30 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 0x7fac541006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 0x7fac541006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41004/0 (socket says 172.21.15.116:41004) 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 -- 172.21.15.116:0/1181212686 learned_addr learned my addr 172.21.15.116:0/1181212686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac527fc640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac54104b30 0x7fac541001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac52ffd640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac54102200 0x7fac54101b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 -- 172.21.15.116:0/1181212686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac54104b30 msgr2=0x7fac541001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac54104b30 0x7fac541001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 -- 172.21.15.116:0/1181212686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac54102200 msgr2=0x7fac54101b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac54102200 0x7fac54101b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.216+0000 7fac537fe640 1 -- 172.21.15.116:0/1181212686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac44009560 con 0x7fac54107460 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.217+0000 7fac537fe640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 0x7fac541006f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00db20 tx=0x7fac4c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.217+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00bea0 con 0x7fac54107460 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.217+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac541a1ad0 con 0x7fac54107460 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.217+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac4c009940 con 0x7fac54107460 2024-03-24T09:19:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.217+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0106c0 con 0x7fac54107460 2024-03-24T09:19:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.217+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac541a1fe0 con 0x7fac54107460 2024-03-24T09:19:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.218+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac540fd6a0 con 0x7fac54107460 2024-03-24T09:19:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.219+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fac4c003970 con 0x7fac54107460 2024-03-24T09:19:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.219+0000 7fac3bfff640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac3c0a20c0 0x7fac3c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.220+0000 7fac52ffd640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac3c0a20c0 0x7fac3c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.220+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fac4c0cd5a0 con 0x7fac54107460 2024-03-24T09:19:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.220+0000 7fac52ffd640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac3c0a20c0 0x7fac3c0a4580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fac44009230 tx=0x7fac44005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.224+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fac4c096950 con 0x7fac54107460 2024-03-24T09:19:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:37.439+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fac5402d130 con 0x7fac54107460 2024-03-24T09:19:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:37 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3829427194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T09:19:37.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:37 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3829427194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T09:19:37.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:37 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3829427194' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T09:19:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.081+0000 7fac3bfff640 1 -- 172.21.15.116:0/1181212686 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fac4c09a010 con 0x7fac54107460 2024-03-24T09:19:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2024-03-24T09:19:38.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.083+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac3c0a20c0 msgr2=0x7fac3c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:38.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac3c0a20c0 0x7fac3c0a4580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fac44009230 tx=0x7fac44005fd0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 msgr2=0x7fac541006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 0x7fac541006f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00db20 tx=0x7fac4c00b260 comp rx=0 tx=0).stop 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 shutdown_connections 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac54107460 0x7fac541006f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac3c0a20c0 0x7fac3c0a4580 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac54104b30 0x7fac541001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 --2- 172.21.15.116:0/1181212686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac54102200 0x7fac54101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 >> 172.21.15.116:0/1181212686 conn(0x7fac540fbbb0 msgr2=0x7fac54102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 shutdown_connections 2024-03-24T09:19:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.084+0000 7fac5928f640 1 -- 172.21.15.116:0/1181212686 wait complete. 2024-03-24T09:19:38.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T09:19:38.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 -- 172.21.15.116:0/3070014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c100610 msgr2=0x7fd67c100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 --2- 172.21.15.116:0/3070014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c100610 0x7fd67c100a30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd674009900 tx=0x7fd67402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 -- 172.21.15.116:0/3070014174 shutdown_connections 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 --2- 172.21.15.116:0/3070014174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c1021c0 0x7fd67c108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 --2- 172.21.15.116:0/3070014174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd67c101800 0x7fd67c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 --2- 172.21.15.116:0/3070014174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c100610 0x7fd67c100a30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 -- 172.21.15.116:0/3070014174 >> 172.21.15.116:0/3070014174 conn(0x7fd67c0fbba0 msgr2=0x7fd67c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 -- 172.21.15.116:0/3070014174 shutdown_connections 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.208+0000 7fd6812a4640 1 -- 172.21.15.116:0/3070014174 wait complete. 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 Processor -- start 2024-03-24T09:19:38.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 -- start start 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c100610 0x7fd67c197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 0x7fd67c197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd67c1021c0 0x7fd67c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c19f250 con 0x7fd67c101800 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c19f3c0 con 0x7fd67c100610 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd6812a4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd67c19f580 con 0x7fd67c1021c0 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 0x7fd67c197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 0x7fd67c197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41026/0 (socket says 172.21.15.116:41026) 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 -- 172.21.15.116:0/93944311 learned_addr learned my addr 172.21.15.116:0/93944311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67affd640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c100610 0x7fd67c197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:38.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 -- 172.21.15.116:0/93944311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd67c1021c0 msgr2=0x7fd67c19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67b7fe640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd67c1021c0 0x7fd67c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd67c1021c0 0x7fd67c19ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 -- 172.21.15.116:0/93944311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c100610 msgr2=0x7fd67c197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c100610 0x7fd67c197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67a7fc640 1 -- 172.21.15.116:0/93944311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd674009560 con 0x7fd67c101800 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.209+0000 7fd67affd640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c100610 0x7fd67c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.210+0000 7fd67a7fc640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 0x7fd67c197ca0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd67000eb90 tx=0x7fd67000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.210+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700099b0 con 0x7fd67c101800 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.210+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd67000ee90 con 0x7fd67c101800 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.210+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67c19f860 con 0x7fd67c101800 2024-03-24T09:19:38.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.210+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670010690 con 0x7fd67c101800 2024-03-24T09:19:38.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.210+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67c19fd20 con 0x7fd67c101800 2024-03-24T09:19:38.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.211+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67c10bed0 con 0x7fd67c101800 2024-03-24T09:19:38.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.212+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fd670005000 con 0x7fd67c101800 2024-03-24T09:19:38.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.212+0000 7fd663fff640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd6640a20c0 0x7fd6640a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:38.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.212+0000 7fd67affd640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd6640a20c0 0x7fd6640a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:38.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.212+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7fd670014070 con 0x7fd67c101800 2024-03-24T09:19:38.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.213+0000 7fd67affd640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd6640a20c0 0x7fd6640a4580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd674006010 tx=0x7fd674005c00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:38.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.216+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd670097900 con 0x7fd67c101800 2024-03-24T09:19:38.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:38.429+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd67c10c060 con 0x7fd67c101800 2024-03-24T09:19:38.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3829427194' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1181212686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:19:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:38 smithi143 ceph-mon[35965]: pgmap v284: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:38.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3829427194' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T09:19:38.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1181212686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:38 smithi117 ceph-mon[35788]: pgmap v284: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3829427194' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T09:19:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T09:19:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1181212686' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-24T09:19:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:19:38.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:38 smithi116 ceph-mon[32207]: pgmap v284: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:19:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.096+0000 7fd663fff640 1 -- 172.21.15.116:0/93944311 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fd67009afc0 con 0x7fd67c101800 2024-03-24T09:19:39.097 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd6640a20c0 msgr2=0x7fd6640a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd6640a20c0 0x7fd6640a4580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd674006010 tx=0x7fd674005c00 comp rx=0 tx=0).stop 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 msgr2=0x7fd67c197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 0x7fd67c197ca0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd67000eb90 tx=0x7fd67000c6a0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 shutdown_connections 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd6640a20c0 0x7fd6640a4580 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd67c1021c0 0x7fd67c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd67c101800 0x7fd67c197ca0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 --2- 172.21.15.116:0/93944311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd67c100610 0x7fd67c197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 >> 172.21.15.116:0/93944311 conn(0x7fd67c0fbba0 msgr2=0x7fd67c0688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 shutdown_connections 2024-03-24T09:19:39.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.099+0000 7fd6812a4640 1 -- 172.21.15.116:0/93944311 wait complete. 2024-03-24T09:19:39.115 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-24T09:19:39.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-24T09:19:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.222+0000 7f6427055640 1 -- 172.21.15.116:0/2166222531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 msgr2=0x7f6420100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.222+0000 7f6427055640 1 --2- 172.21.15.116:0/2166222531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f6420100a30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6410009900 tx=0x7f641002bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- 172.21.15.116:0/2166222531 shutdown_connections 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 --2- 172.21.15.116:0/2166222531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64201021c0 0x7f6420108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 --2- 172.21.15.116:0/2166222531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 0x7f6420101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 --2- 172.21.15.116:0/2166222531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f6420100a30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- 172.21.15.116:0/2166222531 >> 172.21.15.116:0/2166222531 conn(0x7f64200fbb80 msgr2=0x7f64200fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- 172.21.15.116:0/2166222531 shutdown_connections 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- 172.21.15.116:0/2166222531 wait complete. 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 Processor -- start 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- start start 2024-03-24T09:19:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f64201976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 0x7f6420197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64201021c0 0x7f642019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f642019f190 con 0x7f6420100610 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f642019f300 con 0x7f6420101800 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6427055640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f642019f4c0 con 0x7f64201021c0 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6424df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f64201976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6424df1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f64201976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41050/0 (socket says 172.21.15.116:41050) 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.223+0000 7f6424df1640 1 -- 172.21.15.116:0/2610436856 learned_addr learned my addr 172.21.15.116:0/2610436856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f64255f2640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64201021c0 0x7f642019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f641ffff640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 0x7f6420197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6424df1640 1 -- 172.21.15.116:0/2610436856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64201021c0 msgr2=0x7f642019ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6424df1640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64201021c0 0x7f642019ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6424df1640 1 -- 172.21.15.116:0/2610436856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 msgr2=0x7f6420197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6424df1640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 0x7f6420197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6424df1640 1 -- 172.21.15.116:0/2610436856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6410009560 con 0x7f6420100610 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f641ffff640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 0x7f6420197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6424df1640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f64201976f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6410000c00 tx=0x7f64100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64100044e0 con 0x7f6420100610 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642019f740 con 0x7f6420100610 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f641002de90 con 0x7f6420100610 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410005410 con 0x7f6420100610 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.224+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642019fc50 con 0x7f6420100610 2024-03-24T09:19:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.225+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f642010bed0 con 0x7f6420100610 2024-03-24T09:19:39.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.226+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f6410005570 con 0x7f6420100610 2024-03-24T09:19:39.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.227+0000 7f641dffb640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f64080a20c0 0x7f64080a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.227+0000 7f641ffff640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f64080a20c0 0x7f64080a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.227+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f64100efe50 con 0x7f6420100610 2024-03-24T09:19:39.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.227+0000 7f641ffff640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f64080a20c0 0x7f64080a4580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6420198bb0 tx=0x7f6414005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:39.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.231+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f64100b8f70 con 0x7f6420100610 2024-03-24T09:19:39.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.413+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f642010c0b0 con 0x7f6420100610 2024-03-24T09:19:39.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:39 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1181212686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T09:19:39.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:39 smithi116 ceph-mon[32207]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T09:19:39.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:39 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/93944311' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T09:19:39.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:39 smithi116 ceph-mon[32207]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T09:19:39.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:39 smithi116 ceph-mon[32207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:19:39.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:39 smithi116 ceph-mon[32207]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T09:19:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:39 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1181212686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T09:19:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:39 smithi143 ceph-mon[35965]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T09:19:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:39 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/93944311' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T09:19:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:39 smithi143 ceph-mon[35965]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T09:19:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:39 smithi143 ceph-mon[35965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:19:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:39 smithi143 ceph-mon[35965]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T09:19:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:39 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1181212686' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-24T09:19:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:39 smithi117 ceph-mon[35788]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T09:19:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:39 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/93944311' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T09:19:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:39 smithi117 ceph-mon[35788]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T09:19:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:39 smithi117 ceph-mon[35788]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:19:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:39 smithi117 ceph-mon[35788]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T09:19:39.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.708+0000 7f641dffb640 1 -- 172.21.15.116:0/2610436856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f64100bc630 con 0x7f6420100610 2024-03-24T09:19:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f64080a20c0 msgr2=0x7f64080a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f64080a20c0 0x7f64080a4580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6420198bb0 tx=0x7f6414005d90 comp rx=0 tx=0).stop 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 msgr2=0x7f64201976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f64201976f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6410000c00 tx=0x7f64100042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 shutdown_connections 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f64080a20c0 0x7f64080a4580 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f64201021c0 0x7f642019ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6420101800 0x7f6420197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 --2- 172.21.15.116:0/2610436856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420100610 0x7f64201976f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 >> 172.21.15.116:0/2610436856 conn(0x7f64200fbb80 msgr2=0x7f6420068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 shutdown_connections 2024-03-24T09:19:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.711+0000 7f6427055640 1 -- 172.21.15.116:0/2610436856 wait complete. 2024-03-24T09:19:39.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T09:19:39.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/3983017114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 msgr2=0x7fc6c4101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/3983017114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4101ca0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009900 tx=0x7fc6b402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:39.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/3983017114 shutdown_connections 2024-03-24T09:19:39.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/3983017114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c41021e0 0x7fc6c4108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/3983017114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4101ca0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/3983017114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6c4100630 0x7fc6c4100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/3983017114 >> 172.21.15.116:0/3983017114 conn(0x7fc6c40fbbc0 msgr2=0x7fc6c40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/3983017114 shutdown_connections 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/3983017114 wait complete. 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 Processor -- start 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.833+0000 7fc6cc4e8640 1 -- start start 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6c4100630 0x7fc6c410aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c41021e0 0x7fc6c4109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c410a1d0 con 0x7fc6c4101820 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c410a340 con 0x7fc6c4100630 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc6c41a16e0 con 0x7fc6c41021e0 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41080/0 (socket says 172.21.15.116:41080) 2024-03-24T09:19:39.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 -- 172.21.15.116:0/1139737147 learned_addr learned my addr 172.21.15.116:0/1139737147 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6ca284640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6c4100630 0x7fc6c410aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 -- 172.21.15.116:0/1139737147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c41021e0 msgr2=0x7fc6c4109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c41021e0 0x7fc6c4109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 -- 172.21.15.116:0/1139737147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6c4100630 msgr2=0x7fc6c410aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6c4100630 0x7fc6c410aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 -- 172.21.15.116:0/1139737147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b4009560 con 0x7fc6c4101820 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6c9a83640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4109530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009430 tx=0x7fc6b40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4004510 con 0x7fc6c4101820 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc6b402de90 con 0x7fc6c4101820 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4003cc0 con 0x7fc6c4101820 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6c41a1930 con 0x7fc6c4101820 2024-03-24T09:19:39.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.834+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6c41a1e90 con 0x7fc6c4101820 2024-03-24T09:19:39.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.835+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6c4107950 con 0x7fc6c4101820 2024-03-24T09:19:39.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.836+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fc6b4004fd0 con 0x7fc6c4101820 2024-03-24T09:19:39.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.837+0000 7fc6bb7fe640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc6b00a2140 0x7fc6b00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:39.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.837+0000 7fc6ca284640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc6b00a2140 0x7fc6b00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:39.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.837+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fc6b40f0650 con 0x7fc6c4101820 2024-03-24T09:19:39.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.837+0000 7fc6ca284640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc6b00a2140 0x7fc6b00a4600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0005fd0 tx=0x7fc6c0005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:39.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:39.840+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc6b40b95f0 con 0x7fc6c4101820 2024-03-24T09:19:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:40.019+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc6c410cff0 con 0x7fc6c4101820 2024-03-24T09:19:40.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/93944311' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2610436856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2610436856' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: pgmap v288: 97 pgs: 63 active+clean, 34 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-03-24T09:19:40.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:40 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1139737147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T09:19:40.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/93944311' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T09:19:40.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T09:19:40.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T09:19:40.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2610436856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T09:19:40.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2610436856' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T09:19:40.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T09:19:40.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: pgmap v288: 97 pgs: 63 active+clean, 34 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-03-24T09:19:40.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:40 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1139737147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/93944311' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2610436856' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2610436856' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: pgmap v288: 97 pgs: 63 active+clean, 34 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 6 op/s 2024-03-24T09:19:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:40 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1139737147' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T09:19:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:41 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] up:active 2024-03-24T09:19:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:41 smithi143 ceph-mon[35965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:41.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:41 smithi143 ceph-mon[35965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:41.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:41 smithi143 ceph-mon[35965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:19:41.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:41 smithi143 ceph-mon[35965]: Cluster is now healthy 2024-03-24T09:19:41.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:41 smithi143 ceph-mon[35965]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T09:19:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:41 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] up:active 2024-03-24T09:19:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:41 smithi117 ceph-mon[35788]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:41 smithi117 ceph-mon[35788]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:41 smithi117 ceph-mon[35788]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:19:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:41 smithi117 ceph-mon[35788]: Cluster is now healthy 2024-03-24T09:19:41.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:41 smithi117 ceph-mon[35788]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T09:19:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:41 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] up:active 2024-03-24T09:19:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:41 smithi116 ceph-mon[32207]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:41 smithi116 ceph-mon[32207]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:41 smithi116 ceph-mon[32207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:19:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:41 smithi116 ceph-mon[32207]: Cluster is now healthy 2024-03-24T09:19:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:41 smithi116 ceph-mon[32207]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T09:19:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.715+0000 7fc6bb7fe640 1 -- 172.21.15.116:0/1139737147 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fc6b40bccb0 con 0x7fc6c4101820 2024-03-24T09:19:41.716 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data_ec' created 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc6b00a2140 msgr2=0x7fc6b00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc6b00a2140 0x7fc6b00a4600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc6c0005fd0 tx=0x7fc6c0005850 comp rx=0 tx=0).stop 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 msgr2=0x7fc6c4109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4109530 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009430 tx=0x7fc6b40042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 shutdown_connections 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc6b00a2140 0x7fc6b00a4600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc6c41021e0 0x7fc6c4109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6c4101820 0x7fc6c4109530 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 --2- 172.21.15.116:0/1139737147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6c4100630 0x7fc6c410aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 >> 172.21.15.116:0/1139737147 conn(0x7fc6c40fbbc0 msgr2=0x7fc6c4104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:41.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 shutdown_connections 2024-03-24T09:19:41.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.718+0000 7fc6cc4e8640 1 -- 172.21.15.116:0/1139737147 wait complete. 2024-03-24T09:19:41.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-24T09:19:41.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 -- 172.21.15.116:0/2968022063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 msgr2=0x7ff18c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:41.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 --2- 172.21.15.116:0/2968022063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c1081f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff184009ca0 tx=0x7ff18402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 -- 172.21.15.116:0/2968022063 shutdown_connections 2024-03-24T09:19:41.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 --2- 172.21.15.116:0/2968022063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c1081f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 --2- 172.21.15.116:0/2968022063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 0x7ff18c101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 --2- 172.21.15.116:0/2968022063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff18c0ffdb0 0x7ff18c1001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 -- 172.21.15.116:0/2968022063 >> 172.21.15.116:0/2968022063 conn(0x7ff18c0fb300 msgr2=0x7ff18c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 -- 172.21.15.116:0/2968022063 shutdown_connections 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 -- 172.21.15.116:0/2968022063 wait complete. 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 Processor -- start 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.842+0000 7ff19438a640 1 -- start start 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff18c0ffdb0 0x7ff18c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:41.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 0x7ff18c196b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c19e140 con 0x7ff18c101960 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c19e2b0 con 0x7ff18c100fa0 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff18c19e470 con 0x7ff18c0ffdb0 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff191925640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 0x7ff18c196b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c19dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41102/0 (socket says 172.21.15.116:41102) 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 -- 172.21.15.116:0/1040618099 learned_addr learned my addr 172.21.15.116:0/1040618099 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 -- 172.21.15.116:0/1040618099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff18c0ffdb0 msgr2=0x7ff18c196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192126640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff18c0ffdb0 0x7ff18c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:41.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff18c0ffdb0 0x7ff18c196650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 -- 172.21.15.116:0/1040618099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 msgr2=0x7ff18c196b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 0x7ff18c196b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 -- 172.21.15.116:0/1040618099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff184009900 con 0x7ff18c101960 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff191925640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 0x7ff18c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff192927640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c19dc00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff184037040 tx=0x7ff184032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184034e70 con 0x7ff18c101960 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff184003e20 con 0x7ff18c101960 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18c19e6f0 con 0x7ff18c101960 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff184004800 con 0x7ff18c101960 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.843+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18c19ec20 con 0x7ff18c101960 2024-03-24T09:19:41.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.844+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff18c1071a0 con 0x7ff18c101960 2024-03-24T09:19:41.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.845+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7ff18403b040 con 0x7ff18c101960 2024-03-24T09:19:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.846+0000 7ff1837fe640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1780a20c0 0x7ff1780a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.846+0000 7ff192126640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1780a20c0 0x7ff1780a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.846+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7ff1840f09f0 con 0x7ff18c101960 2024-03-24T09:19:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.847+0000 7ff192126640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1780a20c0 0x7ff1780a4580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff18c100e00 tx=0x7ff188008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:41.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:41.850+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1840b97a0 con 0x7ff18c101960 2024-03-24T09:19:42.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.025+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7ff18c02d240 con 0x7ff18c101960 2024-03-24T09:19:42.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.721+0000 7ff1837fe640 1 -- 172.21.15.116:0/1040618099 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7ff1840bce60 con 0x7ff18c101960 2024-03-24T09:19:42.722 INFO:teuthology.orchestra.run.smithi116.stderr:set pool 4 allow_ec_overwrites to true 2024-03-24T09:19:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.723+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1780a20c0 msgr2=0x7ff1780a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.723+0000 7ff19438a640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1780a20c0 0x7ff1780a4580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff18c100e00 tx=0x7ff188008040 comp rx=0 tx=0).stop 2024-03-24T09:19:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.723+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 msgr2=0x7ff18c19dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.723+0000 7ff19438a640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c19dc00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff184037040 tx=0x7ff184032980 comp rx=0 tx=0).stop 2024-03-24T09:19:42.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 shutdown_connections 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff18c101960 0x7ff18c19dc00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff18c100fa0 0x7ff18c196b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1780a20c0 0x7ff1780a4580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 --2- 172.21.15.116:0/1040618099 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff18c0ffdb0 0x7ff18c196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 >> 172.21.15.116:0/1040618099 conn(0x7ff18c0fb300 msgr2=0x7ff18c104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 shutdown_connections 2024-03-24T09:19:42.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.724+0000 7ff19438a640 1 -- 172.21.15.116:0/1040618099 wait complete. 2024-03-24T09:19:42.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-24T09:19:42.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 -- 172.21.15.116:0/1970831568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac100610 msgr2=0x7f29ac100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:42.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/1970831568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac100610 0x7f29ac100a30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009900 tx=0x7f29a802bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:42.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 -- 172.21.15.116:0/1970831568 shutdown_connections 2024-03-24T09:19:42.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/1970831568 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ac1021c0 0x7f29ac108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/1970831568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ac101800 0x7f29ac101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/1970831568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac100610 0x7f29ac100a30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 -- 172.21.15.116:0/1970831568 >> 172.21.15.116:0/1970831568 conn(0x7f29ac0fbba0 msgr2=0x7f29ac0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 -- 172.21.15.116:0/1970831568 shutdown_connections 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.853+0000 7f29b3ef4640 1 -- 172.21.15.116:0/1970831568 wait complete. 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 Processor -- start 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 -- start start 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ac100610 0x7f29ac069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ac101800 0x7f29ac19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 0x7f29ac19eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:42.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac19f2a0 con 0x7f29ac1021c0 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac19f410 con 0x7f29ac100610 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b3ef4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac19f5d0 con 0x7f29ac101800 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 0x7f29ac19eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 0x7f29ac19eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41122/0 (socket says 172.21.15.116:41122) 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 -- 172.21.15.116:0/3048507432 learned_addr learned my addr 172.21.15.116:0/3048507432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b1c90640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ac100610 0x7f29ac069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 -- 172.21.15.116:0/3048507432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ac101800 msgr2=0x7f29ac19c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b148f640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ac101800 0x7f29ac19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ac101800 0x7f29ac19c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 -- 172.21.15.116:0/3048507432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ac100610 msgr2=0x7f29ac069400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ac100610 0x7f29ac069400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 -- 172.21.15.116:0/3048507432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a8009560 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.854+0000 7f29b2491640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 0x7f29ac19eb10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f29a400cad0 tx=0x7f29a400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.855+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a400ae20 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.855+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29a4005020 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.855+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ac19f8b0 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.855+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a400f690 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.855+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ac19fe10 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.856+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ac107aa0 con 0x7f29ac1021c0 2024-03-24T09:19:42.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.857+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f29a4005190 con 0x7f29ac1021c0 2024-03-24T09:19:42.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.857+0000 7f29a2ffd640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29980a2140 0x7f29980a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:42.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.857+0000 7f29b1c90640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29980a2140 0x7f29980a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:42.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.858+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f29a40cde50 con 0x7f29ac1021c0 2024-03-24T09:19:42.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.858+0000 7f29b1c90640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29980a2140 0x7f29980a4600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29a802c7d0 tx=0x7f29a8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:42.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:42.862+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f29a4096c00 con 0x7f29ac1021c0 2024-03-24T09:19:42.989 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:42 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1139737147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T09:19:42.989 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:42 smithi117 ceph-mon[35788]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T09:19:42.990 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:42 smithi117 ceph-mon[35788]: pgmap v291: 129 pgs: 82 active+clean, 47 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 9 op/s 2024-03-24T09:19:42.990 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:42 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1040618099' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T09:19:43.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:42 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1139737147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T09:19:43.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:42 smithi143 ceph-mon[35965]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T09:19:43.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:42 smithi143 ceph-mon[35965]: pgmap v291: 129 pgs: 82 active+clean, 47 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 9 op/s 2024-03-24T09:19:43.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:42 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1040618099' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T09:19:43.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:42 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1139737147' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T09:19:43.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:42 smithi116 ceph-mon[32207]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T09:19:43.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:42 smithi116 ceph-mon[32207]: pgmap v291: 129 pgs: 82 active+clean, 47 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 9 op/s 2024-03-24T09:19:43.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:42 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1040618099' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-24T09:19:43.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.081+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f29ac02d130 con 0x7f29ac1021c0 2024-03-24T09:19:43.316 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:19:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: 2024-03-24T09:19:42.988+0000 7f12dea0a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T09:19:43.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.722+0000 7f29a2ffd640 1 -- 172.21.15.116:0/3048507432 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f29a409a2c0 con 0x7f29ac1021c0 2024-03-24T09:19:43.723 INFO:teuthology.orchestra.run.smithi116.stderr:added data pool 4 to fsmap 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29980a2140 msgr2=0x7f29980a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29980a2140 0x7f29980a4600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f29a802c7d0 tx=0x7f29a8005fd0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 msgr2=0x7f29ac19eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 0x7f29ac19eb10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f29a400cad0 tx=0x7f29a400cf90 comp rx=0 tx=0).stop 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 shutdown_connections 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ac1021c0 0x7f29ac19eb10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f29980a2140 0x7f29980a4600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ac101800 0x7f29ac19c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 --2- 172.21.15.116:0/3048507432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ac100610 0x7f29ac069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 >> 172.21.15.116:0/3048507432 conn(0x7f29ac0fbba0 msgr2=0x7f29ac104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:43.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 shutdown_connections 2024-03-24T09:19:43.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.726+0000 7f29b3ef4640 1 -- 172.21.15.116:0/3048507432 wait complete. 2024-03-24T09:19:43.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 -- 172.21.15.116:0/3945388167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678100650 msgr2=0x7fb678100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/3945388167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678100650 0x7fb678100a70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009900 tx=0x7fb66c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 -- 172.21.15.116:0/3945388167 shutdown_connections 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/3945388167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb678102200 0x7fb678108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/3945388167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678101840 0x7fb678101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/3945388167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678100650 0x7fb678100a70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 -- 172.21.15.116:0/3945388167 >> 172.21.15.116:0/3945388167 conn(0x7fb6780fbbc0 msgr2=0x7fb6780fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:43.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 -- 172.21.15.116:0/3945388167 shutdown_connections 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.847+0000 7fb67f0b1640 1 -- 172.21.15.116:0/3945388167 wait complete. 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 Processor -- start 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 -- start start 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678100650 0x7fb678196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb678101840 0x7fb678196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 0x7fb67819dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb67819e110 con 0x7fb678102200 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb67819e280 con 0x7fb678100650 2024-03-24T09:19:43.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67f0b1640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb67819e440 con 0x7fb678101840 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 0x7fb67819dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67ce4d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678100650 0x7fb678196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67ce4d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678100650 0x7fb678196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:56528/0 (socket says 172.21.15.116:56528) 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 0x7fb67819dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41144/0 (socket says 172.21.15.116:41144) 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67ce4d640 1 -- 172.21.15.116:0/1763662405 learned_addr learned my addr 172.21.15.116:0/1763662405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb677fff640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb678101840 0x7fb678196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 -- 172.21.15.116:0/1763662405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb678101840 msgr2=0x7fb678196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb678101840 0x7fb678196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 -- 172.21.15.116:0/1763662405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678100650 msgr2=0x7fb678196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:43.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678100650 0x7fb678196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.848+0000 7fb67d64e640 1 -- 172.21.15.116:0/1763662405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c009560 con 0x7fb678102200 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.849+0000 7fb67d64e640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 0x7fb67819dbd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb670009fd0 tx=0x7fb67000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.849+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67000ce20 con 0x7fb678102200 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.849+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67819e720 con 0x7fb678102200 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.849+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb670005020 con 0x7fb678102200 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.849+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670010690 con 0x7fb678102200 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.849+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67819ec30 con 0x7fb678102200 2024-03-24T09:19:43.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.850+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb678107a10 con 0x7fb678102200 2024-03-24T09:19:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.851+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fb670005190 con 0x7fb678102200 2024-03-24T09:19:43.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.851+0000 7fb675ffb640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb6680a20c0 0x7fb6680a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.851+0000 7fb67ce4d640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb6680a20c0 0x7fb6680a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.851+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb670014070 con 0x7fb678102200 2024-03-24T09:19:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.852+0000 7fb67ce4d640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb6680a20c0 0x7fb6680a4580 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb66c006010 tx=0x7fb66c005d50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:43.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:43.856+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb6700973c0 con 0x7fb678102200 2024-03-24T09:19:44.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:43 smithi143 ceph-mon[35965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:19:44.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:43 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1040618099' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T09:19:44.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:43 smithi143 ceph-mon[35965]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T09:19:44.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:43 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3048507432' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T09:19:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:43 smithi117 ceph-mon[35788]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:19:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:43 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1040618099' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T09:19:44.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:43 smithi117 ceph-mon[35788]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T09:19:44.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:43 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3048507432' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T09:19:44.067 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:43 smithi116 ceph-mon[32207]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T09:19:44.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:43 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1040618099' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-24T09:19:44.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:43 smithi116 ceph-mon[32207]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T09:19:44.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:43 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3048507432' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-24T09:19:44.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.069+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb67802d160 con 0x7fb678102200 2024-03-24T09:19:44.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.069+0000 7fb675ffb640 1 -- 172.21.15.116:0/1763662405 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7fb6700b2e40 con 0x7fb678102200 2024-03-24T09:19:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:44.070 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:19:43.082899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24515,"qdb_cluster":[24515]},"id":1}]} 2024-03-24T09:19:44.071 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb6680a20c0 msgr2=0x7fb6680a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb6680a20c0 0x7fb6680a4580 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb66c006010 tx=0x7fb66c005d50 comp rx=0 tx=0).stop 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 msgr2=0x7fb67819dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 0x7fb67819dbd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb670009fd0 tx=0x7fb67000ef10 comp rx=0 tx=0).stop 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 shutdown_connections 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb678102200 0x7fb67819dbd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb6680a20c0 0x7fb6680a4580 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb678101840 0x7fb678196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 --2- 172.21.15.116:0/1763662405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb678100650 0x7fb678196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 >> 172.21.15.116:0/1763662405 conn(0x7fb6780fbbc0 msgr2=0x7fb678104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 shutdown_connections 2024-03-24T09:19:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.072+0000 7fb67f0b1640 1 -- 172.21.15.116:0/1763662405 wait complete. 2024-03-24T09:19:44.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:19:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 -- 172.21.15.116:0/4287334471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 msgr2=0x7fb87c109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 --2- 172.21.15.116:0/4287334471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c109b70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb874009ca0 tx=0x7fb87402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 -- 172.21.15.116:0/4287334471 shutdown_connections 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 --2- 172.21.15.116:0/4287334471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c109b70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 --2- 172.21.15.116:0/4287334471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 0x7fb87c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 --2- 172.21.15.116:0/4287334471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb87c101730 0x7fb87c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 -- 172.21.15.116:0/4287334471 >> 172.21.15.116:0/4287334471 conn(0x7fb87c0fccc0 msgr2=0x7fb87c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 -- 172.21.15.116:0/4287334471 shutdown_connections 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 -- 172.21.15.116:0/4287334471 wait complete. 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 Processor -- start 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.185+0000 7fb8845b5640 1 -- start start 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb87c101730 0x7fb87c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 0x7fb87c197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c19f1b0 con 0x7fb87c1032e0 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c19f320 con 0x7fb87c101730 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c19f4e0 con 0x7fb87c102920 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41170/0 (socket says 172.21.15.116:41170) 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 -- 172.21.15.116:0/634379677 learned_addr learned my addr 172.21.15.116:0/634379677 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882351640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb87c101730 0x7fb87c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb881b50640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 0x7fb87c197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 -- 172.21.15.116:0/634379677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 msgr2=0x7fb87c197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 0x7fb87c197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 -- 172.21.15.116:0/634379677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb87c101730 msgr2=0x7fb87c1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb87c101730 0x7fb87c1976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 -- 172.21.15.116:0/634379677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb874009900 con 0x7fb87c1032e0 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb881b50640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 0x7fb87c197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb882b52640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c19ec70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb87402c630 tx=0x7fb874032900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874034e70 con 0x7fb87c1032e0 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c19f760 con 0x7fb87c1032e0 2024-03-24T09:19:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb874003e20 con 0x7fb87c1032e0 2024-03-24T09:19:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874004750 con 0x7fb87c1032e0 2024-03-24T09:19:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.186+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c19fc10 con 0x7fb87c1032e0 2024-03-24T09:19:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.187+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c108a50 con 0x7fb87c1032e0 2024-03-24T09:19:44.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.188+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fb87403b040 con 0x7fb87c1032e0 2024-03-24T09:19:44.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.189+0000 7fb8737fe640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb8680a20c0 0x7fb8680a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.189+0000 7fb882351640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb8680a20c0 0x7fb8680a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.189+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb8740f0490 con 0x7fb87c1032e0 2024-03-24T09:19:44.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.189+0000 7fb882351640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb8680a20c0 0x7fb8680a4580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb87c102780 tx=0x7fb86c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:44.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.193+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb8740b9220 con 0x7fb87c1032e0 2024-03-24T09:19:44.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.369+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb87c02d1d0 con 0x7fb87c1032e0 2024-03-24T09:19:44.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.370+0000 7fb8737fe640 1 -- 172.21.15.116:0/634379677 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7fb8740bc8e0 con 0x7fb87c1032e0 2024-03-24T09:19:44.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:44.372 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:19:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb8680a20c0 msgr2=0x7fb8680a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb8680a20c0 0x7fb8680a4580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb87c102780 tx=0x7fb86c005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 msgr2=0x7fb87c19ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c19ec70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb87402c630 tx=0x7fb874032900 comp rx=0 tx=0).stop 2024-03-24T09:19:44.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 shutdown_connections 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb87c1032e0 0x7fb87c19ec70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb8680a20c0 0x7fb8680a4580 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb87c102920 0x7fb87c197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 --2- 172.21.15.116:0/634379677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb87c101730 0x7fb87c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 >> 172.21.15.116:0/634379677 conn(0x7fb87c0fccc0 msgr2=0x7fb87c105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 shutdown_connections 2024-03-24T09:19:44.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.372+0000 7fb8845b5640 1 -- 172.21.15.116:0/634379677 wait complete. 2024-03-24T09:19:44.385 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 -- 172.21.15.116:0/2210974003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 msgr2=0x7ff810101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 --2- 172.21.15.116:0/2210974003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff810101ca0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff804009900 tx=0x7ff80402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 -- 172.21.15.116:0/2210974003 shutdown_connections 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 --2- 172.21.15.116:0/2210974003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8101021e0 0x7ff810108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 --2- 172.21.15.116:0/2210974003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff810101ca0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 --2- 172.21.15.116:0/2210974003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff810100630 0x7ff810100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 -- 172.21.15.116:0/2210974003 >> 172.21.15.116:0/2210974003 conn(0x7ff8100fbba0 msgr2=0x7ff8100fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 -- 172.21.15.116:0/2210974003 shutdown_connections 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.486+0000 7ff817e64640 1 -- 172.21.15.116:0/2210974003 wait complete. 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 Processor -- start 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 -- start start 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff810100630 0x7ff8100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff81019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8101021e0 0x7ff81019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff81019f240 con 0x7ff810101820 2024-03-24T09:19:44.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff81019f3b0 con 0x7ff810100630 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff81019f570 con 0x7ff8101021e0 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff81019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff81019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41194/0 (socket says 172.21.15.116:41194) 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 -- 172.21.15.116:0/742179247 learned_addr learned my addr 172.21.15.116:0/742179247 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff815c00640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff810100630 0x7ff8100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff816401640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8101021e0 0x7ff81019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 -- 172.21.15.116:0/742179247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8101021e0 msgr2=0x7ff81019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8101021e0 0x7ff81019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 -- 172.21.15.116:0/742179247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff810100630 msgr2=0x7ff8100693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff810100630 0x7ff8100693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 -- 172.21.15.116:0/742179247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff804009560 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff8153ff640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff81019c440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff804005ab0 tx=0x7ff8040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804004ac0 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8040044e0 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81019f790 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8040055c0 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.487+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81019fca0 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.488+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff810107950 con 0x7ff810101820 2024-03-24T09:19:44.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.489+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7ff804004dc0 con 0x7ff810101820 2024-03-24T09:19:44.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.490+0000 7ff802ffd640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff7fc0a20c0 0x7ff7fc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.490+0000 7ff815c00640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff7fc0a20c0 0x7ff7fc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.490+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff80402de40 con 0x7ff810101820 2024-03-24T09:19:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.490+0000 7ff815c00640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff7fc0a20c0 0x7ff7fc0a4580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff810101680 tx=0x7ff80c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:44.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.493+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff8040b9a00 con 0x7ff810101820 2024-03-24T09:19:44.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.667+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff81010c190 con 0x7ff810101820 2024-03-24T09:19:44.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.669+0000 7ff802ffd640 1 -- 172.21.15.116:0/742179247 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7ff804043020 con 0x7ff810101820 2024-03-24T09:19:44.670 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:44.670 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:19:44.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff7fc0a20c0 msgr2=0x7ff7fc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff7fc0a20c0 0x7ff7fc0a4580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff810101680 tx=0x7ff80c005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:44.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 msgr2=0x7ff81019c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff81019c440 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff804005ab0 tx=0x7ff8040042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 shutdown_connections 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff7fc0a20c0 0x7ff7fc0a4580 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8101021e0 0x7ff81019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff810101820 0x7ff81019c440 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 --2- 172.21.15.116:0/742179247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff810100630 0x7ff8100693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 >> 172.21.15.116:0/742179247 conn(0x7ff8100fbba0 msgr2=0x7ff810104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 shutdown_connections 2024-03-24T09:19:44.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.671+0000 7ff817e64640 1 -- 172.21.15.116:0/742179247 wait complete. 2024-03-24T09:19:44.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:19:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 -- 172.21.15.116:0/2256778995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae641021b0 msgr2=0x7fae641045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 --2- 172.21.15.116:0/2256778995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae641021b0 0x7fae641045a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae54009900 tx=0x7fae5402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 -- 172.21.15.116:0/2256778995 shutdown_connections 2024-03-24T09:19:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 --2- 172.21.15.116:0/2256778995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae64107410 0x7fae64109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 --2- 172.21.15.116:0/2256778995 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae64104ae0 0x7fae64106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 --2- 172.21.15.116:0/2256778995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae641021b0 0x7fae641045a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 -- 172.21.15.116:0/2256778995 >> 172.21.15.116:0/2256778995 conn(0x7fae640fbba0 msgr2=0x7fae640fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 -- 172.21.15.116:0/2256778995 shutdown_connections 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.801+0000 7fae6af32640 1 -- 172.21.15.116:0/2256778995 wait complete. 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 Processor -- start 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 -- start start 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae641021b0 0x7fae64197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 0x7fae64197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae64107410 0x7fae6419ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae6419f220 con 0x7fae64104ae0 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fae6419f390 con 0x7fae64107410 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae6af32640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fae6419f550 con 0x7fae641021b0 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 0x7fae64197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 0x7fae64197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41214/0 (socket says 172.21.15.116:41214) 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae63fff640 1 -- 172.21.15.116:0/900564955 learned_addr learned my addr 172.21.15.116:0/900564955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae694cf640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae64107410 0x7fae6419ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.802+0000 7fae68cce640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae641021b0 0x7fae64197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae63fff640 1 -- 172.21.15.116:0/900564955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae641021b0 msgr2=0x7fae64197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae63fff640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae641021b0 0x7fae64197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae63fff640 1 -- 172.21.15.116:0/900564955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae64107410 msgr2=0x7fae6419ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae63fff640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae64107410 0x7fae6419ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae63fff640 1 -- 172.21.15.116:0/900564955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae54009560 con 0x7fae64104ae0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae63fff640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 0x7fae64197cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c970 tx=0x7fae5800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae58007bb0 con 0x7fae64104ae0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fae58007d10 con 0x7fae64104ae0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae6419f830 con 0x7fae64104ae0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae58018660 con 0x7fae64104ae0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.803+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae6419fd40 con 0x7fae64104ae0 2024-03-24T09:19:44.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.804+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae640fd690 con 0x7fae64104ae0 2024-03-24T09:19:44.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.805+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fae5800f450 con 0x7fae64104ae0 2024-03-24T09:19:44.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.806+0000 7fae61ffb640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fae4c0a2140 0x7fae4c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:44.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.806+0000 7fae68cce640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fae4c0a2140 0x7fae4c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:44.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.806+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fae580cde70 con 0x7fae64104ae0 2024-03-24T09:19:44.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.806+0000 7fae68cce640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fae4c0a2140 0x7fae4c0a4600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae54009230 tx=0x7fae54005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:44.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:44.810+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fae58096c00 con 0x7fae64104ae0 2024-03-24T09:19:45.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.022+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae6410cff0 con 0x7fae64104ae0 2024-03-24T09:19:45.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.022+0000 7fae61ffb640 1 -- 172.21.15.116:0/900564955 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11818 (secure 0 0 0) 0x7fae5809a2c0 con 0x7fae64104ae0 2024-03-24T09:19:45.023 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:45.023 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24422,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24451,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:19:43.082899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24515,"qdb_cluster":[24515]},"id":1}]} 2024-03-24T09:19:45.024 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-03-24T09:19:45.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3048507432' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T09:19:45.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T09:19:45.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:45.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: pgmap v294: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-03-24T09:19:45.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1763662405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/634379677' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:44 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/742179247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fae4c0a2140 msgr2=0x7fae4c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fae4c0a2140 0x7fae4c0a4600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae54009230 tx=0x7fae54005fd0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 msgr2=0x7fae64197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 0x7fae64197cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c970 tx=0x7fae5800ce30 comp rx=0 tx=0).stop 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 shutdown_connections 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae64107410 0x7fae6419ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae64104ae0 0x7fae64197cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.024+0000 7fae6af32640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fae4c0a2140 0x7fae4c0a4600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.025+0000 7fae6af32640 1 --2- 172.21.15.116:0/900564955 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fae641021b0 0x7fae64197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.025+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 >> 172.21.15.116:0/900564955 conn(0x7fae640fbba0 msgr2=0x7fae64102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.025+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 shutdown_connections 2024-03-24T09:19:45.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.025+0000 7fae6af32640 1 -- 172.21.15.116:0/900564955 wait complete. 2024-03-24T09:19:45.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3048507432' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T09:19:45.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T09:19:45.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:45.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: pgmap v294: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-03-24T09:19:45.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1763662405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/634379677' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:44 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/742179247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:19:45.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3048507432' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-24T09:19:45.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T09:19:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:19:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: pgmap v294: 129 pgs: 2 creating+peering, 30 unknown, 97 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 3 op/s 2024-03-24T09:19:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1763662405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/634379677' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:44 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/742179247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:45.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 -- 172.21.15.116:0/1289400824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 msgr2=0x7ff154101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1289400824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff154101ca0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff144009900 tx=0x7ff14402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:45.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 -- 172.21.15.116:0/1289400824 shutdown_connections 2024-03-24T09:19:45.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1289400824 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1541021e0 0x7ff154108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1289400824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff154101ca0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1289400824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff154100630 0x7ff154100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.137+0000 7ff15bb81640 1 -- 172.21.15.116:0/1289400824 >> 172.21.15.116:0/1289400824 conn(0x7ff1540fbba0 msgr2=0x7ff1540fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 -- 172.21.15.116:0/1289400824 shutdown_connections 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 -- 172.21.15.116:0/1289400824 wait complete. 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 Processor -- start 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 -- start start 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff154100630 0x7ff154069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff15419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1541021e0 0x7ff15419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff15419f220 con 0x7ff154101820 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff15419f390 con 0x7ff1541021e0 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15bb81640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff15419f550 con 0x7ff154100630 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15911c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff15419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15911c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff15419c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41236/0 (socket says 172.21.15.116:41236) 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15911c640 1 -- 172.21.15.116:0/1027757377 learned_addr learned my addr 172.21.15.116:0/1027757377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15991d640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff154100630 0x7ff154069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.138+0000 7ff15a11e640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1541021e0 0x7ff15419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15911c640 1 -- 172.21.15.116:0/1027757377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff154100630 msgr2=0x7ff154069400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15911c640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff154100630 0x7ff154069400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15911c640 1 -- 172.21.15.116:0/1027757377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1541021e0 msgr2=0x7ff15419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15911c640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1541021e0 0x7ff15419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15911c640 1 -- 172.21.15.116:0/1027757377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff150009840 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15911c640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff15419c450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff144036040 tx=0x7ff144004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1440035f0 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff144009560 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff14402de90 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144005490 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.139+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15419fb50 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.140+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1541079a0 con 0x7ff154101820 2024-03-24T09:19:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.141+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7ff144003750 con 0x7ff154101820 2024-03-24T09:19:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.141+0000 7ff14affd640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1400a20c0 0x7ff1400a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.141+0000 7ff15991d640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1400a20c0 0x7ff1400a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.141+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff1440f0130 con 0x7ff154101820 2024-03-24T09:19:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.142+0000 7ff15991d640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1400a20c0 0x7ff1400a4580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff154101680 tx=0x7ff150009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.146+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1440b8ec0 con 0x7ff154101820 2024-03-24T09:19:45.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.320+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff15402d160 con 0x7ff154101820 2024-03-24T09:19:45.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.322+0000 7ff14affd640 1 -- 172.21.15.116:0/1027757377 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7ff1440bc580 con 0x7ff154101820 2024-03-24T09:19:45.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:45.323 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1400a20c0 msgr2=0x7ff1400a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1400a20c0 0x7ff1400a4580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff154101680 tx=0x7ff150009500 comp rx=0 tx=0).stop 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 msgr2=0x7ff15419c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff15419c450 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff144036040 tx=0x7ff144004140 comp rx=0 tx=0).stop 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 shutdown_connections 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1541021e0 0x7ff15419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff154101820 0x7ff15419c450 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1400a20c0 0x7ff1400a4580 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 --2- 172.21.15.116:0/1027757377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff154100630 0x7ff154069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 >> 172.21.15.116:0/1027757377 conn(0x7ff1540fbba0 msgr2=0x7ff154104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 shutdown_connections 2024-03-24T09:19:45.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.324+0000 7ff15bb81640 1 -- 172.21.15.116:0/1027757377 wait complete. 2024-03-24T09:19:45.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T09:19:38.431545+0000', 'modified': '2024-03-24T09:19:43.082899+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6839/3950675281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3950675281}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3950675281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24515, 'qdb_cluster': [24515]} 2024-03-24T09:19:45.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T09:19:45.336 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 -- 172.21.15.116:0/3669705453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 msgr2=0x7fbed4100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/3669705453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4100a30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbec4009900 tx=0x7fbec402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 -- 172.21.15.116:0/3669705453 shutdown_connections 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/3669705453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed41021c0 0x7fbed4108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/3669705453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbed4101800 0x7fbed4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/3669705453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4100a30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 -- 172.21.15.116:0/3669705453 >> 172.21.15.116:0/3669705453 conn(0x7fbed40fbb80 msgr2=0x7fbed40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 -- 172.21.15.116:0/3669705453 shutdown_connections 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.439+0000 7fbed9e9d640 1 -- 172.21.15.116:0/3669705453 wait complete. 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 Processor -- start 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 -- start start 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed4101800 0x7fbed4197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbed41021c0 0x7fbed419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbed419f260 con 0x7fbed4100610 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbed419f3d0 con 0x7fbed41021c0 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed9e9d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbed419f590 con 0x7fbed4101800 2024-03-24T09:19:45.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4197770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41248/0 (socket says 172.21.15.116:41248) 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 -- 172.21.15.116:0/4049505866 learned_addr learned my addr 172.21.15.116:0/4049505866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed3fff640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbed41021c0 0x7fbed419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed4101800 0x7fbed4197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 -- 172.21.15.116:0/4049505866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed4101800 msgr2=0x7fbed4197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed4101800 0x7fbed4197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 -- 172.21.15.116:0/4049505866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbed41021c0 msgr2=0x7fbed419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbed41021c0 0x7fbed419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed37fe640 1 -- 172.21.15.116:0/4049505866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec4009560 con 0x7fbed4100610 2024-03-24T09:19:45.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.440+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed4101800 0x7fbed4197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:45.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.441+0000 7fbed37fe640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4197770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbec402c7d0 tx=0x7fbec40042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.441+0000 7fbed0ff9640 1 -- 172.21.15.116:0/4049505866 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4004aa0 con 0x7fbed4100610 2024-03-24T09:19:45.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.441+0000 7fbed0ff9640 1 -- 172.21.15.116:0/4049505866 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbec402de90 con 0x7fbed4100610 2024-03-24T09:19:45.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.441+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed419f810 con 0x7fbed4100610 2024-03-24T09:19:45.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.441+0000 7fbed0ff9640 1 -- 172.21.15.116:0/4049505866 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec4005440 con 0x7fbed4100610 2024-03-24T09:19:45.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.441+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed419fda0 con 0x7fbed4100610 2024-03-24T09:19:45.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbed0ff9640 1 -- 172.21.15.116:0/4049505866 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fbec40055a0 con 0x7fbed4100610 2024-03-24T09:19:45.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbedc722740 1 Processor -- start 2024-03-24T09:19:45.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbedc722740 1 -- start start 2024-03-24T09:19:45.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbed0ff9640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbebc0a20c0 0x7fbebc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbedc722740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3560 0x55cb13aa3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbedc722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 0x55cb13aa8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbedc722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13aab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.443+0000 7fbedc722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cb13a12180 con 0x55cb13aa3f50 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbedc722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cb13a0f6b0 con 0x55cb13aa90c0 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbedc722740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cb13a0ae90 con 0x55cb13aa3560 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed3fff640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbebc0a20c0 0x7fbebc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed0ff9640 1 -- 172.21.15.116:0/4049505866 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbec40f0090 con 0x7fbed4100610 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 0x55cb13aa8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 0x55cb13aa8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41250/0 (socket says 172.21.15.116:41250) 2024-03-24T09:19:45.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 -- 172.21.15.116:0/210472345 learned_addr learned my addr 172.21.15.116:0/210472345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed37fe640 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3560 0x55cb13aa3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed3fff640 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13aab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 -- 172.21.15.116:0/210472345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3560 msgr2=0x55cb13aa3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3560 0x55cb13aa3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 -- 172.21.15.116:0/210472345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 msgr2=0x55cb13aab550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13aab550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 -- 172.21.15.116:0/210472345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb139c7d80 con 0x55cb13aa3f50 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed3fff640 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13aab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed3fff640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbebc0a20c0 0x7fbebc0a4580 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009f70 tx=0x7fbecc009650 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 0x55cb13aa8a50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009c00 tx=0x7fbec8010370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cbafb03fcb3026 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbeb9ffb640 1 -- 172.21.15.116:0/210472345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8010da0 con 0x55cb13aa3f50 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbeb9ffb640 1 -- 172.21.15.116:0/210472345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbec80044e0 con 0x55cb13aa3f50 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.444+0000 7fbeb9ffb640 1 -- 172.21.15.116:0/210472345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec80126b0 con 0x55cb13aa3f50 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 -- 172.21.15.116:0/210472345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 msgr2=0x55cb13aa8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 0x55cb13aa8a50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009c00 tx=0x7fbec8010370 comp rx=0 tx=0).stop 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 -- 172.21.15.116:0/210472345 shutdown_connections 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13aab550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3f50 0x55cb13aa8a50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 --2- 172.21.15.116:0/210472345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3560 0x55cb13aa3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 -- 172.21.15.116:0/210472345 >> 172.21.15.116:0/210472345 conn(0x55cb13a9ead0 msgr2=0x55cb13a9eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 -- 172.21.15.116:0/210472345 shutdown_connections 2024-03-24T09:19:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 -- 172.21.15.116:0/210472345 wait complete. 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 -- start start 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.445+0000 7fbedc722740 1 Processor -- start 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13aad540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13aada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13ab4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cb139afd50 con 0x55cb13aa3560 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cb136de310 con 0x55cb13aa90c0 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cb139908f0 con 0x55cb13aa3f50 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 10 client.0 _ll_register_callbacks cb 0x55cb1371bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:19:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13aad540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13aad540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41266/0 (socket says 172.21.15.116:41266) 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 -- 172.21.15.116:0/1123643455 learned_addr learned my addr 172.21.15.116:0/1123643455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13aada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed3fff640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13ab4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 -- 172.21.15.116:0/1123643455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 msgr2=0x55cb13aada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13aada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 -- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 msgr2=0x55cb13ab4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13ab4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 -- 172.21.15.116:0/1123643455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec8009830 con 0x55cb13aa3560 2024-03-24T09:19:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13aada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbed37fe640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13aad540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbec4005940 tx=0x7fbec4004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8feb33ba7577c294 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbeb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1123643455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec40bcb80 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1123643455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbec40c0cd0 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cb1370a6c0 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1123643455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fbec40c5320 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 10 client.14769 fetch_fsmap learned FSMap version 17 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.446+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55cb13a09510 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1123643455 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55cb1370a6c0 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1123643455 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fbec403c070 con 0x55cb13aa3560 2024-03-24T09:19:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbedc722740 10 client.14769 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbedc722740 10 client.14769 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbedc722740 20 client.14769 populate_metadata read hostname 'smithi116' 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbedc722740 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13ab99c0 0x55cb13abae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbedc722740 4 client.14769 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb1375b270 con 0x55cb13ab99c0 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13ab99c0 0x55cb13abae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbeb2ffd640 10 client.14769 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.447+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13ab99c0 0x55cb13abae30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac87671828e7d319 server_cookie=16887318d65768fe in_seq=0 out_seq=0 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1123643455 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cb1375b270 con 0x55cb13ab99c0 2024-03-24T09:19:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbeb2ffd640 10 client.14769 handle_command_reply: tid=0 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 client.14769 shutdown 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 msgr2=0x55cb13aad540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13aad540 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbec4005940 tx=0x7fbec4004dc0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 shutdown_connections 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13ab99c0 0x55cb13abae30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa90c0 0x55cb13ab4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbeb2ffd640 0 client.14769 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13aada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 --2- 172.21.15.116:0/1123643455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13aad540 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 >> 172.21.15.116:0/1123643455 conn(0x55cb13a9ead0 msgr2=0x55cb13a9dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 shutdown_connections 2024-03-24T09:19:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.448+0000 7fbedc722740 1 -- 172.21.15.116:0/1123643455 wait complete. 2024-03-24T09:19:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.449+0000 7fbedc722740 20 client.14769 trim_cache size 0 max 16384 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 Processor -- start 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 -- start start 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13a9eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3f50 0x55cb13aad150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa90c0 0x55cb13aad690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cb13a9e990 con 0x55cb13aa3560 2024-03-24T09:19:45.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cb13ab9520 con 0x55cb13aa3f50 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbedc722740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cb13ab5ab0 con 0x55cb13aa90c0 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.466+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13a9eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13a9eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41282/0 (socket says 172.21.15.116:41282) 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 -- 172.21.15.116:0/1769199789 learned_addr learned my addr 172.21.15.116:0/1769199789 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3f50 0x55cb13aad150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed3fff640 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa90c0 0x55cb13aad690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 -- 172.21.15.116:0/1769199789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa90c0 msgr2=0x55cb13aad690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa90c0 0x55cb13aad690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 -- 172.21.15.116:0/1769199789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3f50 msgr2=0x55cb13aad150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3f50 0x55cb13aad150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 -- 172.21.15.116:0/1769199789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb139a5a80 con 0x55cb13aa3560 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed3fff640 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa90c0 0x55cb13aad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbed37fe640 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13a9eeb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbec40b9550 tx=0x7fbec40bb090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed8914a9cf57d1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbeb97fa640 1 -- 172.21.15.116:0/1769199789 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec40db0f0 con 0x55cb13aa3560 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbeb97fa640 1 -- 172.21.15.116:0/1769199789 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbec40c0cd0 con 0x55cb13aa3560 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbeb97fa640 1 -- 172.21.15.116:0/1769199789 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec40d0bd0 con 0x55cb13aa3560 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbedc722740 1 -- 172.21.15.116:0/1769199789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 msgr2=0x55cb13a9eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.467+0000 7fbedc722740 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13a9eeb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbec40b9550 tx=0x7fbec40bb090 comp rx=0 tx=0).stop 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- 172.21.15.116:0/1769199789 shutdown_connections 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa90c0 0x55cb13aad690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3f50 0x55cb13aad150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 --2- 172.21.15.116:0/1769199789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa3560 0x55cb13a9eeb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- 172.21.15.116:0/1769199789 >> 172.21.15.116:0/1769199789 conn(0x55cb13a9ead0 msgr2=0x55cb13aa8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- 172.21.15.116:0/1769199789 shutdown_connections 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- 172.21.15.116:0/1769199789 wait complete. 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- start start 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 Processor -- start 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3560 0x55cb13b1d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13b1fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 0x55cb13b22290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cb13b22990 con 0x55cb13aa90c0 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cb13b22b00 con 0x55cb13aa3560 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cb13b22cc0 con 0x55cb13aa3f50 2024-03-24T09:19:45.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbedc722740 10 client.0 _ll_register_callbacks cb 0x7fbec40b8f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbed3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 0x55cb13b22290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbed3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 0x55cb13b22290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41294/0 (socket says 172.21.15.116:41294) 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbed3fff640 1 -- 172.21.15.116:0/559741561 learned_addr learned my addr 172.21.15.116:0/559741561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbed37fe640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3560 0x55cb13b1d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.468+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13b1fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed3fff640 1 -- 172.21.15.116:0/559741561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 msgr2=0x55cb13b1fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed3fff640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13b1fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed3fff640 1 -- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3560 msgr2=0x55cb13b1d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed3fff640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3560 0x55cb13b1d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed3fff640 1 -- 172.21.15.116:0/559741561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb13a09510 con 0x55cb13aa90c0 2024-03-24T09:19:45.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed2ffd640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13b1fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed37fe640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3560 0x55cb13b1d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbed3fff640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 0x55cb13b22290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbecc019440 tx=0x7fbecc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c632d3d2cd194e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbeb2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/559741561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc01e040 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/559741561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fbecc024400 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbec80044e0 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/559741561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fbecc010070 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 10 client.14775 fetch_fsmap learned FSMap version 17 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55cb139cb7e0 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/559741561 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbec80044e0 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/559741561 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fbecc021030 con 0x55cb13aa90c0 2024-03-24T09:19:45.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 10 client.14775 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 10 client.14775 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 20 client.14775 populate_metadata read hostname 'smithi116' 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13b28580 0x55cb13b27290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 4 client.14775 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.469+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "status"}) -- 0x55cb13747610 con 0x55cb13b28580 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbed37fe640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13b28580 0x55cb13b27290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbed37fe640 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13b28580 0x55cb13b27290 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67bb059caff06762 server_cookie=ef3fde6b574b151f in_seq=0 out_seq=0 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbeb2ffd640 10 client.14775 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/559741561 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55cb13747610 con 0x55cb13b28580 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbeb2ffd640 10 client.14775 handle_command_reply: tid=0 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 client.14775 shutdown 2024-03-24T09:19:45.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 msgr2=0x55cb13b22290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 0x55cb13b22290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbecc019440 tx=0x7fbecc013000 comp rx=0 tx=0).stop 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 shutdown_connections 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cb13aa90c0 0x55cb13b22290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cb13aa3f50 0x55cb13b1fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55cb13b28580 0x55cb13b27290 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 --2- 172.21.15.116:0/559741561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cb13aa3560 0x55cb13b1d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbeb2ffd640 0 client.14775 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 >> 172.21.15.116:0/559741561 conn(0x55cb13a9ead0 msgr2=0x55cb13aa51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 shutdown_connections 2024-03-24T09:19:45.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 1 -- 172.21.15.116:0/559741561 wait complete. 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.470+0000 7fbedc722740 20 client.14775 trim_cache size 0 max 16384 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24515, 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:19:45.482 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:19:45.483 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 6.3718477470000003, 2024-03-24T09:19:45.483 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 17, 2024-03-24T09:19:45.483 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 89, 2024-03-24T09:19:45.483 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 88, 2024-03-24T09:19:45.483 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 12.512972896999999 2024-03-24T09:19:45.483 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:19:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbebc0a20c0 msgr2=0x7fbebc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbebc0a20c0 0x7fbebc0a4580 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009f70 tx=0x7fbecc009650 comp rx=0 tx=0).stop 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 msgr2=0x7fbed4197770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4197770 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbec402c7d0 tx=0x7fbec40042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 shutdown_connections 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbed41021c0 0x7fbed419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbebc0a20c0 0x7fbebc0a4580 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbed4101800 0x7fbed4197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 --2- 172.21.15.116:0/4049505866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbed4100610 0x7fbed4197770 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 >> 172.21.15.116:0/4049505866 conn(0x7fbed40fbb80 msgr2=0x7fbed4069ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 shutdown_connections 2024-03-24T09:19:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:45.471+0000 7fbed9e9d640 1 -- 172.21.15.116:0/4049505866 wait complete. 2024-03-24T09:19:45.486 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T09:19:45.486 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:45.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T09:19:45.720 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:19:46.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:45 smithi143 ceph-mon[35965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:19:46.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:45 smithi143 ceph-mon[35965]: Cluster is now healthy 2024-03-24T09:19:46.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:45 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/900564955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:46.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:45 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1027757377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:46.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:45 smithi117 ceph-mon[35788]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:19:46.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:45 smithi117 ceph-mon[35788]: Cluster is now healthy 2024-03-24T09:19:46.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:45 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/900564955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:46.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:45 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1027757377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:46.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:45 smithi116 ceph-mon[32207]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T09:19:46.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:45 smithi116 ceph-mon[32207]: Cluster is now healthy 2024-03-24T09:19:46.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:45 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/900564955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:19:46.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:45 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1027757377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:47.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:46 smithi143 ceph-mon[35965]: pgmap v295: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 786 B/s wr, 2 op/s 2024-03-24T09:19:47.038 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:46 smithi116 ceph-mon[32207]: pgmap v295: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 786 B/s wr, 2 op/s 2024-03-24T09:19:47.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:46 smithi117 ceph-mon[35788]: pgmap v295: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 786 B/s wr, 2 op/s 2024-03-24T09:19:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3601040026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a9806aea0 msgr2=0x7f4a9806b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3601040026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a9806aea0 0x7f4a9806b320 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4a88009990 tx=0x7f4a88009ca0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3601040026 shutdown_connections 2024-03-24T09:19:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3601040026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a98104c70 0x7f4a981050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3601040026 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a98107160 0x7f4a9806a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3601040026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a9806aea0 0x7f4a9806b320 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3601040026 >> 172.21.15.116:0/3601040026 conn(0x7f4a98078200 msgr2=0x7f4a98078610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3601040026 shutdown_connections 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3601040026 wait complete. 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 Processor -- start 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.465+0000 7f4a9f4d4700 1 -- start start 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a9806aea0 0x7f4a981a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a98104c70 0x7f4a981a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 0x7f4a9819ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9819f080 con 0x7f4a98107160 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9819f1c0 con 0x7f4a9806aea0 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9819f380 con 0x7f4a98104c70 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 0x7f4a9819ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 0x7f4a9819ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56980/0 (socket says 172.21.15.116:56980) 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 -- 172.21.15.116:0/3900098448 learned_addr learned my addr 172.21.15.116:0/3900098448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9d270700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a9806aea0 0x7f4a981a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9ca6f700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a98104c70 0x7f4a981a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 -- 172.21.15.116:0/3900098448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a98104c70 msgr2=0x7f4a981a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a98104c70 0x7f4a981a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 -- 172.21.15.116:0/3900098448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a9806aea0 msgr2=0x7f4a981a4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a9806aea0 0x7f4a981a4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 -- 172.21.15.116:0/3900098448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a880095f0 con 0x7f4a98107160 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9da71700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 0x7f4a9819ea20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000ef30 tx=0x7f4a9000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90018070 con 0x7f4a98107160 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a9000f040 con 0x7f4a98107160 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9819f600 con 0x7f4a98107160 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000cdd0 con 0x7f4a98107160 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.466+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a981ab8b0 con 0x7f4a98107160 2024-03-24T09:19:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.467+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a98068ec0 con 0x7f4a98107160 2024-03-24T09:19:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.468+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f4a90003a00 con 0x7f4a98107160 2024-03-24T09:19:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.469+0000 7f4a8e7fc700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a840a21d0 0x7f4a840a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.469+0000 7f4a9d270700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a840a21d0 0x7f4a840a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.469+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4a900cd980 con 0x7f4a98107160 2024-03-24T09:19:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.469+0000 7f4a9d270700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a840a21d0 0x7f4a840a4690 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4a94005950 tx=0x7f4a9400a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.473+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a900967a0 con 0x7f4a98107160 2024-03-24T09:19:47.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.710+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4a98118760 con 0x7f4a98107160 2024-03-24T09:19:47.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.711+0000 7f4a8e7fc700 1 -- 172.21.15.116:0/3900098448 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f4a90099e60 con 0x7f4a98107160 2024-03-24T09:19:47.712 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a840a21d0 msgr2=0x7f4a840a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a840a21d0 0x7f4a840a4690 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4a94005950 tx=0x7f4a9400a300 comp rx=0 tx=0).stop 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 msgr2=0x7f4a9819ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 0x7f4a9819ea20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000ef30 tx=0x7f4a9000c3a0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 shutdown_connections 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a840a21d0 0x7f4a840a4690 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a98107160 0x7f4a9819ea20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a9806aea0 0x7f4a981a4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 --2- 172.21.15.116:0/3900098448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a98104c70 0x7f4a981a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 >> 172.21.15.116:0/3900098448 conn(0x7f4a98078200 msgr2=0x7f4a98103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 shutdown_connections 2024-03-24T09:19:47.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:47.713+0000 7f4a9f4d4700 1 -- 172.21.15.116:0/3900098448 wait complete. 2024-03-24T09:19:48.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:47 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3900098448' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:19:48.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:47 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3900098448' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:19:48.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:47 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3900098448' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:19:48.395 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T09:19:48.396 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T09:19:48.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:48.397 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-24T09:19:48.413 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T09:19:48.413 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T09:19:48.413 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T09:19:48.413 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:19:48.413 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:19:48.414 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T09:19:48.415 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:19:48.415 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T09:19:48.415 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42287sec preferred_lft 42287sec 2024-03-24T09:19:48.415 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-24T09:19:48.415 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:19:48.416 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T09:19:48.416 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:19:48.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:19:48.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:48.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:48.613 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T09:19:48.641 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-24T09:19:48.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:48.642 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-24T09:19:48.713 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-24T09:19:48.713 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T09:19:48.713 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T09:19:48.713 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T09:19:48.713 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T09:19:48.714 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:19:48.714 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:19:48.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T09:19:48.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T09:19:48.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T09:19:48.714 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:19:48.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:19:48.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:48 smithi116 ceph-mon[32207]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 495 B/s wr, 1 op/s 2024-03-24T09:19:49.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:49.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:49.024 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-24T09:19:49.025 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:48 smithi143 ceph-mon[35965]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 495 B/s wr, 1 op/s 2024-03-24T09:19:49.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:49.040 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-24T09:19:49.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:48 smithi117 ceph-mon[35788]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 495 B/s wr, 1 op/s 2024-03-24T09:19:49.095 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:19:49.095 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:19:49.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T09:19:49.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-24T09:19:49.095 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:19:49.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:19:49.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:49.218 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-24T09:19:49.218 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:19:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:19:49.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:49.358 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:19:49.358 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:19:49.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-24T09:19:49.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T09:19:49.358 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:19:49.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:19:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:49.440 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-24T09:19:49.440 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:19:49.440 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:49.455 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-24T09:19:49.455 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:19:49.455 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:49.510 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-24T09:19:49.578 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T09:19:49.635 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-24T09:19:49.635 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-24T09:19:49.635 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-24T09:19:49.635 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:19:49.636 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-24T09:19:49.637 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/35f41a9dbcebef80300000130596a6c83d16e0d20a4f3c64f6fcf65c3551145b/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/c7427c083e1aa70ee5719eaef49d1706a13be3aa707c6f38e2bc752286142654/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3a42c195b5c65f2cf516ac08f6f0b6dd04b6aa345e3c1e24cdb74b8899e2c4b9/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6fb648b372ea483bdf49220fd699c479c8a251e10ca97e038d3933433e391567/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/06636e80d376544c2d95157acfaadee5bf17824d0b5b2de4bc9cff1e3ca7909c/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/4634c5834fbaa5a6ca4762eb78372a6622ed6cb6dad36ef2eccc759d01803633/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/f9cbea23acb91d910aab346dbe893b7881e8e5a4d632350f71d0763c94c3fb8d/merged 2024-03-24T09:19:49.638 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/bdda17d67e3e115dae972830f914e98b56622b46d8eb7d7e9d9bc4b670fb8a97/merged 2024-03-24T09:19:49.639 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d3b738dc89f8fedf5f42454e24f014603e5095215d346fcdcb3ab94444177c21/merged 2024-03-24T09:19:49.639 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/75c4d7635463175883f7e9c2a30a7bb3f3b6fe121a2c13484d27003bc55fbf42/merged 2024-03-24T09:19:49.639 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-24T09:19:49.639 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T09:19:49.639 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T09:19:49.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:49.639 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-24T09:19:49.690 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T09:19:49.690 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-24T09:19:49.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-24T09:19:49.760 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T09:19:49.816 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-24T09:19:49.816 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-24T09:19:49.816 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-24T09:19:49.816 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-24T09:19:49.816 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-24T09:19:49.817 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-24T09:19:49.818 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/35f41a9dbcebef80300000130596a6c83d16e0d20a4f3c64f6fcf65c3551145b/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/c7427c083e1aa70ee5719eaef49d1706a13be3aa707c6f38e2bc752286142654/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3a42c195b5c65f2cf516ac08f6f0b6dd04b6aa345e3c1e24cdb74b8899e2c4b9/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6fb648b372ea483bdf49220fd699c479c8a251e10ca97e038d3933433e391567/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/06636e80d376544c2d95157acfaadee5bf17824d0b5b2de4bc9cff1e3ca7909c/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/4634c5834fbaa5a6ca4762eb78372a6622ed6cb6dad36ef2eccc759d01803633/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/f9cbea23acb91d910aab346dbe893b7881e8e5a4d632350f71d0763c94c3fb8d/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/bdda17d67e3e115dae972830f914e98b56622b46d8eb7d7e9d9bc4b670fb8a97/merged 2024-03-24T09:19:49.819 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d3b738dc89f8fedf5f42454e24f014603e5095215d346fcdcb3ab94444177c21/merged 2024-03-24T09:19:49.820 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/75c4d7635463175883f7e9c2a30a7bb3f3b6fe121a2c13484d27003bc55fbf42/merged 2024-03-24T09:19:49.820 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-24T09:19:49.820 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T09:19:49.820 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T09:19:49.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:49.820 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-24T09:19:49.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:2024-03-24T09:19:49.828+0000 7f09b4652480 -1 init, newargv = 0x55944f9c8710 newargc=15 2024-03-24T09:19:49.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[76252]: starting ceph client 2024-03-24T09:19:49.834 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[76252]: starting fuse 2024-03-24T09:19:49.844 INFO:teuthology.orchestra.run.smithi116.stdout:119 2024-03-24T09:19:49.844 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-24T09:19:49.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:19:49.845 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-03-24T09:19:49.846 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T09:19:49.847 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:19:49.847 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.admin")) 2024-03-24T09:19:49.847 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-03-24T09:19:49.966 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.admin.76252.asok 2024-03-24T09:19:49.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:49.977 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76252.asok 2024-03-24T09:19:49.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:49.977 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76252.asok status 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "admin", 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-03-24T09:19:50.114 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "76252", 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 14787, 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-03-24T09:19:50.115 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 14787 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2725550054 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-24T09:19:50.116 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2725550054 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.14787 192.168.0.1:0/2725550054", 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/2725550054", 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 17, 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 89, 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2024-03-24T09:19:50.117 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:19:50.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:50.123 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-03-24T09:19:50.123 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:50.138 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-03-24T09:19:50.138 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:50.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:50.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:50.207 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-24T09:19:50.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-24T09:19:50.322 INFO:teuthology.orchestra.run.smithi116.stdout:# file: . 2024-03-24T09:19:50.322 INFO:teuthology.orchestra.run.smithi116.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-24T09:19:50.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:50.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:19:50.329 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:19:50.329 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:19:50.345 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:19:50.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- 172.21.15.116:0/96186975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 msgr2=0x7f4e4c100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- 172.21.15.116:0/96186975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c100a60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4e44009900 tx=0x7f4e4402bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- 172.21.15.116:0/96186975 shutdown_connections 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- 172.21.15.116:0/96186975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e4c1021f0 0x7f4e4c108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- 172.21.15.116:0/96186975 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e4c101830 0x7f4e4c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- 172.21.15.116:0/96186975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c100a60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- 172.21.15.116:0/96186975 >> 172.21.15.116:0/96186975 conn(0x7f4e4c0fbbf0 msgr2=0x7f4e4c0fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- 172.21.15.116:0/96186975 shutdown_connections 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- 172.21.15.116:0/96186975 wait complete. 2024-03-24T09:19:50.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 Processor -- start 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- start start 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e4c101830 0x7f4e4c197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e4c1021f0 0x7f4e4c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c19f290 con 0x7f4e4c100640 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c19f400 con 0x7f4e4c1021f0 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e5381c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4c19f5c0 con 0x7f4e4c101830 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e515b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e515b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56998/0 (socket says 172.21.15.116:56998) 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.487+0000 7f4e515b8640 1 -- 172.21.15.116:0/3756335377 learned_addr learned my addr 172.21.15.116:0/3756335377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:50.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e51db9640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e4c1021f0 0x7f4e4c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e50db7640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e4c101830 0x7f4e4c197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e515b8640 1 -- 172.21.15.116:0/3756335377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e4c101830 msgr2=0x7f4e4c197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e515b8640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e4c101830 0x7f4e4c197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e515b8640 1 -- 172.21.15.116:0/3756335377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e4c1021f0 msgr2=0x7f4e4c19ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e515b8640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e4c1021f0 0x7f4e4c19ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e515b8640 1 -- 172.21.15.116:0/3756335377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e44009560 con 0x7f4e4c100640 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e515b8640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c1977a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402c7d0 tx=0x7f4e440042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44004aa0 con 0x7f4e4c100640 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e4402de90 con 0x7f4e4c100640 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e4c19f840 con 0x7f4e4c100640 2024-03-24T09:19:50.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44005440 con 0x7f4e4c100640 2024-03-24T09:19:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.488+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e4c19fdd0 con 0x7f4e4c100640 2024-03-24T09:19:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.489+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e4c107a30 con 0x7f4e4c100640 2024-03-24T09:19:50.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.490+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f4e440055a0 con 0x7f4e4c100640 2024-03-24T09:19:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.491+0000 7f4e427fc640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4e380a20c0 0x7f4e380a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.491+0000 7f4e50db7640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4e380a20c0 0x7f4e380a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.491+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4e440f0650 con 0x7f4e4c100640 2024-03-24T09:19:50.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.491+0000 7f4e50db7640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4e380a20c0 0x7f4e380a4580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c198cb0 tx=0x7f4e3c008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:50.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.495+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4e440b93e0 con 0x7f4e4c100640 2024-03-24T09:19:50.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.669+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e4c02d160 con 0x7f4e4c100640 2024-03-24T09:19:50.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.671+0000 7f4e427fc640 1 -- 172.21.15.116:0/3756335377 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7f4e440bcaa0 con 0x7f4e4c100640 2024-03-24T09:19:50.672 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:50.672 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4e380a20c0 msgr2=0x7f4e380a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4e380a20c0 0x7f4e380a4580 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c198cb0 tx=0x7f4e3c008040 comp rx=0 tx=0).stop 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 msgr2=0x7f4e4c1977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c1977a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4e4402c7d0 tx=0x7f4e440042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 shutdown_connections 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e4c1021f0 0x7f4e4c19ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4e380a20c0 0x7f4e380a4580 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e4c101830 0x7f4e4c197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 --2- 172.21.15.116:0/3756335377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e4c100640 0x7f4e4c1977a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 >> 172.21.15.116:0/3756335377 conn(0x7f4e4c0fbbf0 msgr2=0x7f4e4c104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 shutdown_connections 2024-03-24T09:19:50.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.673+0000 7f4e5381c640 1 -- 172.21.15.116:0/3756335377 wait complete. 2024-03-24T09:19:50.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:19:50.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:19:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 -- 172.21.15.116:0/2763747498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c20101710 msgr2=0x7f9c20101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2763747498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c20101710 0x7f9c20101b30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009860 tx=0x7f9c1c02bb70 comp rx=0 tx=0).stop 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 -- 172.21.15.116:0/2763747498 shutdown_connections 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2763747498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c201032c0 0x7f9c20109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2763747498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20102900 0x7f9c20102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2763747498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c20101710 0x7f9c20101b30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 -- 172.21.15.116:0/2763747498 >> 172.21.15.116:0/2763747498 conn(0x7f9c200fcc80 msgr2=0x7f9c200ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 -- 172.21.15.116:0/2763747498 shutdown_connections 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.791+0000 7f9c280e8640 1 -- 172.21.15.116:0/2763747498 wait complete. 2024-03-24T09:19:50.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 Processor -- start 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 -- start start 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 0x7f9c20197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20102900 0x7f9c20197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c201032c0 0x7f9c2019ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2019f220 con 0x7f9c20101710 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2019f390 con 0x7f9c201032c0 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c280e8640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2019f550 con 0x7f9c20102900 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 0x7f9c20197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 0x7f9c20197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57010/0 (socket says 172.21.15.116:57010) 2024-03-24T09:19:50.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 -- 172.21.15.116:0/2051949305 learned_addr learned my addr 172.21.15.116:0/2051949305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c26685640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c201032c0 0x7f9c2019ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 -- 172.21.15.116:0/2051949305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20102900 msgr2=0x7f9c20197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25683640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20102900 0x7f9c20197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20102900 0x7f9c20197c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 -- 172.21.15.116:0/2051949305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c201032c0 msgr2=0x7f9c2019ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c201032c0 0x7f9c2019ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.792+0000 7f9c25e84640 1 -- 172.21.15.116:0/2051949305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c18009900 con 0x7f9c20101710 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.793+0000 7f9c25e84640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 0x7f9c20197730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c02c0b0 tx=0x7f9c1c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:50.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.793+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c02c920 con 0x7f9c20101710 2024-03-24T09:19:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.793+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1c009490 con 0x7f9c20101710 2024-03-24T09:19:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.793+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9c1c02ce90 con 0x7f9c20101710 2024-03-24T09:19:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.793+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c032600 con 0x7f9c20101710 2024-03-24T09:19:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.793+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c2019faf0 con 0x7f9c20101710 2024-03-24T09:19:50.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.794+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c20108b00 con 0x7f9c20101710 2024-03-24T09:19:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.795+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f9c1c03b040 con 0x7f9c20101710 2024-03-24T09:19:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.796+0000 7f9c16ffd640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9c0c0a20c0 0x7f9c0c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.796+0000 7f9c25683640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9c0c0a20c0 0x7f9c0c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.796+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9c1c0efae0 con 0x7f9c20101710 2024-03-24T09:19:50.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.796+0000 7f9c25683640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9c0c0a20c0 0x7f9c0c0a4580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9c10005fd0 tx=0x7f9c10005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:50.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.800+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9c1c0b8870 con 0x7f9c20101710 2024-03-24T09:19:50.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.974+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9c2002d160 con 0x7f9c20101710 2024-03-24T09:19:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.976+0000 7f9c16ffd640 1 -- 172.21.15.116:0/2051949305 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f9c1c0bbf30 con 0x7f9c20101710 2024-03-24T09:19:50.977 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:19:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9c0c0a20c0 msgr2=0x7f9c0c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9c0c0a20c0 0x7f9c0c0a4580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9c10005fd0 tx=0x7f9c10005850 comp rx=0 tx=0).stop 2024-03-24T09:19:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 msgr2=0x7f9c20197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 0x7f9c20197730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c02c0b0 tx=0x7f9c1c0042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 shutdown_connections 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c201032c0 0x7f9c2019ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9c0c0a20c0 0x7f9c0c0a4580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9c20102900 0x7f9c20197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 --2- 172.21.15.116:0/2051949305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c20101710 0x7f9c20197730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 >> 172.21.15.116:0/2051949305 conn(0x7f9c200fcc80 msgr2=0x7f9c20105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 shutdown_connections 2024-03-24T09:19:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:50.978+0000 7f9c280e8640 1 -- 172.21.15.116:0/2051949305 wait complete. 2024-03-24T09:19:50.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:19:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 -- 172.21.15.116:0/1096513671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de0101800 msgr2=0x7f7de0101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 --2- 172.21.15.116:0/1096513671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de0101800 0x7f7de0101c80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009900 tx=0x7f7dcc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 -- 172.21.15.116:0/1096513671 shutdown_connections 2024-03-24T09:19:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 --2- 172.21.15.116:0/1096513671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de01021c0 0x7f7de0108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 --2- 172.21.15.116:0/1096513671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de0101800 0x7f7de0101c80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 --2- 172.21.15.116:0/1096513671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de0100610 0x7f7de0100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 -- 172.21.15.116:0/1096513671 >> 172.21.15.116:0/1096513671 conn(0x7f7de00fbba0 msgr2=0x7f7de00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 -- 172.21.15.116:0/1096513671 shutdown_connections 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 -- 172.21.15.116:0/1096513671 wait complete. 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 Processor -- start 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.094+0000 7f7de70f4640 1 -- start start 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de70f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de0100610 0x7f7de0196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de70f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de0101800 0x7f7de0196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de70f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 0x7f7de019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de70f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7de019e0f0 con 0x7f7de01021c0 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de70f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7de019e260 con 0x7f7de0101800 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de70f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7de019e420 con 0x7f7de0100610 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 0x7f7de019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 0x7f7de019dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57024/0 (socket says 172.21.15.116:57024) 2024-03-24T09:19:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 -- 172.21.15.116:0/186278732 learned_addr learned my addr 172.21.15.116:0/186278732 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7ddffff640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de0101800 0x7f7de0196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de4e90640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de0100610 0x7f7de0196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 -- 172.21.15.116:0/186278732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de0100610 msgr2=0x7f7de0196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de0100610 0x7f7de0196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 -- 172.21.15.116:0/186278732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de0101800 msgr2=0x7f7de0196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de0101800 0x7f7de0196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 -- 172.21.15.116:0/186278732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd4009490 con 0x7f7de01021c0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de4e90640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de0100610 0x7f7de0196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7de5691640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 0x7f7de019dbb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800cad0 tx=0x7f7dd800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.095+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd800ae20 con 0x7f7de01021c0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.096+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7dd8005020 con 0x7f7de01021c0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.096+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dcc009560 con 0x7f7de01021c0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.096+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd800f690 con 0x7f7de01021c0 2024-03-24T09:19:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.096+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7de019ea50 con 0x7f7de01021c0 2024-03-24T09:19:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.097+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7de0107a50 con 0x7f7de01021c0 2024-03-24T09:19:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.098+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f7dd8005190 con 0x7f7de01021c0 2024-03-24T09:19:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.098+0000 7f7dddffb640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7dd00a20c0 0x7f7dd00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.098+0000 7f7de4e90640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7dd00a20c0 0x7f7dd00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.098+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7dd80cdf00 con 0x7f7de01021c0 2024-03-24T09:19:51.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.099+0000 7f7de4e90640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7dd00a20c0 0x7f7dd00a4580 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7de0101660 tx=0x7f7dd4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.103+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7dd8096c90 con 0x7f7de01021c0 2024-03-24T09:19:51.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:50 smithi143 ceph-mon[35965]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 1 op/s 2024-03-24T09:19:51.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:50 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3756335377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.274+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7de002d1d0 con 0x7f7de01021c0 2024-03-24T09:19:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.274+0000 7f7dddffb640 1 -- 172.21.15.116:0/186278732 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f7dd809a350 con 0x7f7de01021c0 2024-03-24T09:19:51.275 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:19:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7dd00a20c0 msgr2=0x7f7dd00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7dd00a20c0 0x7f7dd00a4580 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7de0101660 tx=0x7f7dd4005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 msgr2=0x7f7de019dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 0x7f7de019dbb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7dd800cad0 tx=0x7f7dd800cf90 comp rx=0 tx=0).stop 2024-03-24T09:19:51.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 shutdown_connections 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7de01021c0 0x7f7de019dbb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7de0101800 0x7f7de0196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7dd00a20c0 0x7f7dd00a4580 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 --2- 172.21.15.116:0/186278732 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7de0100610 0x7f7de0196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.276+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 >> 172.21.15.116:0/186278732 conn(0x7f7de00fbba0 msgr2=0x7f7de0104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.277+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 shutdown_connections 2024-03-24T09:19:51.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.277+0000 7f7de70f4640 1 -- 172.21.15.116:0/186278732 wait complete. 2024-03-24T09:19:51.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:50 smithi116 ceph-mon[32207]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 1 op/s 2024-03-24T09:19:51.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:50 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3756335377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:51.288 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:19:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:50 smithi117 ceph-mon[35788]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s wr, 1 op/s 2024-03-24T09:19:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:50 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3756335377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:51.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 -- 172.21.15.116:0/3358537787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1201032c0 msgr2=0x7fe120109b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 --2- 172.21.15.116:0/3358537787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1201032c0 0x7fe120109b50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe118009ca0 tx=0x7fe11802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 -- 172.21.15.116:0/3358537787 shutdown_connections 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 --2- 172.21.15.116:0/3358537787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1201032c0 0x7fe120109b50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 --2- 172.21.15.116:0/3358537787 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe120102900 0x7fe120102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 --2- 172.21.15.116:0/3358537787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe120101710 0x7fe120101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 -- 172.21.15.116:0/3358537787 >> 172.21.15.116:0/3358537787 conn(0x7fe1200fcc80 msgr2=0x7fe1200ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 -- 172.21.15.116:0/3358537787 shutdown_connections 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.391+0000 7fe12708f640 1 -- 172.21.15.116:0/3358537787 wait complete. 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 Processor -- start 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 -- start start 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 0x7fe120197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe120102900 0x7fe120197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1201032c0 0x7fe12019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe12019f180 con 0x7fe120101710 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe12019f2f0 con 0x7fe120102900 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12708f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe12019f4b0 con 0x7fe1201032c0 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 0x7fe120197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 0x7fe120197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57034/0 (socket says 172.21.15.116:57034) 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 -- 172.21.15.116:0/844608479 learned_addr learned my addr 172.21.15.116:0/844608479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe11ffff640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe120102900 0x7fe120197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12562c640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1201032c0 0x7fe12019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 -- 172.21.15.116:0/844608479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1201032c0 msgr2=0x7fe12019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1201032c0 0x7fe12019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 -- 172.21.15.116:0/844608479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe120102900 msgr2=0x7fe120197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe120102900 0x7fe120197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 -- 172.21.15.116:0/844608479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe118009900 con 0x7fe120101710 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe12562c640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1201032c0 0x7fe12019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe124e2b640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 0x7fe120197690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe11400e9c0 tx=0x7fe11400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.392+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1140096e0 con 0x7fe120101710 2024-03-24T09:19:51.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.393+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe1140044e0 con 0x7fe120101710 2024-03-24T09:19:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.393+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114010430 con 0x7fe120101710 2024-03-24T09:19:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.393+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12019f790 con 0x7fe120101710 2024-03-24T09:19:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.393+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe12019fc50 con 0x7fe120101710 2024-03-24T09:19:51.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.394+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe120108ab0 con 0x7fe120101710 2024-03-24T09:19:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.395+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fe114003f80 con 0x7fe120101710 2024-03-24T09:19:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.395+0000 7fe11dffb640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe1100a20c0 0x7fe1100a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.395+0000 7fe11ffff640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe1100a20c0 0x7fe1100a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.396+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe114014070 con 0x7fe120101710 2024-03-24T09:19:51.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.396+0000 7fe11ffff640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe1100a20c0 0x7fe1100a4580 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe120198ba0 tx=0x7fe10c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.400+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe1140970a0 con 0x7fe120101710 2024-03-24T09:19:51.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.572+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe12010cff0 con 0x7fe120101710 2024-03-24T09:19:51.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.573+0000 7fe11dffb640 1 -- 172.21.15.116:0/844608479 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fe11409a760 con 0x7fe120101710 2024-03-24T09:19:51.573 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:19:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe1100a20c0 msgr2=0x7fe1100a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe1100a20c0 0x7fe1100a4580 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe120198ba0 tx=0x7fe10c00b390 comp rx=0 tx=0).stop 2024-03-24T09:19:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 msgr2=0x7fe120197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 0x7fe120197690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe11400e9c0 tx=0x7fe11400ee80 comp rx=0 tx=0).stop 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 shutdown_connections 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe1100a20c0 0x7fe1100a4580 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe1201032c0 0x7fe12019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe120102900 0x7fe120197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 --2- 172.21.15.116:0/844608479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe120101710 0x7fe120197690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 >> 172.21.15.116:0/844608479 conn(0x7fe1200fcc80 msgr2=0x7fe120105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 shutdown_connections 2024-03-24T09:19:51.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.575+0000 7fe12708f640 1 -- 172.21.15.116:0/844608479 wait complete. 2024-03-24T09:19:51.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 -- 172.21.15.116:0/1048938067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f0101820 msgr2=0x7fb1f0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/1048938067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f0101820 0x7fb1f0101ca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009860 tx=0x7fb1e002bb70 comp rx=0 tx=0).stop 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 -- 172.21.15.116:0/1048938067 shutdown_connections 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/1048938067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f01021e0 0x7fb1f0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/1048938067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f0101820 0x7fb1f0101ca0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/1048938067 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f0100630 0x7fb1f0100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 -- 172.21.15.116:0/1048938067 >> 172.21.15.116:0/1048938067 conn(0x7fb1f00fbbc0 msgr2=0x7fb1f00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 -- 172.21.15.116:0/1048938067 shutdown_connections 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.690+0000 7fb1f6b01640 1 -- 172.21.15.116:0/1048938067 wait complete. 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 Processor -- start 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 -- start start 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 0x7fb1f0196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f0101820 0x7fb1f0196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f01021e0 0x7fb1f019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f019e050 con 0x7fb1f0100630 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f019e1c0 con 0x7fb1f01021e0 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f6b01640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f019e380 con 0x7fb1f0101820 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 0x7fb1f0196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 0x7fb1f0196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57048/0 (socket says 172.21.15.116:57048) 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 -- 172.21.15.116:0/3220572299 learned_addr learned my addr 172.21.15.116:0/3220572299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f509e640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f01021e0 0x7fb1f019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1effff640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f0101820 0x7fb1f0196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 -- 172.21.15.116:0/3220572299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f0101820 msgr2=0x7fb1f0196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f0101820 0x7fb1f0196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 -- 172.21.15.116:0/3220572299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f01021e0 msgr2=0x7fb1f019db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f01021e0 0x7fb1f019db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.691+0000 7fb1f489d640 1 -- 172.21.15.116:0/3220572299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d8009560 con 0x7fb1f0100630 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.692+0000 7fb1f489d640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 0x7fb1f0196560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800ccb0 tx=0x7fb1e800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.692+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8003b70 con 0x7fb1f0100630 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.692+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e0009490 con 0x7fb1f0100630 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.692+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1e800ae50 con 0x7fb1f0100630 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.692+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e800f6b0 con 0x7fb1f0100630 2024-03-24T09:19:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.692+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f019ea30 con 0x7fb1f0100630 2024-03-24T09:19:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.693+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f0107970 con 0x7fb1f0100630 2024-03-24T09:19:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.694+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fb1e8004000 con 0x7fb1f0100630 2024-03-24T09:19:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.694+0000 7fb1edffb640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb1dc0a20c0 0x7fb1dc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.695+0000 7fb1effff640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb1dc0a20c0 0x7fb1dc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.695+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb1e80ce380 con 0x7fb1f0100630 2024-03-24T09:19:51.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.695+0000 7fb1effff640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb1dc0a20c0 0x7fb1dc0a4580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0037040 tx=0x7fb1e0034ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.699+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb1e8097110 con 0x7fb1f0100630 2024-03-24T09:19:51.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.873+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb1f002d160 con 0x7fb1f0100630 2024-03-24T09:19:51.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.874+0000 7fb1edffb640 1 -- 172.21.15.116:0/3220572299 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fb1e809a7d0 con 0x7fb1f0100630 2024-03-24T09:19:51.874 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:19:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.875+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb1dc0a20c0 msgr2=0x7fb1dc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.875+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb1dc0a20c0 0x7fb1dc0a4580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0037040 tx=0x7fb1e0034ab0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.875+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 msgr2=0x7fb1f0196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.875+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 0x7fb1f0196560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb1e800ccb0 tx=0x7fb1e800a620 comp rx=0 tx=0).stop 2024-03-24T09:19:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.875+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 shutdown_connections 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.875+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1f01021e0 0x7fb1f019db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.876+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb1dc0a20c0 0x7fb1dc0a4580 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.876+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1f0101820 0x7fb1f0196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.876+0000 7fb1f6b01640 1 --2- 172.21.15.116:0/3220572299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1f0100630 0x7fb1f0196560 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.876+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 >> 172.21.15.116:0/3220572299 conn(0x7fb1f00fbbc0 msgr2=0x7fb1f0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.876+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 shutdown_connections 2024-03-24T09:19:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.876+0000 7fb1f6b01640 1 -- 172.21.15.116:0/3220572299 wait complete. 2024-03-24T09:19:51.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T09:19:51.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 -- 172.21.15.116:0/1431493176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 msgr2=0x7f50a0108a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 --2- 172.21.15.116:0/1431493176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a0108a60 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5098009ca0 tx=0x7f509802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 -- 172.21.15.116:0/1431493176 shutdown_connections 2024-03-24T09:19:51.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 --2- 172.21.15.116:0/1431493176 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a0108a60 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 --2- 172.21.15.116:0/1431493176 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50a0101810 0x7f50a0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 --2- 172.21.15.116:0/1431493176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50a0100620 0x7f50a0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 -- 172.21.15.116:0/1431493176 >> 172.21.15.116:0/1431493176 conn(0x7f50a00fbbb0 msgr2=0x7f50a00fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 -- 172.21.15.116:0/1431493176 shutdown_connections 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 -- 172.21.15.116:0/1431493176 wait complete. 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 Processor -- start 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.991+0000 7f50a7afb640 1 -- start start 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50a0100620 0x7f50a0196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50a0101810 0x7f50a0196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50a019e110 con 0x7f50a01021d0 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f50a019e280 con 0x7f50a0101810 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f50a019e440 con 0x7f50a0100620 2024-03-24T09:19:51.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a019dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57056/0 (socket says 172.21.15.116:57056) 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 -- 172.21.15.116:0/2442529806 learned_addr learned my addr 172.21.15.116:0/2442529806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a5096640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50a0101810 0x7f50a0196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 -- 172.21.15.116:0/2442529806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50a0100620 msgr2=0x7f50a0196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50a0100620 0x7f50a0196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 -- 172.21.15.116:0/2442529806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50a0101810 msgr2=0x7f50a0196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50a0101810 0x7f50a0196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 -- 172.21.15.116:0/2442529806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509c009490 con 0x7f50a01021d0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a6098640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a019dbd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5098037040 tx=0x7f5098032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098034c70 con 0x7f50a01021d0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5098003e20 con 0x7f50a01021d0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5098009900 con 0x7f50a01021d0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098004390 con 0x7f50a01021d0 2024-03-24T09:19:51.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.992+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a006a080 con 0x7f50a01021d0 2024-03-24T09:19:51.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.993+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50a006aa60 con 0x7f50a01021d0 2024-03-24T09:19:51.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.994+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f509803b070 con 0x7f50a01021d0 2024-03-24T09:19:51.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.995+0000 7f5096ffd640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f508c0a20c0 0x7f508c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:51.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.995+0000 7f50a5897640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f508c0a20c0 0x7f508c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:51.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.995+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f50980f23f0 con 0x7f50a01021d0 2024-03-24T09:19:51.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:51.995+0000 7f50a5897640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f508c0a20c0 0x7f508c0a4580 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f509c005d00 tx=0x7f509c005c50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.000+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f50980bb180 con 0x7f50a01021d0 2024-03-24T09:19:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.184+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f50a010c0d0 con 0x7f50a01021d0 2024-03-24T09:19:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.185+0000 7f5096ffd640 1 -- 172.21.15.116:0/2442529806 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+599 (secure 0 0 0) 0x7f5098039270 con 0x7f50a01021d0 2024-03-24T09:19:52.185 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3305720445 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:52.185 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3598626147 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3855909741 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/448304506 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3108228013 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:52.186 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2688022316 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f508c0a20c0 msgr2=0x7f508c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f508c0a20c0 0x7f508c0a4580 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f509c005d00 tx=0x7f509c005c50 comp rx=0 tx=0).stop 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 msgr2=0x7f50a019dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a019dbd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5098037040 tx=0x7f5098032980 comp rx=0 tx=0).stop 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 shutdown_connections 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50a01021d0 0x7f50a019dbd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50a0101810 0x7f50a0196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f508c0a20c0 0x7f508c0a4580 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 --2- 172.21.15.116:0/2442529806 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f50a0100620 0x7f50a0196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 >> 172.21.15.116:0/2442529806 conn(0x7f50a00fbbb0 msgr2=0x7f50a0104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 shutdown_connections 2024-03-24T09:19:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.187+0000 7f50a7afb640 1 -- 172.21.15.116:0/2442529806 wait complete. 2024-03-24T09:19:52.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:51 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2051949305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:51 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/186278732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:51 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/844608479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:51 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3220572299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.198 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:19:52.199 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:19:52.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:19:52.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:51 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2051949305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:51 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/186278732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:51 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/844608479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:51 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3220572299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:51 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2051949305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:51 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/186278732' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:51 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/844608479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:51 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3220572299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:19:52.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 -- 172.21.15.116:0/1232919491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 msgr2=0x7f88b8101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 --2- 172.21.15.116:0/1232919491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b8101ca0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f88a8009900 tx=0x7f88a802bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 -- 172.21.15.116:0/1232919491 shutdown_connections 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 --2- 172.21.15.116:0/1232919491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88b81021e0 0x7f88b8108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 --2- 172.21.15.116:0/1232919491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b8101ca0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 --2- 172.21.15.116:0/1232919491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88b8100630 0x7f88b8100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 -- 172.21.15.116:0/1232919491 >> 172.21.15.116:0/1232919491 conn(0x7f88b80fbba0 msgr2=0x7f88b80fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 -- 172.21.15.116:0/1232919491 shutdown_connections 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.355+0000 7f88becf4640 1 -- 172.21.15.116:0/1232919491 wait complete. 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 Processor -- start 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 -- start start 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88b8100630 0x7f88b80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b819c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88b81021e0 0x7f88b819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88b819f250 con 0x7f88b8101820 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88b819f3c0 con 0x7f88b81021e0 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88becf4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f88b819f580 con 0x7f88b8100630 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b819c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b819c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57064/0 (socket says 172.21.15.116:57064) 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 -- 172.21.15.116:0/2036574957 learned_addr learned my addr 172.21.15.116:0/2036574957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88bd291640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88b81021e0 0x7f88b819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 -- 172.21.15.116:0/2036574957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88b8100630 msgr2=0x7f88b80693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88bca90640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88b8100630 0x7f88b80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88b8100630 0x7f88b80693e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 -- 172.21.15.116:0/2036574957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88b81021e0 msgr2=0x7f88b819ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88b81021e0 0x7f88b819ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 -- 172.21.15.116:0/2036574957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a8009560 con 0x7f88b8101820 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.356+0000 7f88affff640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b819c450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88a8036040 tx=0x7f88a8004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.357+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a80035f0 con 0x7f88b8101820 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.357+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f88a802de90 con 0x7f88b8101820 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.357+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b819f7a0 con 0x7f88b8101820 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.357+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a8005490 con 0x7f88b8101820 2024-03-24T09:19:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.357+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b819fcb0 con 0x7f88b8101820 2024-03-24T09:19:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.358+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b8107970 con 0x7f88b8101820 2024-03-24T09:19:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.359+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f88a8003750 con 0x7f88b8101820 2024-03-24T09:19:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.359+0000 7f88adffb640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f88a00a20c0 0x7f88a00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.360+0000 7f88bca90640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f88a00a20c0 0x7f88a00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.360+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f88a80f08a0 con 0x7f88b8101820 2024-03-24T09:19:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.360+0000 7f88bca90640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f88a00a20c0 0x7f88a00a4580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f88b8101680 tx=0x7f88b4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.363+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f88a80b9630 con 0x7f88b8101820 2024-03-24T09:19:52.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.538+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88b810c140 con 0x7f88b8101820 2024-03-24T09:19:52.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.540+0000 7f88adffb640 1 -- 172.21.15.116:0/2036574957 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7f88a80bccf0 con 0x7f88b8101820 2024-03-24T09:19:52.540 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:19:52.541 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f88a00a20c0 msgr2=0x7f88a00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f88a00a20c0 0x7f88a00a4580 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f88b8101680 tx=0x7f88b4005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 msgr2=0x7f88b819c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b819c450 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f88a8036040 tx=0x7f88a8004140 comp rx=0 tx=0).stop 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 shutdown_connections 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88b81021e0 0x7f88b819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88b8101820 0x7f88b819c450 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f88a00a20c0 0x7f88a00a4580 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 --2- 172.21.15.116:0/2036574957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f88b8100630 0x7f88b80693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 >> 172.21.15.116:0/2036574957 conn(0x7f88b80fbba0 msgr2=0x7f88b8104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 shutdown_connections 2024-03-24T09:19:52.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.542+0000 7f88becf4640 1 -- 172.21.15.116:0/2036574957 wait complete. 2024-03-24T09:19:52.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:19:52.554 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2342331708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 msgr2=0x7fdaf0100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2342331708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0100a30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009900 tx=0x7fdae002bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2342331708 shutdown_connections 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2342331708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf01021c0 0x7fdaf0108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2342331708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf0101800 0x7fdaf0101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2342331708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0100a30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2342331708 >> 172.21.15.116:0/2342331708 conn(0x7fdaf00fbb80 msgr2=0x7fdaf00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2342331708 shutdown_connections 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.657+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2342331708 wait complete. 2024-03-24T09:19:52.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 Processor -- start 2024-03-24T09:19:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 -- start start 2024-03-24T09:19:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf0101800 0x7fdaf019c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf01021c0 0x7fdaf019eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf019f230 con 0x7fdaf0100610 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf019f3a0 con 0x7fdaf01021c0 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaf4cc3640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf019f560 con 0x7fdaf0101800 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0069400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57082/0 (socket says 172.21.15.116:57082) 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 -- 172.21.15.116:0/2957837803 learned_addr learned my addr 172.21.15.116:0/2957837803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaeed9d640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf01021c0 0x7fdaf019eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdae5d9b640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf0101800 0x7fdaf019c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 -- 172.21.15.116:0/2957837803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf0101800 msgr2=0x7fdaf019c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf0101800 0x7fdaf019c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 -- 172.21.15.116:0/2957837803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf01021c0 msgr2=0x7fdaf019eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf01021c0 0x7fdaf019eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.658+0000 7fdaee59c640 1 -- 172.21.15.116:0/2957837803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae8009840 con 0x7fdaf0100610 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.659+0000 7fdaee59c640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0069400 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdae002c240 tx=0x7fdae00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.659+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae00044e0 con 0x7fdaf0100610 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.659+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdae0004aa0 con 0x7fdaf0100610 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.659+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae0009560 con 0x7fdaf0100610 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.659+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae00054c0 con 0x7fdaf0100610 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.659+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf019fb60 con 0x7fdaf0100610 2024-03-24T09:19:52.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.660+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaf0107a00 con 0x7fdaf0100610 2024-03-24T09:19:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.661+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fdae0003800 con 0x7fdaf0100610 2024-03-24T09:19:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.661+0000 7fdae7fff640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdadc0a20c0 0x7fdadc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.661+0000 7fdae5d9b640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdadc0a20c0 0x7fdadc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.662+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdae002de40 con 0x7fdaf0100610 2024-03-24T09:19:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.662+0000 7fdae5d9b640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdadc0a20c0 0x7fdadc0a4580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0069170 tx=0x7fdad8005c60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.665+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdae00b99b0 con 0x7fdaf0100610 2024-03-24T09:19:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.838+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdaf010c170 con 0x7fdaf0100610 2024-03-24T09:19:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.838+0000 7fdae7fff640 1 -- 172.21.15.116:0/2957837803 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fdae0043020 con 0x7fdaf0100610 2024-03-24T09:19:52.839 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:19:52.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.840+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdadc0a20c0 msgr2=0x7fdadc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.840+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdadc0a20c0 0x7fdadc0a4580 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0069170 tx=0x7fdad8005c60 comp rx=0 tx=0).stop 2024-03-24T09:19:52.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.840+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 msgr2=0x7fdaf0069400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.840+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0069400 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fdae002c240 tx=0x7fdae00042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 shutdown_connections 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf01021c0 0x7fdaf019eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdadc0a20c0 0x7fdadc0a4580 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdaf0101800 0x7fdaf019c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 --2- 172.21.15.116:0/2957837803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaf0100610 0x7fdaf0069400 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 >> 172.21.15.116:0/2957837803 conn(0x7fdaf00fbb80 msgr2=0x7fdaf0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 shutdown_connections 2024-03-24T09:19:52.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.841+0000 7fdaf4cc3640 1 -- 172.21.15.116:0/2957837803 wait complete. 2024-03-24T09:19:52.852 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:19:52.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 -- 172.21.15.116:0/1210154212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c101720 msgr2=0x7fe55c101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 --2- 172.21.15.116:0/1210154212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c101720 0x7fe55c101b40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009900 tx=0x7fe54c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:52.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 -- 172.21.15.116:0/1210154212 shutdown_connections 2024-03-24T09:19:52.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 --2- 172.21.15.116:0/1210154212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe55c1032d0 0x7fe55c109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 --2- 172.21.15.116:0/1210154212 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe55c102910 0x7fe55c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 --2- 172.21.15.116:0/1210154212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c101720 0x7fe55c101b40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 -- 172.21.15.116:0/1210154212 >> 172.21.15.116:0/1210154212 conn(0x7fe55c0fccb0 msgr2=0x7fe55c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 -- 172.21.15.116:0/1210154212 shutdown_connections 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 -- 172.21.15.116:0/1210154212 wait complete. 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 Processor -- start 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.957+0000 7fe5628fb640 1 -- start start 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe55c101720 0x7fe55c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 0x7fe55c197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe55c1032d0 0x7fe55c19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c19f200 con 0x7fe55c102910 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c19f370 con 0x7fe55c101720 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c19f530 con 0x7fe55c1032d0 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 0x7fe55c197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 0x7fe55c197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57114/0 (socket says 172.21.15.116:57114) 2024-03-24T09:19:52.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 -- 172.21.15.116:0/4192485090 learned_addr learned my addr 172.21.15.116:0/4192485090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55bfff640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe55c101720 0x7fe55c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe560e98640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe55c1032d0 0x7fe55c19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 -- 172.21.15.116:0/4192485090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe55c1032d0 msgr2=0x7fe55c19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe55c1032d0 0x7fe55c19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 -- 172.21.15.116:0/4192485090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe55c101720 msgr2=0x7fe55c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe55c101720 0x7fe55c197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 -- 172.21.15.116:0/4192485090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54c009560 con 0x7fe55c102910 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe55b7fe640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 0x7fe55c197c50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe55000d770 tx=0x7fe55000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe550009c30 con 0x7fe55c102910 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe55000de80 con 0x7fe55c102910 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55c19f810 con 0x7fe55c102910 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe550004e00 con 0x7fe55c102910 2024-03-24T09:19:52.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.958+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe55c19fcd0 con 0x7fe55c102910 2024-03-24T09:19:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.959+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe55c108bb0 con 0x7fe55c102910 2024-03-24T09:19:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.960+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fe550004f60 con 0x7fe55c102910 2024-03-24T09:19:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.961+0000 7fe5597fa640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe5440a20c0 0x7fe5440a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.961+0000 7fe55bfff640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe5440a20c0 0x7fe5440a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.961+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe5500cd970 con 0x7fe55c102910 2024-03-24T09:19:52.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.961+0000 7fe55bfff640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe5440a20c0 0x7fe5440a4580 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe54c02c7d0 tx=0x7fe54c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:52.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:52.965+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe550096700 con 0x7fe55c102910 2024-03-24T09:19:53.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.139+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe55c02d160 con 0x7fe55c102910 2024-03-24T09:19:53.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.140+0000 7fe5597fa640 1 -- 172.21.15.116:0/4192485090 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fe550099dc0 con 0x7fe55c102910 2024-03-24T09:19:53.140 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:19:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.141+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe5440a20c0 msgr2=0x7fe5440a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.141+0000 7fe5628fb640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe5440a20c0 0x7fe5440a4580 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe54c02c7d0 tx=0x7fe54c005fd0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.141+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 msgr2=0x7fe55c197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.141+0000 7fe5628fb640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 0x7fe55c197c50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe55000d770 tx=0x7fe55000dc30 comp rx=0 tx=0).stop 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 shutdown_connections 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe5440a20c0 0x7fe5440a4580 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe55c1032d0 0x7fe55c19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe55c102910 0x7fe55c197c50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 --2- 172.21.15.116:0/4192485090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe55c101720 0x7fe55c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 >> 172.21.15.116:0/4192485090 conn(0x7fe55c0fccb0 msgr2=0x7fe55c105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 shutdown_connections 2024-03-24T09:19:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.142+0000 7fe5628fb640 1 -- 172.21.15.116:0/4192485090 wait complete. 2024-03-24T09:19:53.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:52 smithi116 ceph-mon[32207]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-24T09:19:53.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:52 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2442529806' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:19:53.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:52 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2036574957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:53.152 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:52 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2957837803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:53.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:19:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 -- 172.21.15.116:0/635117271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c0ffa10 msgr2=0x7f951c0ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 --2- 172.21.15.116:0/635117271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c0ffa10 0x7f951c0ffe30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f950c009900 tx=0x7f950c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 -- 172.21.15.116:0/635117271 shutdown_connections 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 --2- 172.21.15.116:0/635117271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c103440 0x7f951c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 --2- 172.21.15.116:0/635117271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 0x7f951c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 --2- 172.21.15.116:0/635117271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c0ffa10 0x7f951c0ffe30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 -- 172.21.15.116:0/635117271 >> 172.21.15.116:0/635117271 conn(0x7f951c0fb320 msgr2=0x7f951c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 -- 172.21.15.116:0/635117271 shutdown_connections 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 -- 172.21.15.116:0/635117271 wait complete. 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 Processor -- start 2024-03-24T09:19:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.258+0000 7f95228d4640 1 -- start start 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 0x7f951c197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 0x7f951c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c103440 0x7f951c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f951c19f240 con 0x7f951c103440 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f951c19f3b0 con 0x7f951c0ffa10 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f951c19f570 con 0x7f951c102a80 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c103440 0x7f951c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f951bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 0x7f951c197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f951bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 0x7f951c197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46280/0 (socket says 172.21.15.116:46280) 2024-03-24T09:19:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f951bfff640 1 -- 172.21.15.116:0/827076131 learned_addr learned my addr 172.21.15.116:0/827076131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f951b7fe640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 0x7f951c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 -- 172.21.15.116:0/827076131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 msgr2=0x7f951c197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 0x7f951c197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 -- 172.21.15.116:0/827076131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 msgr2=0x7f951c197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 0x7f951c197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 -- 172.21.15.116:0/827076131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9514009840 con 0x7f951c103440 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f951b7fe640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 0x7f951c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f951bfff640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 0x7f951c197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f9520e71640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c103440 0x7f951c19ed00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f951400ec50 tx=0x7f951400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514003b70 con 0x7f951c103440 2024-03-24T09:19:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f951400ce70 con 0x7f951c103440 2024-03-24T09:19:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950c009560 con 0x7f951c103440 2024-03-24T09:19:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9514010670 con 0x7f951c103440 2024-03-24T09:19:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.259+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f951c19fb80 con 0x7f951c103440 2024-03-24T09:19:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.261+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f951c108c70 con 0x7f951c103440 2024-03-24T09:19:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.261+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f9514004020 con 0x7f951c103440 2024-03-24T09:19:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.262+0000 7f95197fa640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f95040a20c0 0x7f95040a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.262+0000 7f951bfff640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f95040a20c0 0x7f95040a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.262+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9514014070 con 0x7f951c103440 2024-03-24T09:19:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.262+0000 7f951bfff640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f95040a20c0 0x7f95040a4580 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f950c005940 tx=0x7f950c005890 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:53.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.266+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9514097470 con 0x7f951c103440 2024-03-24T09:19:53.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:52 smithi143 ceph-mon[35965]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-24T09:19:53.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:52 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2442529806' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:19:53.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:52 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2036574957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:53.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:52 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2957837803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:52 smithi117 ceph-mon[35788]: pgmap v298: 129 pgs: 129 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-03-24T09:19:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:52 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2442529806' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:19:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:52 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2036574957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:19:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:52 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2957837803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:19:53.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.443+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f951c02d130 con 0x7f951c103440 2024-03-24T09:19:53.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.443+0000 7f95197fa640 1 -- 172.21.15.116:0/827076131 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f9514003e60 con 0x7f951c103440 2024-03-24T09:19:53.444 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:19:53.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f95040a20c0 msgr2=0x7f95040a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f95040a20c0 0x7f95040a4580 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f950c005940 tx=0x7f950c005890 comp rx=0 tx=0).stop 2024-03-24T09:19:53.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c103440 msgr2=0x7f951c19ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c103440 0x7f951c19ed00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f951400ec50 tx=0x7f951400c6a0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 shutdown_connections 2024-03-24T09:19:53.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f951c103440 0x7f951c19ed00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f95040a20c0 0x7f95040a4580 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f951c102a80 0x7f951c197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 --2- 172.21.15.116:0/827076131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f951c0ffa10 0x7f951c197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 >> 172.21.15.116:0/827076131 conn(0x7f951c0fb320 msgr2=0x7f951c105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:53.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 shutdown_connections 2024-03-24T09:19:53.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.445+0000 7f95228d4640 1 -- 172.21.15.116:0/827076131 wait complete. 2024-03-24T09:19:53.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.561+0000 7f2329d4f640 1 -- 172.21.15.116:0/3336334594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324101800 msgr2=0x7f2324101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.561+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3336334594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324101800 0x7f2324101c80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f230c009900 tx=0x7f230c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- 172.21.15.116:0/3336334594 shutdown_connections 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3336334594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 0x7f2324108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3336334594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324101800 0x7f2324101c80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3336334594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2324100610 0x7f2324100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- 172.21.15.116:0/3336334594 >> 172.21.15.116:0/3336334594 conn(0x7f23240fbba0 msgr2=0x7f23240fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- 172.21.15.116:0/3336334594 shutdown_connections 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- 172.21.15.116:0/3336334594 wait complete. 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 Processor -- start 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- start start 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 0x7f2324196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2324101800 0x7f2324196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 0x7f232419dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f232419e110 con 0x7f2324100610 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f232419e280 con 0x7f23241021c0 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f2329d4f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f232419e440 con 0x7f2324101800 2024-03-24T09:19:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.562+0000 7f23237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 0x7f2324196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 0x7f232419dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f23237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 0x7f2324196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57148/0 (socket says 172.21.15.116:57148) 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f23237fe640 1 -- 172.21.15.116:0/3924277440 learned_addr learned my addr 172.21.15.116:0/3924277440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f231affd640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2324101800 0x7f2324196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 -- 172.21.15.116:0/3924277440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2324101800 msgr2=0x7f2324196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2324101800 0x7f2324196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 -- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 msgr2=0x7f2324196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 0x7f2324196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 -- 172.21.15.116:0/3924277440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231c009840 con 0x7f23241021c0 2024-03-24T09:19:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f23237fe640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 0x7f2324196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2323fff640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 0x7f232419dbd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f231c00ec50 tx=0x7f231c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c003b70 con 0x7f23241021c0 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230c009560 con 0x7f23241021c0 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f232419eaa0 con 0x7f23241021c0 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f231c00ce70 con 0x7f23241021c0 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.563+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c010670 con 0x7f23241021c0 2024-03-24T09:19:53.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.564+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2324107a50 con 0x7f23241021c0 2024-03-24T09:19:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.565+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f231c004b40 con 0x7f23241021c0 2024-03-24T09:19:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.566+0000 7f23217fa640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f23100a20c0 0x7f23100a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.566+0000 7f23237fe640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f23100a20c0 0x7f23100a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.566+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f231c014070 con 0x7f23241021c0 2024-03-24T09:19:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.566+0000 7f23237fe640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f23100a20c0 0x7f23100a4580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2314005fd0 tx=0x7f231400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.574+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f231c097430 con 0x7f23241021c0 2024-03-24T09:19:53.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.747+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f232410c080 con 0x7f23241021c0 2024-03-24T09:19:53.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.748+0000 7f23217fa640 1 -- 172.21.15.116:0/3924277440 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f231c09aaf0 con 0x7f23241021c0 2024-03-24T09:19:53.749 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f23100a20c0 msgr2=0x7f23100a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f23100a20c0 0x7f23100a4580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2314005fd0 tx=0x7f231400b410 comp rx=0 tx=0).stop 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 msgr2=0x7f232419dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 0x7f232419dbd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f231c00ec50 tx=0x7f231c00c6a0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 shutdown_connections 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23241021c0 0x7f232419dbd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f23100a20c0 0x7f23100a4580 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2324101800 0x7f2324196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 --2- 172.21.15.116:0/3924277440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2324100610 0x7f2324196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 >> 172.21.15.116:0/3924277440 conn(0x7f23240fbba0 msgr2=0x7f2324104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 shutdown_connections 2024-03-24T09:19:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.750+0000 7f2329d4f640 1 -- 172.21.15.116:0/3924277440 wait complete. 2024-03-24T09:19:53.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T09:19:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 -- 172.21.15.116:0/646901103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 msgr2=0x7f5a5c101b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 --2- 172.21.15.116:0/646901103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c101b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c009900 tx=0x7f5a4c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:19:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 -- 172.21.15.116:0/646901103 shutdown_connections 2024-03-24T09:19:53.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 --2- 172.21.15.116:0/646901103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a5c1032e0 0x7f5a5c109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 --2- 172.21.15.116:0/646901103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 0x7f5a5c102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 --2- 172.21.15.116:0/646901103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c101b50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 -- 172.21.15.116:0/646901103 >> 172.21.15.116:0/646901103 conn(0x7f5a5c0fcca0 msgr2=0x7f5a5c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 -- 172.21.15.116:0/646901103 shutdown_connections 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 -- 172.21.15.116:0/646901103 wait complete. 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.864+0000 7f5a64247640 1 Processor -- start 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 -- start start 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 0x7f5a5c197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a5c1032e0 0x7f5a5c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c19f250 con 0x7f5a5c101730 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c19f3c0 con 0x7f5a5c102920 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5c19f580 con 0x7f5a5c1032e0 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57158/0 (socket says 172.21.15.116:57158) 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 -- 172.21.15.116:0/1299296450 learned_addr learned my addr 172.21.15.116:0/1299296450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a617e2640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 0x7f5a5c197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 -- 172.21.15.116:0/1299296450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a5c1032e0 msgr2=0x7f5a5c19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a5c1032e0 0x7f5a5c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 -- 172.21.15.116:0/1299296450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 msgr2=0x7f5a5c197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 0x7f5a5c197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 -- 172.21.15.116:0/1299296450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a58009490 con 0x7f5a5c101730 2024-03-24T09:19:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a617e2640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 0x7f5a5c197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a61fe3640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c197760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c02c7d0 tx=0x7f5a4c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c004aa0 con 0x7f5a5c101730 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5a4c02de90 con 0x7f5a5c101730 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4c009560 con 0x7f5a5c101730 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c005440 con 0x7f5a5c101730 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.865+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a5c19fb80 con 0x7f5a5c101730 2024-03-24T09:19:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.866+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a5c108b70 con 0x7f5a5c101730 2024-03-24T09:19:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.867+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f5a4c0055a0 con 0x7f5a5c101730 2024-03-24T09:19:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.868+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f5a480a20c0 0x7f5a480a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:19:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.868+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5a4c0f0120 con 0x7f5a5c101730 2024-03-24T09:19:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.868+0000 7f5a617e2640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f5a480a20c0 0x7f5a480a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:19:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.868+0000 7f5a617e2640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f5a480a20c0 0x7f5a480a4580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c198c70 tx=0x7f5a58005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:19:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:53.872+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5a4c0b8eb0 con 0x7f5a5c101730 2024-03-24T09:19:54.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.064+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a5c02d1d0 con 0x7f5a5c101730 2024-03-24T09:19:54.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.064+0000 7f5a52ffd640 1 -- 172.21.15.116:0/1299296450 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+599 (secure 0 0 0) 0x7f5a4c0bc570 con 0x7f5a5c101730 2024-03-24T09:19:54.065 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3305720445 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:54.065 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3598626147 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:54.065 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:54.065 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:54.065 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:54.065 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3855909741 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:54.066 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/448304506 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:54.066 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:54.066 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3108228013 2024-03-25T09:12:07.749841+0000 2024-03-24T09:19:54.066 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2688022316 2024-03-25T09:12:33.839806+0000 2024-03-24T09:19:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-24T09:19:54.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f5a480a20c0 msgr2=0x7f5a480a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f5a480a20c0 0x7f5a480a4580 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5a5c198c70 tx=0x7f5a58005f90 comp rx=0 tx=0).stop 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 msgr2=0x7f5a5c197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c197760 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c02c7d0 tx=0x7f5a4c0042f0 comp rx=0 tx=0).stop 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 shutdown_connections 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f5a480a20c0 0x7f5a480a4580 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5a5c1032e0 0x7f5a5c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a5c102920 0x7f5a5c197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 --2- 172.21.15.116:0/1299296450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a5c101730 0x7f5a5c197760 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 >> 172.21.15.116:0/1299296450 conn(0x7f5a5c0fcca0 msgr2=0x7f5a5c105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:19:54.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 shutdown_connections 2024-03-24T09:19:54.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:19:54.066+0000 7f5a64247640 1 -- 172.21.15.116:0/1299296450 wait complete. 2024-03-24T09:19:54.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4192485090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/827076131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.069 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3924277440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.078 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-03-24T09:19:54.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:19:54.078 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-24T09:19:54.105 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi116.stderr:ceph-fuse[76252]: fuse finished with error 0 and tester_r 0 2024-03-24T09:19:54.106 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T09:19:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4192485090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/827076131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3924277440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4192485090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/827076131' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:19:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3924277440' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:19:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:54 smithi143 ceph-mon[35965]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 301 B/s wr, 0 op/s 2024-03-24T09:19:55.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1299296450' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:19:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:54 smithi117 ceph-mon[35788]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 301 B/s wr, 0 op/s 2024-03-24T09:19:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1299296450' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:19:55.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:54 smithi116 ceph-mon[32207]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 301 B/s wr, 0 op/s 2024-03-24T09:19:55.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1299296450' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:19:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:56 smithi143 ceph-mon[35965]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T09:19:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:56 smithi117 ceph-mon[35788]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T09:19:57.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:56 smithi116 ceph-mon[32207]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-24T09:19:59.246 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:19:58 smithi117 ceph-mon[35788]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:19:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:19:58 smithi143 ceph-mon[35965]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:19:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:19:58 smithi116 ceph-mon[32207]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:00.208 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:20:00.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:20:00.208 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-24T09:20:00.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:20:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 -- 172.21.15.116:0/3016517140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f0101820 msgr2=0x7f62f0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- 172.21.15.116:0/3016517140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f0101820 0x7f62f0101ca0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f62e0009900 tx=0x7f62e002bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 -- 172.21.15.116:0/3016517140 shutdown_connections 2024-03-24T09:20:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- 172.21.15.116:0/3016517140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f01021e0 0x7f62f0108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- 172.21.15.116:0/3016517140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f0101820 0x7f62f0101ca0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- 172.21.15.116:0/3016517140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0100630 0x7f62f0100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 -- 172.21.15.116:0/3016517140 >> 172.21.15.116:0/3016517140 conn(0x7f62f00fbba0 msgr2=0x7f62f00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 -- 172.21.15.116:0/3016517140 shutdown_connections 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 -- 172.21.15.116:0/3016517140 wait complete. 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 Processor -- start 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 -- start start 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f0100630 0x7f62f00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0101820 0x7f62f019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.380+0000 7f62f8684640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 0x7f62f01a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f8684640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62f01a1460 con 0x7f62f01021e0 2024-03-24T09:20:00.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f8684640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f62f01a15d0 con 0x7f62f0101820 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f8684640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f62f01a1790 con 0x7f62f0100630 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 0x7f62f01a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 0x7f62f01a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49870/0 (socket says 172.21.15.116:49870) 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 -- 172.21.15.116:0/2073029335 learned_addr learned my addr 172.21.15.116:0/2073029335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f5c1f640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0101820 0x7f62f019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6420640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f0100630 0x7f62f00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 -- 172.21.15.116:0/2073029335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f0100630 msgr2=0x7f62f00693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f0100630 0x7f62f00693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 -- 172.21.15.116:0/2073029335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0101820 msgr2=0x7f62f019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0101820 0x7f62f019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 -- 172.21.15.116:0/2073029335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec009840 con 0x7f62f01021e0 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6420640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f0100630 0x7f62f00693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f5c1f640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0101820 0x7f62f019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f6c21640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 0x7f62f01a0da0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c970 tx=0x7f62e800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8007bb0 con 0x7f62f01021e0 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f62e8007d10 con 0x7f62f01021e0 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62e0009560 con 0x7f62f01021e0 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8004e40 con 0x7f62f01021e0 2024-03-24T09:20:00.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.381+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f01a1db0 con 0x7f62f01021e0 2024-03-24T09:20:00.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.382+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f0107970 con 0x7f62f01021e0 2024-03-24T09:20:00.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.384+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f62e8005000 con 0x7f62f01021e0 2024-03-24T09:20:00.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.384+0000 7f62e77fe640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f62dc0a2170 0x7f62dc0a4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.385+0000 7f62f6420640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f62dc0a2170 0x7f62dc0a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.385+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f62e80cdeb0 con 0x7f62f01021e0 2024-03-24T09:20:00.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.385+0000 7f62f6420640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f62dc0a2170 0x7f62dc0a4630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f62f0101680 tx=0x7f62ec009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:00.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.388+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f62e80fc280 con 0x7f62f01021e0 2024-03-24T09:20:00.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.570+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62f010bef0 con 0x7f62f01021e0 2024-03-24T09:20:00.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.572+0000 7f62e77fe640 1 -- 172.21.15.116:0/2073029335 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7f62e801d070 con 0x7f62f01021e0 2024-03-24T09:20:00.573 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:00.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f62dc0a2170 msgr2=0x7f62dc0a4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f62dc0a2170 0x7f62dc0a4630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f62f0101680 tx=0x7f62ec009500 comp rx=0 tx=0).stop 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 msgr2=0x7f62f01a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 0x7f62f01a0da0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c970 tx=0x7f62e800ce30 comp rx=0 tx=0).stop 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 shutdown_connections 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62f01021e0 0x7f62f01a0da0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f0101820 0x7f62f019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f62dc0a2170 0x7f62dc0a4630 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 --2- 172.21.15.116:0/2073029335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f62f0100630 0x7f62f00693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 >> 172.21.15.116:0/2073029335 conn(0x7f62f00fbba0 msgr2=0x7f62f0104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 shutdown_connections 2024-03-24T09:20:00.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.574+0000 7f62f8684640 1 -- 172.21.15.116:0/2073029335 wait complete. 2024-03-24T09:20:00.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:20:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.689+0000 7f4ef3de9640 1 -- 172.21.15.116:0/2088036102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec100630 msgr2=0x7f4eec100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.689+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/2088036102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec100630 0x7f4eec100a50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009900 tx=0x7f4edc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- 172.21.15.116:0/2088036102 shutdown_connections 2024-03-24T09:20:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/2088036102 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec1021e0 0x7f4eec108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/2088036102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec101820 0x7f4eec101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/2088036102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec100630 0x7f4eec100a50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- 172.21.15.116:0/2088036102 >> 172.21.15.116:0/2088036102 conn(0x7f4eec0fbba0 msgr2=0x7f4eec0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- 172.21.15.116:0/2088036102 shutdown_connections 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- 172.21.15.116:0/2088036102 wait complete. 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 Processor -- start 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- start start 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec100630 0x7f4eec196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 0x7f4eec196bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec1021e0 0x7f4eec19dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec19e180 con 0x7f4eec101820 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec19e2f0 con 0x7f4eec100630 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef3de9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4eec19e4b0 con 0x7f4eec1021e0 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef1384640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 0x7f4eec196bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef1b85640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec100630 0x7f4eec196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.690+0000 7f4ef1384640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 0x7f4eec196bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49882/0 (socket says 172.21.15.116:49882) 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 -- 172.21.15.116:0/729618958 learned_addr learned my addr 172.21.15.116:0/729618958 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef2386640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec1021e0 0x7f4eec19dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 -- 172.21.15.116:0/729618958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec1021e0 msgr2=0x7f4eec19dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec1021e0 0x7f4eec19dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 -- 172.21.15.116:0/729618958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec100630 msgr2=0x7f4eec196690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec100630 0x7f4eec196690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 -- 172.21.15.116:0/729618958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4edc009560 con 0x7f4eec101820 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1b85640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec100630 0x7f4eec196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef1384640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 0x7f4eec196bd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800e9f0 tx=0x7f4ee800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8009700 con 0x7f4eec101820 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eec19e790 con 0x7f4eec101820 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ee800cd80 con 0x7f4eec101820 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8018840 con 0x7f4eec101820 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.691+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eec19ed70 con 0x7f4eec101820 2024-03-24T09:20:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.692+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4eec107a20 con 0x7f4eec101820 2024-03-24T09:20:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.694+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f4ee80189a0 con 0x7f4eec101820 2024-03-24T09:20:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.694+0000 7f4ee2ffd640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4ed80a20c0 0x7f4ed80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.695+0000 7f4ef1b85640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4ed80a20c0 0x7f4ed80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.695+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4ee8014070 con 0x7f4eec101820 2024-03-24T09:20:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.695+0000 7f4ef1b85640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4ed80a20c0 0x7f4ed80a4580 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4edc02c7d0 tx=0x7f4edc005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:00.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.698+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4ee80974b0 con 0x7f4eec101820 2024-03-24T09:20:00.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.876+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4eec02d130 con 0x7f4eec101820 2024-03-24T09:20:00.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.877+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/729618958 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7f4ee809ab70 con 0x7f4eec101820 2024-03-24T09:20:00.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:00.878 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4ed80a20c0 msgr2=0x7f4ed80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4ed80a20c0 0x7f4ed80a4580 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4edc02c7d0 tx=0x7f4edc005fd0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 msgr2=0x7f4eec196bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 0x7f4eec196bd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800e9f0 tx=0x7f4ee800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 shutdown_connections 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4ed80a20c0 0x7f4ed80a4580 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4eec1021e0 0x7f4eec19dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4eec101820 0x7f4eec196bd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 --2- 172.21.15.116:0/729618958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4eec100630 0x7f4eec196690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:00.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 >> 172.21.15.116:0/729618958 conn(0x7f4eec0fbba0 msgr2=0x7f4eec104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:00.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 shutdown_connections 2024-03-24T09:20:00.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:00.879+0000 7f4ef3de9640 1 -- 172.21.15.116:0/729618958 wait complete. 2024-03-24T09:20:00.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 -- 172.21.15.116:0/1804412088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba90100610 msgr2=0x7fba90100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 --2- 172.21.15.116:0/1804412088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba90100610 0x7fba90100a30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fba80009860 tx=0x7fba8002bb70 comp rx=0 tx=0).stop 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 -- 172.21.15.116:0/1804412088 shutdown_connections 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 --2- 172.21.15.116:0/1804412088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba901021c0 0x7fba90108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 --2- 172.21.15.116:0/1804412088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba90101800 0x7fba90101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 --2- 172.21.15.116:0/1804412088 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba90100610 0x7fba90100a30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 -- 172.21.15.116:0/1804412088 >> 172.21.15.116:0/1804412088 conn(0x7fba900fbba0 msgr2=0x7fba900fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:01.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 -- 172.21.15.116:0/1804412088 shutdown_connections 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.000+0000 7fba98519640 1 -- 172.21.15.116:0/1804412088 wait complete. 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 Processor -- start 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 -- start start 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba90100610 0x7fba90196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 0x7fba90196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba901021c0 0x7fba9019dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fba9019e170 con 0x7fba90101800 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fba9019e2e0 con 0x7fba901021c0 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fba9019e4a0 con 0x7fba90100610 2024-03-24T09:20:01.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 0x7fba90196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 0x7fba90196bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49902/0 (socket says 172.21.15.116:49902) 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 -- 172.21.15.116:0/1700319183 learned_addr learned my addr 172.21.15.116:0/1700319183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba96ab6640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba901021c0 0x7fba9019dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba962b5640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba90100610 0x7fba90196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 -- 172.21.15.116:0/1700319183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba90100610 msgr2=0x7fba90196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba90100610 0x7fba90196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 -- 172.21.15.116:0/1700319183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba901021c0 msgr2=0x7fba9019dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba901021c0 0x7fba9019dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 -- 172.21.15.116:0/1700319183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba88009900 con 0x7fba90101800 2024-03-24T09:20:01.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba95ab4640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 0x7fba90196bc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00d770 tx=0x7fba8c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c009c30 con 0x7fba90101800 2024-03-24T09:20:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba8c00de80 con 0x7fba90101800 2024-03-24T09:20:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.001+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba80009490 con 0x7fba90101800 2024-03-24T09:20:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.002+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c004e00 con 0x7fba90101800 2024-03-24T09:20:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.002+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba9019eb00 con 0x7fba90101800 2024-03-24T09:20:01.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.002+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba90107a40 con 0x7fba90101800 2024-03-24T09:20:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.004+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fba8c004f60 con 0x7fba90101800 2024-03-24T09:20:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.005+0000 7fba877fe640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fba7c0a2140 0x7fba7c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.005+0000 7fba962b5640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fba7c0a2140 0x7fba7c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:01.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.005+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fba8c0cdcb0 con 0x7fba90101800 2024-03-24T09:20:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.005+0000 7fba962b5640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fba7c0a2140 0x7fba7c0a4600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fba800349e0 tx=0x7fba80000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:01.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.008+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba8c096ac0 con 0x7fba90101800 2024-03-24T09:20:01.223 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:00 smithi116 ceph-mon[32207]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:01.223 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:00 smithi116 ceph-mon[32207]: overall HEALTH_OK 2024-03-24T09:20:01.223 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:00 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2073029335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:01.223 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:00 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/729618958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:01.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:01.225+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fba9010bed0 con 0x7fba90101800 2024-03-24T09:20:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:00 smithi143 ceph-mon[35965]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:00 smithi143 ceph-mon[35965]: overall HEALTH_OK 2024-03-24T09:20:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:00 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2073029335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:00 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/729618958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:00 smithi117 ceph-mon[35788]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:00 smithi117 ceph-mon[35788]: overall HEALTH_OK 2024-03-24T09:20:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:00 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2073029335' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:00 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/729618958' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:02.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.006+0000 7fba877fe640 1 -- 172.21.15.116:0/1700319183 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fba8c020070 con 0x7fba90101800 2024-03-24T09:20:02.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fba7c0a2140 msgr2=0x7fba7c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fba7c0a2140 0x7fba7c0a4600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fba800349e0 tx=0x7fba80000bc0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 msgr2=0x7fba90196bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 0x7fba90196bc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00d770 tx=0x7fba8c00dc30 comp rx=0 tx=0).stop 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 shutdown_connections 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba901021c0 0x7fba9019dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba90101800 0x7fba90196bc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fba7c0a2140 0x7fba7c0a4600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 --2- 172.21.15.116:0/1700319183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba90100610 0x7fba90196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 >> 172.21.15.116:0/1700319183 conn(0x7fba900fbba0 msgr2=0x7fba90104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 shutdown_connections 2024-03-24T09:20:02.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.009+0000 7fba98519640 1 -- 172.21.15.116:0/1700319183 wait complete. 2024-03-24T09:20:02.022 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-03-24T09:20:02.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 -- 172.21.15.116:0/985537641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 msgr2=0x7fa4ec102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 --2- 172.21.15.116:0/985537641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec102d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009900 tx=0x7fa4e002bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 -- 172.21.15.116:0/985537641 shutdown_connections 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 --2- 172.21.15.116:0/985537641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ec1032d0 0x7fa4ec109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 --2- 172.21.15.116:0/985537641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec102d90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 --2- 172.21.15.116:0/985537641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4ec101720 0x7fa4ec101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 -- 172.21.15.116:0/985537641 >> 172.21.15.116:0/985537641 conn(0x7fa4ec0fccb0 msgr2=0x7fa4ec0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:02.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 -- 172.21.15.116:0/985537641 shutdown_connections 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 -- 172.21.15.116:0/985537641 wait complete. 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.134+0000 7fa4f208f640 1 Processor -- start 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 -- start start 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ec101720 0x7fa4ec0754c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4ec1032d0 0x7fa4ec074040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec0747a0 con 0x7fa4ec102910 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec074910 con 0x7fa4ec1032d0 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ec1a17a0 con 0x7fa4ec101720 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec073b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49912/0 (socket says 172.21.15.116:49912) 2024-03-24T09:20:02.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 -- 172.21.15.116:0/3117028386 learned_addr learned my addr 172.21.15.116:0/3117028386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4ebfff640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4ec1032d0 0x7fa4ec074040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eb7fe640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ec101720 0x7fa4ec0754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 -- 172.21.15.116:0/3117028386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ec101720 msgr2=0x7fa4ec0754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ec101720 0x7fa4ec0754c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 -- 172.21.15.116:0/3117028386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4ec1032d0 msgr2=0x7fa4ec074040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4ec1032d0 0x7fa4ec074040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 -- 172.21.15.116:0/3117028386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4e0009560 con 0x7fa4ec102910 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4eaffd640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec073b00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009430 tx=0x7fa4e0004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0004dc0 con 0x7fa4ec102910 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ec1a19f0 con 0x7fa4ec102910 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa4e002de90 con 0x7fa4ec102910 2024-03-24T09:20:02.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.135+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0003d10 con 0x7fa4ec102910 2024-03-24T09:20:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.136+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ec1a1fd0 con 0x7fa4ec102910 2024-03-24T09:20:02.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.137+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ec108b60 con 0x7fa4ec102910 2024-03-24T09:20:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.138+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fa4e0004f20 con 0x7fa4ec102910 2024-03-24T09:20:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.138+0000 7fa4e8ff9640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa4d40a20c0 0x7fa4d40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.138+0000 7fa4eb7fe640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa4d40a20c0 0x7fa4d40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.139+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa4e00f0370 con 0x7fa4ec102910 2024-03-24T09:20:02.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.139+0000 7fa4eb7fe640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa4d40a20c0 0x7fa4d40a4580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec102770 tx=0x7fa4dc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:02.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.143+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa4e00b9100 con 0x7fa4ec102910 2024-03-24T09:20:02.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:01 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1700319183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T09:20:02.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:01 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1700319183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T09:20:02.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:01 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1700319183' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T09:20:02.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:02.354+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fa4ec02d160 con 0x7fa4ec102910 2024-03-24T09:20:03.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.012+0000 7fa4e8ff9640 1 -- 172.21.15.116:0/3117028386 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fa4e00bc7c0 con 0x7fa4ec102910 2024-03-24T09:20:03.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa4d40a20c0 msgr2=0x7fa4d40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa4d40a20c0 0x7fa4d40a4580 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec102770 tx=0x7fa4dc005f90 comp rx=0 tx=0).stop 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 msgr2=0x7fa4ec073b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec073b00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009430 tx=0x7fa4e0004140 comp rx=0 tx=0).stop 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 shutdown_connections 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4ec1032d0 0x7fa4ec074040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4ec102910 0x7fa4ec073b00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa4d40a20c0 0x7fa4d40a4580 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 --2- 172.21.15.116:0/3117028386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4ec101720 0x7fa4ec0754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 >> 172.21.15.116:0/3117028386 conn(0x7fa4ec0fccb0 msgr2=0x7fa4ec105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 shutdown_connections 2024-03-24T09:20:03.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.016+0000 7fa4f208f640 1 -- 172.21.15.116:0/3117028386 wait complete. 2024-03-24T09:20:03.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-24T09:20:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 -- 172.21.15.116:0/2292759230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0107410 msgr2=0x7f24a0109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2292759230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0107410 0x7f24a0109860 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2498009ca0 tx=0x7f249802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 -- 172.21.15.116:0/2292759230 shutdown_connections 2024-03-24T09:20:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2292759230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0107410 0x7f24a0109860 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2292759230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24a0104ae0 0x7f24a0106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2292759230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24a01021b0 0x7f24a01045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 -- 172.21.15.116:0/2292759230 >> 172.21.15.116:0/2292759230 conn(0x7f24a00fbbc0 msgr2=0x7f24a00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 -- 172.21.15.116:0/2292759230 shutdown_connections 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.141+0000 7f24a5e88640 1 -- 172.21.15.116:0/2292759230 wait complete. 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 Processor -- start 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 -- start start 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24a01021b0 0x7f24a01976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 0x7f24a0197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24a0107410 0x7f24a019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24a019f1c0 con 0x7f24a0104ae0 2024-03-24T09:20:03.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f24a019f330 con 0x7f24a0107410 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f24a5e88640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f24a019f4f0 con 0x7f24a01021b0 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f249effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 0x7f24a0197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f249effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 0x7f24a0197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49926/0 (socket says 172.21.15.116:49926) 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f249effd640 1 -- 172.21.15.116:0/2456148075 learned_addr learned my addr 172.21.15.116:0/2456148075 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f249ffff640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24a0107410 0x7f24a019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.142+0000 7f249f7fe640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24a01021b0 0x7f24a01976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249effd640 1 -- 172.21.15.116:0/2456148075 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24a01021b0 msgr2=0x7f24a01976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249effd640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24a01021b0 0x7f24a01976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249effd640 1 -- 172.21.15.116:0/2456148075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24a0107410 msgr2=0x7f24a019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249effd640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24a0107410 0x7f24a019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249effd640 1 -- 172.21.15.116:0/2456148075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2490009490 con 0x7f24a0104ae0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249effd640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 0x7f24a0197c10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f249400e9f0 tx=0x7f249400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24940096c0 con 0x7f24a0104ae0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2498009900 con 0x7f24a0104ae0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f24940054d0 con 0x7f24a0104ae0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494010430 con 0x7f24a0104ae0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.143+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a019fb50 con 0x7f24a0104ae0 2024-03-24T09:20:03.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.144+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a00fd6b0 con 0x7f24a0104ae0 2024-03-24T09:20:03.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.145+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f2494005000 con 0x7f24a0104ae0 2024-03-24T09:20:03.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.145+0000 7f249cff9640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f24880a20c0 0x7f24880a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:03.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.146+0000 7f249f7fe640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f24880a20c0 0x7f24880a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:03.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.146+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2494014070 con 0x7f24a0104ae0 2024-03-24T09:20:03.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.146+0000 7f249f7fe640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f24880a20c0 0x7f24880a4580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2490009cf0 tx=0x7f2490005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:03.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.150+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2494097080 con 0x7f24a0104ae0 2024-03-24T09:20:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:03 smithi143 ceph-mon[35965]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:03 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1700319183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T09:20:03.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:03 smithi143 ceph-mon[35965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:20:03.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:03 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3117028386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T09:20:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:03 smithi117 ceph-mon[35788]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:03 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1700319183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T09:20:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:03 smithi117 ceph-mon[35788]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:20:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:03 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3117028386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T09:20:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:03 smithi116 ceph-mon[32207]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:03 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1700319183' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T09:20:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:03 smithi116 ceph-mon[32207]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:20:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:03 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3117028386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T09:20:03.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.360+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f24a002d130 con 0x7f24a0104ae0 2024-03-24T09:20:03.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.885+0000 7f249cff9640 1 -- 172.21.15.116:0/2456148075 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f249409a740 con 0x7f24a0104ae0 2024-03-24T09:20:03.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f24880a20c0 msgr2=0x7f24880a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f24880a20c0 0x7f24880a4580 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2490009cf0 tx=0x7f2490005f90 comp rx=0 tx=0).stop 2024-03-24T09:20:03.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 msgr2=0x7f24a0197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:03.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 0x7f24a0197c10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f249400e9f0 tx=0x7f249400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 shutdown_connections 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24a0107410 0x7f24a019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24a0104ae0 0x7f24a0197c10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f24880a20c0 0x7f24880a4580 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 --2- 172.21.15.116:0/2456148075 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24a01021b0 0x7f24a01976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 >> 172.21.15.116:0/2456148075 conn(0x7f24a00fbbc0 msgr2=0x7f24a0102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 shutdown_connections 2024-03-24T09:20:03.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:03.888+0000 7f24a5e88640 1 -- 172.21.15.116:0/2456148075 wait complete. 2024-03-24T09:20:03.905 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 -- 172.21.15.116:0/1257647011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8101810 msgr2=0x7f2bb8101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/1257647011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8101810 0x7f2bb8101c90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009900 tx=0x7f2bac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 -- 172.21.15.116:0/1257647011 shutdown_connections 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/1257647011 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb81021d0 0x7f2bb8108a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/1257647011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8101810 0x7f2bb8101c90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/1257647011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bb8100620 0x7f2bb8100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 -- 172.21.15.116:0/1257647011 >> 172.21.15.116:0/1257647011 conn(0x7f2bb80fbbb0 msgr2=0x7f2bb80fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:04.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.014+0000 7f2bbe04e640 1 -- 172.21.15.116:0/1257647011 shutdown_connections 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 -- 172.21.15.116:0/1257647011 wait complete. 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 Processor -- start 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 -- start start 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 0x7f2bb806a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bb8101810 0x7f2bb819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb81021d0 0x7f2bb819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb819f1e0 con 0x7f2bb8100620 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb819f350 con 0x7f2bb8101810 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bbe04e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb819f510 con 0x7f2bb81021d0 2024-03-24T09:20:04.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bb77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 0x7f2bb806a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bb77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 0x7f2bb806a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49940/0 (socket says 172.21.15.116:49940) 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2954510123 learned_addr learned my addr 172.21.15.116:0/2954510123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bb7fff640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb81021d0 0x7f2bb819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.015+0000 7f2bb6ffd640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bb8101810 0x7f2bb819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2954510123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb81021d0 msgr2=0x7f2bb819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb77fe640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb81021d0 0x7f2bb819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2954510123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bb8101810 msgr2=0x7f2bb819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb77fe640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bb8101810 0x7f2bb819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2954510123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bac009560 con 0x7f2bb8100620 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb77fe640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 0x7f2bb806a8f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e9c0 tx=0x7f2ba800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8009700 con 0x7f2bb8100620 2024-03-24T09:20:04.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb819f7f0 con 0x7f2bb8100620 2024-03-24T09:20:04.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2ba80044d0 con 0x7f2bb8100620 2024-03-24T09:20:04.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8010430 con 0x7f2bb8100620 2024-03-24T09:20:04.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.016+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb819fcd0 con 0x7f2bb8100620 2024-03-24T09:20:04.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.017+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb8107a30 con 0x7f2bb8100620 2024-03-24T09:20:04.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.018+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f2ba8003f30 con 0x7f2bb8100620 2024-03-24T09:20:04.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.019+0000 7f2bb4ff9640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2ba00a20c0 0x7f2ba00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:04.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.019+0000 7f2bb6ffd640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2ba00a20c0 0x7f2ba00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:04.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.019+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2ba8014070 con 0x7f2bb8100620 2024-03-24T09:20:04.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.019+0000 7f2bb6ffd640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2ba00a20c0 0x7f2ba00a4580 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009230 tx=0x7f2bac005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:04.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.023+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2ba8097050 con 0x7f2bb8100620 2024-03-24T09:20:04.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.233+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f2bb810bee0 con 0x7f2bb8100620 2024-03-24T09:20:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T09:20:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3117028386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T09:20:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T09:20:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:20:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T09:20:04.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2456148075' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2456148075' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] up:active 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-24T09:20:04.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:04 smithi143 ceph-mon[35965]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3117028386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2456148075' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2456148075' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] up:active 2024-03-24T09:20:04.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T09:20:04.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-24T09:20:04.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-24T09:20:04.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:04 smithi117 ceph-mon[35788]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-24T09:20:04.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T09:20:04.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3117028386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2456148075' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2456148075' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] up:active 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: daemon mds.f assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 2 up:standby-replay 8 up:standby 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:creating} 2 up:standby-replay 7 up:standby 2024-03-24T09:20:04.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:04 smithi116 ceph-mon[32207]: daemon mds.f is now active in filesystem cephfs as rank 2 2024-03-24T09:20:04.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.896+0000 7f2bb4ff9640 1 -- 172.21.15.116:0/2954510123 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f2ba809a710 con 0x7f2bb8100620 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2ba00a20c0 msgr2=0x7f2ba00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2ba00a20c0 0x7f2ba00a4580 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2bac009230 tx=0x7f2bac005fd0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 msgr2=0x7f2bb806a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 0x7f2bb806a8f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800e9c0 tx=0x7f2ba800ee80 comp rx=0 tx=0).stop 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 shutdown_connections 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2ba00a20c0 0x7f2ba00a4580 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2bb81021d0 0x7f2bb819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bb8101810 0x7f2bb819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 --2- 172.21.15.116:0/2954510123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bb8100620 0x7f2bb806a8f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 >> 172.21.15.116:0/2954510123 conn(0x7f2bb80fbbb0 msgr2=0x7f2bb8104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 shutdown_connections 2024-03-24T09:20:04.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:04.900+0000 7f2bbe04e640 1 -- 172.21.15.116:0/2954510123 wait complete. 2024-03-24T09:20:04.916 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-03-24T09:20:04.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-24T09:20:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.022+0000 7fdcd6c40640 1 -- 172.21.15.116:0/384351364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01033a0 msgr2=0x7fdcd0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.022+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/384351364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01033a0 0x7fdcd0109c30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009ca0 tx=0x7fdcc802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.022+0000 7fdcd6c40640 1 -- 172.21.15.116:0/384351364 shutdown_connections 2024-03-24T09:20:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.022+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/384351364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01033a0 0x7fdcd0109c30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/384351364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01029e0 0x7fdcd0102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/384351364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd00ffa10 0x7fdcd00ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- 172.21.15.116:0/384351364 >> 172.21.15.116:0/384351364 conn(0x7fdcd00fb340 msgr2=0x7fdcd00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:05.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- 172.21.15.116:0/384351364 shutdown_connections 2024-03-24T09:20:05.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- 172.21.15.116:0/384351364 wait complete. 2024-03-24T09:20:05.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 Processor -- start 2024-03-24T09:20:05.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- start start 2024-03-24T09:20:05.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd00ffa10 0x7fdcd0197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 0x7fdcd0197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01033a0 0x7fdcd019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd019f140 con 0x7fdcd01029e0 2024-03-24T09:20:05.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd019f2b0 con 0x7fdcd01033a0 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd6c40640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd019f470 con 0x7fdcd00ffa10 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdccffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 0x7fdcd0197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdccffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 0x7fdcd0197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49952/0 (socket says 172.21.15.116:49952) 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd49dc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd00ffa10 0x7fdcd0197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdccffff640 1 -- 172.21.15.116:0/3072775514 learned_addr learned my addr 172.21.15.116:0/3072775514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.023+0000 7fdcd51dd640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01033a0 0x7fdcd019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccffff640 1 -- 172.21.15.116:0/3072775514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd00ffa10 msgr2=0x7fdcd0197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccffff640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd00ffa10 0x7fdcd0197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccffff640 1 -- 172.21.15.116:0/3072775514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01033a0 msgr2=0x7fdcd019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccffff640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01033a0 0x7fdcd019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccffff640 1 -- 172.21.15.116:0/3072775514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc8009900 con 0x7fdcd01029e0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdcd51dd640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01033a0 0x7fdcd019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccffff640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 0x7fdcd0197b90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400e9f0 tx=0x7fdcc400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4009700 con 0x7fdcd01029e0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc400cd80 con 0x7fdcd01029e0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcd019f750 con 0x7fdcd01029e0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4018840 con 0x7fdcd01029e0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.024+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd019fc60 con 0x7fdcd01029e0 2024-03-24T09:20:05.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.025+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcd0108ab0 con 0x7fdcd01029e0 2024-03-24T09:20:05.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.026+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7fdcc40189a0 con 0x7fdcd01029e0 2024-03-24T09:20:05.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.027+0000 7fdccdffb640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdcb80a20c0 0x7fdcb80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.027+0000 7fdcd49dc640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdcb80a20c0 0x7fdcb80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.027+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdcc4014070 con 0x7fdcd01029e0 2024-03-24T09:20:05.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.028+0000 7fdcd49dc640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdcb80a20c0 0x7fdcb80a4580 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0069b80 tx=0x7fdcc0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:05.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.032+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdcc4053280 con 0x7fdcd01029e0 2024-03-24T09:20:05.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.231+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fdcd002d240 con 0x7fdcb80a20c0 2024-03-24T09:20:05.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:05.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2954510123' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T09:20:05.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2954510123' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T09:20:05.237 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] up:active 2024-03-24T09:20:05.238 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T09:20:05.238 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T09:20:05.238 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-03-24T09:20:05.238 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-mon[32207]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-03-24T09:20:05.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.256+0000 7fdccdffb640 1 -- 172.21.15.116:0/3072775514 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdcd002d240 con 0x7fdcb80a20c0 2024-03-24T09:20:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdcb80a20c0 msgr2=0x7fdcb80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdcb80a20c0 0x7fdcb80a4580 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdcd0069b80 tx=0x7fdcc0005f90 comp rx=0 tx=0).stop 2024-03-24T09:20:05.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 msgr2=0x7fdcd0197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 0x7fdcd0197b90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400e9f0 tx=0x7fdcc400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 shutdown_connections 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01033a0 0x7fdcd019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd01029e0 0x7fdcd0197b90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdcb80a20c0 0x7fdcb80a4580 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 --2- 172.21.15.116:0/3072775514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd00ffa10 0x7fdcd0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 >> 172.21.15.116:0/3072775514 conn(0x7fdcd00fb340 msgr2=0x7fdcd0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 shutdown_connections 2024-03-24T09:20:05.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.258+0000 7fdcd6c40640 1 -- 172.21.15.116:0/3072775514 wait complete. 2024-03-24T09:20:05.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-24T09:20:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2954510123' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T09:20:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2954510123' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T09:20:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] up:active 2024-03-24T09:20:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T09:20:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T09:20:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-03-24T09:20:05.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:05 smithi143 ceph-mon[35965]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-03-24T09:20:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-24T09:20:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2954510123' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T09:20:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2954510123' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T09:20:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] up:active 2024-03-24T09:20:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T09:20:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=f=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T09:20:05.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:creating} 2 up:standby-replay 6 up:standby 2024-03-24T09:20:05.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:05 smithi117 ceph-mon[35788]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-03-24T09:20:05.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 -- 172.21.15.116:0/1896844439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb140ffa20 msgr2=0x7feb140ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 --2- 172.21.15.116:0/1896844439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb140ffa20 0x7feb140ffe40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7feb04009ca0 tx=0x7feb0402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 -- 172.21.15.116:0/1896844439 shutdown_connections 2024-03-24T09:20:05.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 --2- 172.21.15.116:0/1896844439 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb14103450 0x7feb14109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 --2- 172.21.15.116:0/1896844439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb14102a90 0x7feb14102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 --2- 172.21.15.116:0/1896844439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb140ffa20 0x7feb140ffe40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 -- 172.21.15.116:0/1896844439 >> 172.21.15.116:0/1896844439 conn(0x7feb140fb310 msgr2=0x7feb140fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 -- 172.21.15.116:0/1896844439 shutdown_connections 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 -- 172.21.15.116:0/1896844439 wait complete. 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 Processor -- start 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.376+0000 7feb195dd640 1 -- start start 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb140ffa20 0x7feb14197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb14102a90 0x7feb14197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 0x7feb1419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb1419f230 con 0x7feb14103450 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feb1419f3a0 con 0x7feb14102a90 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7feb1419f560 con 0x7feb140ffa20 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 0x7feb1419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 0x7feb1419ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49994/0 (socket says 172.21.15.116:49994) 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 -- 172.21.15.116:0/4227029988 learned_addr learned my addr 172.21.15.116:0/4227029988 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb12ffd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb140ffa20 0x7feb14197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb127fc640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb14102a90 0x7feb14197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 -- 172.21.15.116:0/4227029988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb140ffa20 msgr2=0x7feb14197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb140ffa20 0x7feb14197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 -- 172.21.15.116:0/4227029988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb14102a90 msgr2=0x7feb14197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb14102a90 0x7feb14197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 -- 172.21.15.116:0/4227029988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb04009900 con 0x7feb14103450 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb137fe640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 0x7feb1419ecf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00e970 tx=0x7feb0c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c009720 con 0x7feb14103450 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb0c0054d0 con 0x7feb14103450 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1419f840 con 0x7feb14103450 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c010430 con 0x7feb14103450 2024-03-24T09:20:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.377+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1419fd50 con 0x7feb14103450 2024-03-24T09:20:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.378+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb14108ce0 con 0x7feb14103450 2024-03-24T09:20:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.380+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7feb0c004f10 con 0x7feb14103450 2024-03-24T09:20:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.380+0000 7feafbfff640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feafc0a20c0 0x7feafc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.380+0000 7feb12ffd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feafc0a20c0 0x7feafc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.380+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feb0c014070 con 0x7feb14103450 2024-03-24T09:20:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.381+0000 7feb12ffd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feafc0a20c0 0x7feafc0a4580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feb0402c700 tx=0x7feb040325d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.384+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7feb0c0971b0 con 0x7feb14103450 2024-03-24T09:20:05.568 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:20:05.235+0000 7f8592cfa700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:20:05.568 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:20:05.235+0000 7f8592cfa700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:20:05.568 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:20:05.235+0000 7f8592cfa700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:20:05.568 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:20:05.235+0000 7f8592cfa700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:20:05.568 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:20:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:20:05.235+0000 7f8592cfa700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:20:05.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.584+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7feb1402d240 con 0x7feafc0a20c0 2024-03-24T09:20:05.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.610+0000 7feafbfff640 1 -- 172.21.15.116:0/4227029988 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7feb1402d240 con 0x7feafc0a20c0 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feafc0a20c0 msgr2=0x7feafc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feafc0a20c0 0x7feafc0a4580 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7feb0402c700 tx=0x7feb040325d0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 msgr2=0x7feb1419ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 0x7feb1419ecf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7feb0c00e970 tx=0x7feb0c00ed70 comp rx=0 tx=0).stop 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 shutdown_connections 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb14103450 0x7feb1419ecf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb14102a90 0x7feb14197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feafc0a20c0 0x7feafc0a4580 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 --2- 172.21.15.116:0/4227029988 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb140ffa20 0x7feb14197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 >> 172.21.15.116:0/4227029988 conn(0x7feb140fb310 msgr2=0x7feb14105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 shutdown_connections 2024-03-24T09:20:05.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.612+0000 7feb195dd640 1 -- 172.21.15.116:0/4227029988 wait complete. 2024-03-24T09:20:05.623 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-24T09:20:05.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 -- 172.21.15.116:0/3982501673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4102210 msgr2=0x7f8bd4104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/3982501673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4102210 0x7f8bd4104600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009860 tx=0x7f8bc402bb70 comp rx=0 tx=0).stop 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 -- 172.21.15.116:0/3982501673 shutdown_connections 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/3982501673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bd4107470 0x7f8bd4109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/3982501673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4104b40 0x7f8bd4106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/3982501673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4102210 0x7f8bd4104600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.729+0000 7f8bd8df7640 1 -- 172.21.15.116:0/3982501673 >> 172.21.15.116:0/3982501673 conn(0x7f8bd40fbba0 msgr2=0x7f8bd40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 -- 172.21.15.116:0/3982501673 shutdown_connections 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 -- 172.21.15.116:0/3982501673 wait complete. 2024-03-24T09:20:05.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 Processor -- start 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 -- start start 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 0x7f8bd4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bd4104b40 0x7f8bd4197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4107470 0x7f8bd419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd419f230 con 0x7f8bd4107470 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd419f3a0 con 0x7f8bd4102210 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.730+0000 7f8bd8df7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd419f560 con 0x7f8bd4104b40 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 0x7f8bd4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4107470 0x7f8bd419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 0x7f8bd4197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:38350/0 (socket says 172.21.15.116:38350) 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd259c640 1 -- 172.21.15.116:0/934894188 learned_addr learned my addr 172.21.15.116:0/934894188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bcbfff640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bd4104b40 0x7f8bd4197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 -- 172.21.15.116:0/934894188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bd4104b40 msgr2=0x7f8bd4197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bd4104b40 0x7f8bd4197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 -- 172.21.15.116:0/934894188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 msgr2=0x7f8bd4197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 0x7f8bd4197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 -- 172.21.15.116:0/934894188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc4009490 con 0x7f8bd4107470 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd259c640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 0x7f8bd4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd2d9d640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4107470 0x7f8bd419ecf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009fd0 tx=0x7f8bcc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:05.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc00cd60 con 0x7f8bd4107470 2024-03-24T09:20:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd419f840 con 0x7f8bd4107470 2024-03-24T09:20:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8bcc005020 con 0x7f8bd4107470 2024-03-24T09:20:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc010690 con 0x7f8bd4107470 2024-03-24T09:20:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.731+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd419fd50 con 0x7f8bd4107470 2024-03-24T09:20:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.732+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd40fd690 con 0x7f8bd4107470 2024-03-24T09:20:05.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.733+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f8bcc005190 con 0x7f8bd4107470 2024-03-24T09:20:05.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.734+0000 7f8bcb7fe640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8bc00a20c0 0x7f8bc00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:05.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.734+0000 7f8bd259c640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8bc00a20c0 0x7f8bc00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:05.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.734+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8bcc014070 con 0x7f8bd4107470 2024-03-24T09:20:05.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.735+0000 7f8bd259c640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8bc00a20c0 0x7f8bc00a4580 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8bc402c180 tx=0x7f8bc402e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:05.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.739+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8bcc097320 con 0x7f8bd4107470 2024-03-24T09:20:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.947+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f8bd402d240 con 0x7f8bc00a20c0 2024-03-24T09:20:05.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.952+0000 7f8bcb7fe640 1 -- 172.21.15.116:0/934894188 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f8bd402d240 con 0x7f8bc00a20c0 2024-03-24T09:20:05.953 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_0/f1e517f0-a347-4e9a-8ab1-7fed09784049 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8bc00a20c0 msgr2=0x7f8bc00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8bc00a20c0 0x7f8bc00a4580 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8bc402c180 tx=0x7f8bc402e040 comp rx=0 tx=0).stop 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4107470 msgr2=0x7f8bd419ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4107470 0x7f8bd419ecf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009fd0 tx=0x7f8bcc00ee50 comp rx=0 tx=0).stop 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 shutdown_connections 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bd4107470 0x7f8bd419ecf0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8bc00a20c0 0x7f8bc00a4580 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bd4104b40 0x7f8bd4197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 --2- 172.21.15.116:0/934894188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bd4102210 0x7f8bd4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 >> 172.21.15.116:0/934894188 conn(0x7f8bd40fbba0 msgr2=0x7f8bd4102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 shutdown_connections 2024-03-24T09:20:05.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:05.954+0000 7f8bd8df7640 1 -- 172.21.15.116:0/934894188 wait complete. 2024-03-24T09:20:05.966 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-24T09:20:06.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 -- 172.21.15.116:0/3871904058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4101300 msgr2=0x7f78a4101780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 --2- 172.21.15.116:0/3871904058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4101300 0x7f78a4101780 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f789c009ca0 tx=0x7f789c02c180 comp rx=0 tx=0).stop 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 -- 172.21.15.116:0/3871904058 shutdown_connections 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 --2- 172.21.15.116:0/3871904058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4101300 0x7f78a4101780 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 --2- 172.21.15.116:0/3871904058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100960 0x7f78a4100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 --2- 172.21.15.116:0/3871904058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a4100000 0x7f78a4100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 -- 172.21.15.116:0/3871904058 >> 172.21.15.116:0/3871904058 conn(0x7f78a40fbba0 msgr2=0x7f78a40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 -- 172.21.15.116:0/3871904058 shutdown_connections 2024-03-24T09:20:06.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.072+0000 7f78aa7af640 1 -- 172.21.15.116:0/3871904058 wait complete. 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 Processor -- start 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 -- start start 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100000 0x7f78a41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 0x7f78a4199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a4101300 0x7f78a41a0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41a13e0 con 0x7f78a4100960 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41a1550 con 0x7f78a4100000 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78aa7af640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41a1710 con 0x7f78a4101300 2024-03-24T09:20:06.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 0x7f78a4199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 0x7f78a4199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50038/0 (socket says 172.21.15.116:50038) 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78a37fe640 1 -- 172.21.15.116:0/1545575534 learned_addr learned my addr 172.21.15.116:0/1545575534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78a8d4c640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a4101300 0x7f78a41a0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.073+0000 7f78a3fff640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100000 0x7f78a41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a37fe640 1 -- 172.21.15.116:0/1545575534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a4101300 msgr2=0x7f78a41a0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a37fe640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a4101300 0x7f78a41a0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a37fe640 1 -- 172.21.15.116:0/1545575534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100000 msgr2=0x7f78a41998f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a37fe640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100000 0x7f78a41998f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a37fe640 1 -- 172.21.15.116:0/1545575534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7898009490 con 0x7f78a4100960 2024-03-24T09:20:06.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a3fff640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100000 0x7f78a41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a37fe640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 0x7f78a4199e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f789800ea60 tx=0x7f789800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789800cde0 con 0x7f78a4100960 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78980044d0 con 0x7f78a4100960 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789c009900 con 0x7f78a4100960 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7898010690 con 0x7f78a4100960 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.074+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a41a1dc0 con 0x7f78a4100960 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.075+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a410af90 con 0x7f78a4100960 2024-03-24T09:20:06.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.076+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f7898003e40 con 0x7f78a4100960 2024-03-24T09:20:06.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.077+0000 7f78a17fa640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f788c0a2140 0x7f788c0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.077+0000 7f78a3fff640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f788c0a2140 0x7f788c0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.077+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7898014070 con 0x7f78a4100960 2024-03-24T09:20:06.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.077+0000 7f78a3fff640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f788c0a2140 0x7f788c0a4600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7894009fc0 tx=0x7f7894005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:06.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.080+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78980971a0 con 0x7f78a4100960 2024-03-24T09:20:06.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.283+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f78a402d240 con 0x7f788c0a2140 2024-03-24T09:20:06.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.307+0000 7f78a17fa640 1 -- 172.21.15.116:0/1545575534 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f78a402d240 con 0x7f788c0a2140 2024-03-24T09:20:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f788c0a2140 msgr2=0x7f788c0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f788c0a2140 0x7f788c0a4600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7894009fc0 tx=0x7f7894005f90 comp rx=0 tx=0).stop 2024-03-24T09:20:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 msgr2=0x7f78a4199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 0x7f78a4199e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f789800ea60 tx=0x7f789800ef20 comp rx=0 tx=0).stop 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 shutdown_connections 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f788c0a2140 0x7f788c0a4600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78a4101300 0x7f78a41a0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78a4100960 0x7f78a4199e30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 --2- 172.21.15.116:0/1545575534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78a4100000 0x7f78a41998f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 >> 172.21.15.116:0/1545575534 conn(0x7f78a40fbba0 msgr2=0x7f78a40fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 shutdown_connections 2024-03-24T09:20:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.309+0000 7f78aa7af640 1 -- 172.21.15.116:0/1545575534 wait complete. 2024-03-24T09:20:06.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-24T09:20:06.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 -- 172.21.15.116:0/3345866960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4100630 msgr2=0x7f04a4100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 --2- 172.21.15.116:0/3345866960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4100630 0x7f04a4100a50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0494009900 tx=0x7f049402bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 -- 172.21.15.116:0/3345866960 shutdown_connections 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 --2- 172.21.15.116:0/3345866960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41021e0 0x7f04a4108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 --2- 172.21.15.116:0/3345866960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04a4101820 0x7f04a4101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 --2- 172.21.15.116:0/3345866960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4100630 0x7f04a4100a50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 -- 172.21.15.116:0/3345866960 >> 172.21.15.116:0/3345866960 conn(0x7f04a40fbbc0 msgr2=0x7f04a40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:06.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 -- 172.21.15.116:0/3345866960 shutdown_connections 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.426+0000 7f04aa15e640 1 -- 172.21.15.116:0/3345866960 wait complete. 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 Processor -- start 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 -- start start 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04a4100630 0x7f04a4196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 0x7f04a4196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41021e0 0x7f04a419dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04a419e0f0 con 0x7f04a4101820 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f04a419e260 con 0x7f04a41021e0 2024-03-24T09:20:06.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04aa15e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f04a419e420 con 0x7f04a4100630 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f049affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 0x7f04a4196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f049affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 0x7f04a4196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50054/0 (socket says 172.21.15.116:50054) 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f049affd640 1 -- 172.21.15.116:0/2765258386 learned_addr learned my addr 172.21.15.116:0/2765258386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04a37fe640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04a4100630 0x7f04a4196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.427+0000 7f04a3fff640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41021e0 0x7f04a419dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f049affd640 1 -- 172.21.15.116:0/2765258386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04a4100630 msgr2=0x7f04a4196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f049affd640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04a4100630 0x7f04a4196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f049affd640 1 -- 172.21.15.116:0/2765258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41021e0 msgr2=0x7f04a419dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f049affd640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41021e0 0x7f04a419dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f049affd640 1 -- 172.21.15.116:0/2765258386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0494009560 con 0x7f04a4101820 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f049affd640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 0x7f04a4196b40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f048c00eb00 tx=0x7f048c00ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c013070 con 0x7f04a4101820 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a4069d80 con 0x7f04a4101820 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f048c0054d0 con 0x7f04a4101820 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a406a240 con 0x7f04a4101820 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.428+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c004d00 con 0x7f04a4101820 2024-03-24T09:20:06.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.429+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a4068950 con 0x7f04a4101820 2024-03-24T09:20:06.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.430+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f048c00bcc0 con 0x7f04a4101820 2024-03-24T09:20:06.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.431+0000 7f04a17fa640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f04900a20c0 0x7f04900a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.431+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f048c0cfdc0 con 0x7f04a4101820 2024-03-24T09:20:06.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.431+0000 7f04a37fe640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f04900a20c0 0x7f04900a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.431+0000 7f04a37fe640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f04900a20c0 0x7f04900a4580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f049402c7d0 tx=0x7f0494005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:06.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.434+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f048c098c00 con 0x7f04a4101820 2024-03-24T09:20:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.638+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f04a4107540 con 0x7f04900a20c0 2024-03-24T09:20:06.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.643+0000 7f04a17fa640 1 -- 172.21.15.116:0/2765258386 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f04a4107540 con 0x7f04900a20c0 2024-03-24T09:20:06.644 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_1/d846435b-34e1-4ca3-8aaf-8eb21a3c45da 2024-03-24T09:20:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f04900a20c0 msgr2=0x7f04900a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f04900a20c0 0x7f04900a4580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f049402c7d0 tx=0x7f0494005fd0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 msgr2=0x7f04a4196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 0x7f04a4196b40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f048c00eb00 tx=0x7f048c00ef00 comp rx=0 tx=0).stop 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 shutdown_connections 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41021e0 0x7f04a419dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a4101820 0x7f04a4196b40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f04900a20c0 0x7f04900a4580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 --2- 172.21.15.116:0/2765258386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04a4100630 0x7f04a4196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 >> 172.21.15.116:0/2765258386 conn(0x7f04a40fbbc0 msgr2=0x7f04a4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 shutdown_connections 2024-03-24T09:20:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.645+0000 7f04aa15e640 1 -- 172.21.15.116:0/2765258386 wait complete. 2024-03-24T09:20:06.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:20:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 -- 172.21.15.116:0/1193245112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088102230 msgr2=0x7f1088104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1193245112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088102230 0x7f1088104620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1078009860 tx=0x7f107802bb70 comp rx=0 tx=0).stop 2024-03-24T09:20:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 -- 172.21.15.116:0/1193245112 shutdown_connections 2024-03-24T09:20:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1193245112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088107490 0x7f10881098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1193245112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088104b60 0x7f1088106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1193245112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088102230 0x7f1088104620 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 -- 172.21.15.116:0/1193245112 >> 172.21.15.116:0/1193245112 conn(0x7f10880fbc40 msgr2=0x7f10880fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 -- 172.21.15.116:0/1193245112 shutdown_connections 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.760+0000 7f108dfe4640 1 -- 172.21.15.116:0/1193245112 wait complete. 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 Processor -- start 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 -- start start 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088102230 0x7f1088197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088104b60 0x7f1088197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 0x7f108819ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f108819f210 con 0x7f1088107490 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f108819f380 con 0x7f1088104b60 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f108dfe4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f108819f540 con 0x7f1088102230 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 0x7f108819ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f10877fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088102230 0x7f1088197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 0x7f108819ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34442/0 (socket says 172.21.15.116:34442) 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 -- 172.21.15.116:0/1716264360 learned_addr learned my addr 172.21.15.116:0/1716264360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1086ffd640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088104b60 0x7f1088197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 -- 172.21.15.116:0/1716264360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088102230 msgr2=0x7f1088197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088102230 0x7f1088197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 -- 172.21.15.116:0/1716264360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088104b60 msgr2=0x7f1088197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088104b60 0x7f1088197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f1087fff640 1 -- 172.21.15.116:0/1716264360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f107c009560 con 0x7f1088107490 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.761+0000 7f10877fe640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088102230 0x7f1088197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.762+0000 7f1087fff640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 0x7f108819ecd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f108000eb10 tx=0x7f108000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.762+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080004b20 con 0x7f1088107490 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.762+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1078009490 con 0x7f1088107490 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.762+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1080004c80 con 0x7f1088107490 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.762+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1080010690 con 0x7f1088107490 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.762+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f108819fba0 con 0x7f1088107490 2024-03-24T09:20:06.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.763+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10880fd730 con 0x7f1088107490 2024-03-24T09:20:06.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.764+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 136757+0+0 (secure 0 0 0) 0x7f1080005020 con 0x7f1088107490 2024-03-24T09:20:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.764+0000 7f1084ff9640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f10700a20c0 0x7f10700a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.764+0000 7f10877fe640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f10700a20c0 0x7f10700a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.764+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1080014070 con 0x7f1088107490 2024-03-24T09:20:06.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.765+0000 7f10877fe640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f10700a20c0 0x7f10700a4580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f107802c180 tx=0x7f107802e040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:06.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.769+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1080097220 con 0x7f1088107490 2024-03-24T09:20:06.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.923+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f10800975f0 con 0x7f1088107490 2024-03-24T09:20:06.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.990+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10880fd1a0 con 0x7f1088107490 2024-03-24T09:20:06.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.991+0000 7f1084ff9640 1 -- 172.21.15.116:0/1716264360 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f1080055d60 con 0x7f1088107490 2024-03-24T09:20:06.992 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:06.992 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:20:06.925494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:20:06.992 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:20:06.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f10700a20c0 msgr2=0x7f10700a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f10700a20c0 0x7f10700a4580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f107802c180 tx=0x7f107802e040 comp rx=0 tx=0).stop 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 msgr2=0x7f108819ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 0x7f108819ecd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f108000eb10 tx=0x7f108000c2d0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 shutdown_connections 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1088107490 0x7f108819ecd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1088104b60 0x7f1088197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f10700a20c0 0x7f10700a4580 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 --2- 172.21.15.116:0/1716264360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1088102230 0x7f1088197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.993+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 >> 172.21.15.116:0/1716264360 conn(0x7f10880fbc40 msgr2=0x7f1088102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:06.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.994+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 shutdown_connections 2024-03-24T09:20:06.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:06.994+0000 7f108dfe4640 1 -- 172.21.15.116:0/1716264360 wait complete. 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] up:active 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: Cluster is now healthy 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T09:20:07.004 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-24T09:20:07.005 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-24T09:20:07.005 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-24T09:20:07.005 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:06 smithi116 ceph-mon[32207]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 -- 172.21.15.116:0/3786406393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 msgr2=0x7fdef41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 --2- 172.21.15.116:0/3786406393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef41098c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009900 tx=0x7fdedc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 -- 172.21.15.116:0/3786406393 shutdown_connections 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 --2- 172.21.15.116:0/3786406393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef41098c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 --2- 172.21.15.116:0/3786406393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef4104ba0 0x7fdef4106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 --2- 172.21.15.116:0/3786406393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4102270 0x7fdef4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 -- 172.21.15.116:0/3786406393 >> 172.21.15.116:0/3786406393 conn(0x7fdef40fbba0 msgr2=0x7fdef40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:07.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 -- 172.21.15.116:0/3786406393 shutdown_connections 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 -- 172.21.15.116:0/3786406393 wait complete. 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 Processor -- start 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.108+0000 7fdefae09640 1 -- start start 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef4102270 0x7fdef41976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4104ba0 0x7fdef4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdef419f1d0 con 0x7fdef41074d0 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdef419f340 con 0x7fdef4104ba0 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdef419f500 con 0x7fdef4102270 2024-03-24T09:20:07.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef8ba5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef4102270 0x7fdef41976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef419ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34468/0 (socket says 172.21.15.116:34468) 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4104ba0 0x7fdef4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 -- 172.21.15.116:0/3219627815 learned_addr learned my addr 172.21.15.116:0/3219627815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 -- 172.21.15.116:0/3219627815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef4102270 msgr2=0x7fdef41976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef4102270 0x7fdef41976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 -- 172.21.15.116:0/3219627815 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4104ba0 msgr2=0x7fdef4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4104ba0 0x7fdef4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 -- 172.21.15.116:0/3219627815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeec009840 con 0x7fdef41074d0 2024-03-24T09:20:07.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef3fff640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4104ba0 0x7fdef4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef93a6640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef419ec90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009230 tx=0x7fdedc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc0044e0 con 0x7fdef41074d0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdedc02de90 con 0x7fdef41074d0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc009560 con 0x7fdef41074d0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.109+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc0053c0 con 0x7fdef41074d0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.110+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef419fb00 con 0x7fdef41074d0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.110+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef40fd690 con 0x7fdef41074d0 2024-03-24T09:20:07.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.111+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fdedc005520 con 0x7fdef41074d0 2024-03-24T09:20:07.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.112+0000 7fdef1ffb640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdee00a2150 0x7fdee00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.112+0000 7fdef8ba5640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdee00a2150 0x7fdee00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.112+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdedc0f0100 con 0x7fdef41074d0 2024-03-24T09:20:07.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.112+0000 7fdef8ba5640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdee00a2150 0x7fdee00a4610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdeec005fd0 tx=0x7fdeec009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:07.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.116+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdedc0b8e40 con 0x7fdef41074d0 2024-03-24T09:20:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] up:active 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: Cluster is now healthy 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-24T09:20:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-24T09:20:07.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:06 smithi143 ceph-mon[35965]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.290+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdef402d1d0 con 0x7fdef41074d0 2024-03-24T09:20:07.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.292+0000 7fdef1ffb640 1 -- 172.21.15.116:0/3219627815 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7fdedc0bc500 con 0x7fdef41074d0 2024-03-24T09:20:07.293 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:07.293 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdee00a2150 msgr2=0x7fdee00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdee00a2150 0x7fdee00a4610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdeec005fd0 tx=0x7fdeec009500 comp rx=0 tx=0).stop 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 msgr2=0x7fdef419ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef419ec90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdedc009230 tx=0x7fdedc0042f0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 shutdown_connections 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef41074d0 0x7fdef419ec90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdef4104ba0 0x7fdef4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdee00a2150 0x7fdee00a4610 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 --2- 172.21.15.116:0/3219627815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdef4102270 0x7fdef41976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 >> 172.21.15.116:0/3219627815 conn(0x7fdef40fbba0 msgr2=0x7fdef4102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 shutdown_connections 2024-03-24T09:20:07.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.294+0000 7fdefae09640 1 -- 172.21.15.116:0/3219627815 wait complete. 2024-03-24T09:20:07.306 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T09:20:07.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T09:20:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] up:active 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: Cluster is now healthy 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=f=up:active,3=e=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-24T09:20:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:06 smithi117 ceph-mon[35788]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 -- 172.21.15.116:0/3249691829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110101740 msgr2=0x7ff110101b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 --2- 172.21.15.116:0/3249691829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110101740 0x7ff110101b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff104009900 tx=0x7ff10402bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 -- 172.21.15.116:0/3249691829 shutdown_connections 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 --2- 172.21.15.116:0/3249691829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1101032f0 0x7ff110109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 --2- 172.21.15.116:0/3249691829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff110102930 0x7ff110102db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 --2- 172.21.15.116:0/3249691829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110101740 0x7ff110101b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 -- 172.21.15.116:0/3249691829 >> 172.21.15.116:0/3249691829 conn(0x7ff1100fccf0 msgr2=0x7ff1100ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:07.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 -- 172.21.15.116:0/3249691829 shutdown_connections 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 -- 172.21.15.116:0/3249691829 wait complete. 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 Processor -- start 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 -- start start 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.407+0000 7ff116c47640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff110101740 0x7ff1101977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 0x7ff110197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1101032f0 0x7ff11019ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff11019f240 con 0x7ff110102930 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff11019f3b0 con 0x7ff110101740 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff11019f570 con 0x7ff1101032f0 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 0x7ff110197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff1149e3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff110101740 0x7ff1101977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 0x7ff110197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34484/0 (socket says 172.21.15.116:34484) 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 -- 172.21.15.116:0/947618498 learned_addr learned my addr 172.21.15.116:0/947618498 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff1151e4640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1101032f0 0x7ff11019ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 -- 172.21.15.116:0/947618498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1101032f0 msgr2=0x7ff11019ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1101032f0 0x7ff11019ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 -- 172.21.15.116:0/947618498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff110101740 msgr2=0x7ff1101977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff110101740 0x7ff1101977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 -- 172.21.15.116:0/947618498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0fc009490 con 0x7ff110102930 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff1149e3640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff110101740 0x7ff1101977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10ffff640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 0x7ff110197ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00ea60 tx=0x7ff0fc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:07.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00cd20 con 0x7ff110102930 2024-03-24T09:20:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff108009840 con 0x7ff110102930 2024-03-24T09:20:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0fc0044d0 con 0x7ff110102930 2024-03-24T09:20:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc010430 con 0x7ff110102930 2024-03-24T09:20:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.408+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff104009560 con 0x7ff110102930 2024-03-24T09:20:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.409+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff110108bd0 con 0x7ff110102930 2024-03-24T09:20:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.410+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7ff0fc003780 con 0x7ff110102930 2024-03-24T09:20:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.411+0000 7ff10dffb640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1000a2150 0x7ff1000a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.411+0000 7ff1149e3640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1000a2150 0x7ff1000a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.411+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff0fc014070 con 0x7ff110102930 2024-03-24T09:20:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.411+0000 7ff1149e3640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1000a2150 0x7ff1000a4610 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff10402c7d0 tx=0x7ff104005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:07.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.415+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff0fc097100 con 0x7ff110102930 2024-03-24T09:20:07.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.586+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff110061b50 con 0x7ff1000a2150 2024-03-24T09:20:07.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.595+0000 7ff10dffb640 1 -- 172.21.15.116:0/947618498 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340625 (secure 0 0 0) 0x7ff110061b50 con 0x7ff1000a2150 2024-03-24T09:20:07.603 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:07.606 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:20:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1000a2150 msgr2=0x7ff1000a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1000a2150 0x7ff1000a4610 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff10402c7d0 tx=0x7ff104005fd0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 msgr2=0x7ff110197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 0x7ff110197ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00ea60 tx=0x7ff0fc00ee60 comp rx=0 tx=0).stop 2024-03-24T09:20:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 shutdown_connections 2024-03-24T09:20:07.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff1000a2150 0x7ff1000a4610 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1101032f0 0x7ff11019ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff110102930 0x7ff110197ce0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 --2- 172.21.15.116:0/947618498 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff110101740 0x7ff1101977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 >> 172.21.15.116:0/947618498 conn(0x7ff1100fccf0 msgr2=0x7ff110105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 shutdown_connections 2024-03-24T09:20:07.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.599+0000 7ff116c47640 1 -- 172.21.15.116:0/947618498 wait complete. 2024-03-24T09:20:07.610 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-03-24T09:20:05.926077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598352,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":149,"num_read_kb":131,"num_write":310,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":253,"ondisk_log_size":253,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377408,"kb_used_data":11600,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564416,"statfs":{"total":1151940427776,"available":1151553961984,"internally_reserved":0,"allocated":11878400,"data_stored":7515878,"data_compressed":283428,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3485,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":82,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003885"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211311+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:19:44.211311+0000","last_peered":"2024-03-24T09:19:44.211311+0000","last_clean":"2024-03-24T09:19:44.211311+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:19:44.211311+0000","last_undegraded":"2024-03-24T09:19:44.211311+0000","last_fullsized":"2024-03-24T09:19:44.211311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212391+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:19:44.212391+0000","last_peered":"2024-03-24T09:19:44.212391+0000","last_clean":"2024-03-24T09:19:44.212391+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:19:44.212391+0000","last_undegraded":"2024-03-24T09:19:44.212391+0000","last_fullsized":"2024-03-24T09:19:44.212391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211636+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:19:44.211636+0000","last_peered":"2024-03-24T09:19:44.211636+0000","last_clean":"2024-03-24T09:19:44.211636+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:19:44.211636+0000","last_undegraded":"2024-03-24T09:19:44.211636+0000","last_fullsized":"2024-03-24T09:19:44.211636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211599+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:19:44.211599+0000","last_peered":"2024-03-24T09:19:44.211599+0000","last_clean":"2024-03-24T09:19:44.211599+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:19:44.211599+0000","last_undegraded":"2024-03-24T09:19:44.211599+0000","last_fullsized":"2024-03-24T09:19:44.211599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211092+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:19:44.211092+0000","last_peered":"2024-03-24T09:19:44.211092+0000","last_clean":"2024-03-24T09:19:44.211092+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:19:44.211092+0000","last_undegraded":"2024-03-24T09:19:44.211092+0000","last_fullsized":"2024-03-24T09:19:44.211092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726099+0000","last_change":"2024-03-24T09:19:38.724773+0000","last_active":"2024-03-24T09:19:43.726099+0000","last_peered":"2024-03-24T09:19:43.726099+0000","last_clean":"2024-03-24T09:19:43.726099+0000","last_became_active":"2024-03-24T09:19:38.724398+0000","last_became_peered":"2024-03-24T09:19:38.724398+0000","last_unstale":"2024-03-24T09:19:43.726099+0000","last_undegraded":"2024-03-24T09:19:43.726099+0000","last_fullsized":"2024-03-24T09:19:43.726099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:28.883052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040130+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:19:44.040130+0000","last_peered":"2024-03-24T09:19:44.040130+0000","last_clean":"2024-03-24T09:19:44.040130+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:19:44.040130+0000","last_undegraded":"2024-03-24T09:19:44.040130+0000","last_fullsized":"2024-03-24T09:19:44.040130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211002+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:19:44.211002+0000","last_peered":"2024-03-24T09:19:44.211002+0000","last_clean":"2024-03-24T09:19:44.211002+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:19:44.211002+0000","last_undegraded":"2024-03-24T09:19:44.211002+0000","last_fullsized":"2024-03-24T09:19:44.211002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211221+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:19:44.211221+0000","last_peered":"2024-03-24T09:19:44.211221+0000","last_clean":"2024-03-24T09:19:44.211221+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:19:44.211221+0000","last_undegraded":"2024-03-24T09:19:44.211221+0000","last_fullsized":"2024-03-24T09:19:44.211221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726444+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:19:43.726444+0000","last_peered":"2024-03-24T09:19:43.726444+0000","last_clean":"2024-03-24T09:19:43.726444+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:19:43.726444+0000","last_undegraded":"2024-03-24T09:19:43.726444+0000","last_fullsized":"2024-03-24T09:19:43.726444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040468+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:19:44.040468+0000","last_peered":"2024-03-24T09:19:44.040468+0000","last_clean":"2024-03-24T09:19:44.040468+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:19:44.040468+0000","last_undegraded":"2024-03-24T09:19:44.040468+0000","last_fullsized":"2024-03-24T09:19:44.040468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211494+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:19:44.211494+0000","last_peered":"2024-03-24T09:19:44.211494+0000","last_clean":"2024-03-24T09:19:44.211494+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:19:44.211494+0000","last_undegraded":"2024-03-24T09:19:44.211494+0000","last_fullsized":"2024-03-24T09:19:44.211494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211138+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:19:44.211138+0000","last_peered":"2024-03-24T09:19:44.211138+0000","last_clean":"2024-03-24T09:19:44.211138+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:19:44.211138+0000","last_undegraded":"2024-03-24T09:19:44.211138+0000","last_fullsized":"2024-03-24T09:19:44.211138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211141+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:19:44.211141+0000","last_peered":"2024-03-24T09:19:44.211141+0000","last_clean":"2024-03-24T09:19:44.211141+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:19:44.211141+0000","last_undegraded":"2024-03-24T09:19:44.211141+0000","last_fullsized":"2024-03-24T09:19:44.211141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211879+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:19:44.211879+0000","last_peered":"2024-03-24T09:19:44.211879+0000","last_clean":"2024-03-24T09:19:44.211879+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:19:44.211879+0000","last_undegraded":"2024-03-24T09:19:44.211879+0000","last_fullsized":"2024-03-24T09:19:44.211879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210533+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:19:44.210533+0000","last_peered":"2024-03-24T09:19:44.210533+0000","last_clean":"2024-03-24T09:19:44.210533+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:19:44.210533+0000","last_undegraded":"2024-03-24T09:19:44.210533+0000","last_fullsized":"2024-03-24T09:19:44.210533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211087+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:19:44.211087+0000","last_peered":"2024-03-24T09:19:44.211087+0000","last_clean":"2024-03-24T09:19:44.211087+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:19:44.211087+0000","last_undegraded":"2024-03-24T09:19:44.211087+0000","last_fullsized":"2024-03-24T09:19:44.211087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726691+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:19:43.726691+0000","last_peered":"2024-03-24T09:19:43.726691+0000","last_clean":"2024-03-24T09:19:43.726691+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:19:43.726691+0000","last_undegraded":"2024-03-24T09:19:43.726691+0000","last_fullsized":"2024-03-24T09:19:43.726691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726452+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:19:43.726452+0000","last_peered":"2024-03-24T09:19:43.726452+0000","last_clean":"2024-03-24T09:19:43.726452+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:19:43.726452+0000","last_undegraded":"2024-03-24T09:19:43.726452+0000","last_fullsized":"2024-03-24T09:19:43.726452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726075+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:19:43.726075+0000","last_peered":"2024-03-24T09:19:43.726075+0000","last_clean":"2024-03-24T09:19:43.726075+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:19:43.726075+0000","last_undegraded":"2024-03-24T09:19:43.726075+0000","last_fullsized":"2024-03-24T09:19:43.726075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.912517+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:20:04.912517+0000","last_peered":"2024-03-24T09:20:04.912517+0000","last_clean":"2024-03-24T09:20:04.912517+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:20:04.912517+0000","last_undegraded":"2024-03-24T09:20:04.912517+0000","last_fullsized":"2024-03-24T09:20:04.912517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212660+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:19:44.212660+0000","last_peered":"2024-03-24T09:19:44.212660+0000","last_clean":"2024-03-24T09:19:44.212660+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:19:44.212660+0000","last_undegraded":"2024-03-24T09:19:44.212660+0000","last_fullsized":"2024-03-24T09:19:44.212660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210728+0000","last_change":"2024-03-24T09:19:42.746142+0000","last_active":"2024-03-24T09:19:44.210728+0000","last_peered":"2024-03-24T09:19:44.210728+0000","last_clean":"2024-03-24T09:19:44.210728+0000","last_became_active":"2024-03-24T09:19:42.745735+0000","last_became_peered":"2024-03-24T09:19:42.745735+0000","last_unstale":"2024-03-24T09:19:44.210728+0000","last_undegraded":"2024-03-24T09:19:44.210728+0000","last_fullsized":"2024-03-24T09:19:44.210728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:33:28.908417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212800+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:19:44.212800+0000","last_peered":"2024-03-24T09:19:44.212800+0000","last_clean":"2024-03-24T09:19:44.212800+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:19:44.212800+0000","last_undegraded":"2024-03-24T09:19:44.212800+0000","last_fullsized":"2024-03-24T09:19:44.212800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211742+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:19:44.211742+0000","last_peered":"2024-03-24T09:19:44.211742+0000","last_clean":"2024-03-24T09:19:44.211742+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:19:44.211742+0000","last_undegraded":"2024-03-24T09:19:44.211742+0000","last_fullsized":"2024-03-24T09:19:44.211742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211284+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:19:44.211284+0000","last_peered":"2024-03-24T09:19:44.211284+0000","last_clean":"2024-03-24T09:19:44.211284+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:19:44.211284+0000","last_undegraded":"2024-03-24T09:19:44.211284+0000","last_fullsized":"2024-03-24T09:19:44.211284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.903185+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:20:03.903185+0000","last_peered":"2024-03-24T09:20:03.903185+0000","last_clean":"2024-03-24T09:20:03.903185+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:20:03.903185+0000","last_undegraded":"2024-03-24T09:20:03.903185+0000","last_fullsized":"2024-03-24T09:20:03.903185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211002+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:19:44.211002+0000","last_peered":"2024-03-24T09:19:44.211002+0000","last_clean":"2024-03-24T09:19:44.211002+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:19:44.211002+0000","last_undegraded":"2024-03-24T09:19:44.211002+0000","last_fullsized":"2024-03-24T09:19:44.211002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211581+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:19:44.211581+0000","last_peered":"2024-03-24T09:19:44.211581+0000","last_clean":"2024-03-24T09:19:44.211581+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:19:44.211581+0000","last_undegraded":"2024-03-24T09:19:44.211581+0000","last_fullsized":"2024-03-24T09:19:44.211581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211383+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:19:44.211383+0000","last_peered":"2024-03-24T09:19:44.211383+0000","last_clean":"2024-03-24T09:19:44.211383+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:19:44.211383+0000","last_undegraded":"2024-03-24T09:19:44.211383+0000","last_fullsized":"2024-03-24T09:19:44.211383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212492+0000","last_change":"2024-03-24T09:19:38.733015+0000","last_active":"2024-03-24T09:19:44.212492+0000","last_peered":"2024-03-24T09:19:44.212492+0000","last_clean":"2024-03-24T09:19:44.212492+0000","last_became_active":"2024-03-24T09:19:38.732691+0000","last_became_peered":"2024-03-24T09:19:38.732691+0000","last_unstale":"2024-03-24T09:19:44.212492+0000","last_undegraded":"2024-03-24T09:19:44.212492+0000","last_fullsized":"2024-03-24T09:19:44.212492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:50:02.190295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726450+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:19:43.726450+0000","last_peered":"2024-03-24T09:19:43.726450+0000","last_clean":"2024-03-24T09:19:43.726450+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:19:43.726450+0000","last_undegraded":"2024-03-24T09:19:43.726450+0000","last_fullsized":"2024-03-24T09:19:43.726450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.895768+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:20:03.895768+0000","last_peered":"2024-03-24T09:20:03.895768+0000","last_clean":"2024-03-24T09:20:03.895768+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:20:03.895768+0000","last_undegraded":"2024-03-24T09:20:03.895768+0000","last_fullsized":"2024-03-24T09:20:03.895768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211405+0000","last_change":"2024-03-24T09:19:38.725875+0000","last_active":"2024-03-24T09:19:44.211405+0000","last_peered":"2024-03-24T09:19:44.211405+0000","last_clean":"2024-03-24T09:19:44.211405+0000","last_became_active":"2024-03-24T09:19:38.725536+0000","last_became_peered":"2024-03-24T09:19:38.725536+0000","last_unstale":"2024-03-24T09:19:44.211405+0000","last_undegraded":"2024-03-24T09:19:44.211405+0000","last_fullsized":"2024-03-24T09:19:44.211405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:20:36.250703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726250+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:19:43.726250+0000","last_peered":"2024-03-24T09:19:43.726250+0000","last_clean":"2024-03-24T09:19:43.726250+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:19:43.726250+0000","last_undegraded":"2024-03-24T09:19:43.726250+0000","last_fullsized":"2024-03-24T09:19:43.726250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726050+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:19:43.726050+0000","last_peered":"2024-03-24T09:19:43.726050+0000","last_clean":"2024-03-24T09:19:43.726050+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:19:43.726050+0000","last_undegraded":"2024-03-24T09:19:43.726050+0000","last_fullsized":"2024-03-24T09:19:43.726050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212684+0000","last_change":"2024-03-24T09:19:38.734122+0000","last_active":"2024-03-24T09:19:44.212684+0000","last_peered":"2024-03-24T09:19:44.212684+0000","last_clean":"2024-03-24T09:19:44.212684+0000","last_became_active":"2024-03-24T09:19:38.733805+0000","last_became_peered":"2024-03-24T09:19:38.733805+0000","last_unstale":"2024-03-24T09:19:44.212684+0000","last_undegraded":"2024-03-24T09:19:44.212684+0000","last_fullsized":"2024-03-24T09:19:44.212684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:57.423822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210944+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:19:44.210944+0000","last_peered":"2024-03-24T09:19:44.210944+0000","last_clean":"2024-03-24T09:19:44.210944+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:19:44.210944+0000","last_undegraded":"2024-03-24T09:19:44.210944+0000","last_fullsized":"2024-03-24T09:19:44.210944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.904008+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:20:03.904008+0000","last_peered":"2024-03-24T09:20:03.904008+0000","last_clean":"2024-03-24T09:20:03.904008+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:20:03.904008+0000","last_undegraded":"2024-03-24T09:20:03.904008+0000","last_fullsized":"2024-03-24T09:20:03.904008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211329+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:19:44.211329+0000","last_peered":"2024-03-24T09:19:44.211329+0000","last_clean":"2024-03-24T09:19:44.211329+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:19:44.211329+0000","last_undegraded":"2024-03-24T09:19:44.211329+0000","last_fullsized":"2024-03-24T09:19:44.211329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212319+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:19:44.212319+0000","last_peered":"2024-03-24T09:19:44.212319+0000","last_clean":"2024-03-24T09:19:44.212319+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:19:44.212319+0000","last_undegraded":"2024-03-24T09:19:44.212319+0000","last_fullsized":"2024-03-24T09:19:44.212319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212118+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:19:44.212118+0000","last_peered":"2024-03-24T09:19:44.212118+0000","last_clean":"2024-03-24T09:19:44.212118+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:19:44.212118+0000","last_undegraded":"2024-03-24T09:19:44.212118+0000","last_fullsized":"2024-03-24T09:19:44.212118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726332+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:19:43.726332+0000","last_peered":"2024-03-24T09:19:43.726332+0000","last_clean":"2024-03-24T09:19:43.726332+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:19:43.726332+0000","last_undegraded":"2024-03-24T09:19:43.726332+0000","last_fullsized":"2024-03-24T09:19:43.726332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212523+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:19:44.212523+0000","last_peered":"2024-03-24T09:19:44.212523+0000","last_clean":"2024-03-24T09:19:44.212523+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:19:44.212523+0000","last_undegraded":"2024-03-24T09:19:44.212523+0000","last_fullsized":"2024-03-24T09:19:44.212523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726322+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:19:43.726322+0000","last_peered":"2024-03-24T09:19:43.726322+0000","last_clean":"2024-03-24T09:19:43.726322+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:19:43.726322+0000","last_undegraded":"2024-03-24T09:19:43.726322+0000","last_fullsized":"2024-03-24T09:19:43.726322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210962+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:19:44.210962+0000","last_peered":"2024-03-24T09:19:44.210962+0000","last_clean":"2024-03-24T09:19:44.210962+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:19:44.210962+0000","last_undegraded":"2024-03-24T09:19:44.210962+0000","last_fullsized":"2024-03-24T09:19:44.210962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726574+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:19:43.726574+0000","last_peered":"2024-03-24T09:19:43.726574+0000","last_clean":"2024-03-24T09:19:43.726574+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:19:43.726574+0000","last_undegraded":"2024-03-24T09:19:43.726574+0000","last_fullsized":"2024-03-24T09:19:43.726574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029691+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:20:03.029691+0000","last_peered":"2024-03-24T09:20:03.029691+0000","last_clean":"2024-03-24T09:20:03.029691+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:20:03.029691+0000","last_undegraded":"2024-03-24T09:20:03.029691+0000","last_fullsized":"2024-03-24T09:20:03.029691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211830+0000","last_change":"2024-03-24T09:19:38.730068+0000","last_active":"2024-03-24T09:19:44.211830+0000","last_peered":"2024-03-24T09:19:44.211830+0000","last_clean":"2024-03-24T09:19:44.211830+0000","last_became_active":"2024-03-24T09:19:38.729747+0000","last_became_peered":"2024-03-24T09:19:38.729747+0000","last_unstale":"2024-03-24T09:19:44.211830+0000","last_undegraded":"2024-03-24T09:19:44.211830+0000","last_fullsized":"2024-03-24T09:19:44.211830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:48.285768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211573+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:19:44.211573+0000","last_peered":"2024-03-24T09:19:44.211573+0000","last_clean":"2024-03-24T09:19:44.211573+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:19:44.211573+0000","last_undegraded":"2024-03-24T09:19:44.211573+0000","last_fullsized":"2024-03-24T09:19:44.211573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.031031+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:20:03.031031+0000","last_peered":"2024-03-24T09:20:03.031031+0000","last_clean":"2024-03-24T09:20:03.031031+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:20:03.031031+0000","last_undegraded":"2024-03-24T09:20:03.031031+0000","last_fullsized":"2024-03-24T09:20:03.031031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211115+0000","last_change":"2024-03-24T09:19:38.736495+0000","last_active":"2024-03-24T09:19:44.211115+0000","last_peered":"2024-03-24T09:19:44.211115+0000","last_clean":"2024-03-24T09:19:44.211115+0000","last_became_active":"2024-03-24T09:19:38.736113+0000","last_became_peered":"2024-03-24T09:19:38.736113+0000","last_unstale":"2024-03-24T09:19:44.211115+0000","last_undegraded":"2024-03-24T09:19:44.211115+0000","last_fullsized":"2024-03-24T09:19:44.211115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:32:45.254878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726606+0000","last_change":"2024-03-24T09:19:42.742649+0000","last_active":"2024-03-24T09:19:43.726606+0000","last_peered":"2024-03-24T09:19:43.726606+0000","last_clean":"2024-03-24T09:19:43.726606+0000","last_became_active":"2024-03-24T09:19:42.742256+0000","last_became_peered":"2024-03-24T09:19:42.742256+0000","last_unstale":"2024-03-24T09:19:43.726606+0000","last_undegraded":"2024-03-24T09:19:43.726606+0000","last_fullsized":"2024-03-24T09:19:43.726606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:46:59.749664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:54.107837+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:19:54.107837+0000","last_peered":"2024-03-24T09:19:54.107837+0000","last_clean":"2024-03-24T09:19:54.107837+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:19:54.107837+0000","last_undegraded":"2024-03-24T09:19:54.107837+0000","last_fullsized":"2024-03-24T09:19:54.107837+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211656+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:19:44.211656+0000","last_peered":"2024-03-24T09:19:44.211656+0000","last_clean":"2024-03-24T09:19:44.211656+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:19:44.211656+0000","last_undegraded":"2024-03-24T09:19:44.211656+0000","last_fullsized":"2024-03-24T09:19:44.211656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212796+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:19:44.212796+0000","last_peered":"2024-03-24T09:19:44.212796+0000","last_clean":"2024-03-24T09:19:44.212796+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:19:44.212796+0000","last_undegraded":"2024-03-24T09:19:44.212796+0000","last_fullsized":"2024-03-24T09:19:44.212796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.904231+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:20:03.904231+0000","last_peered":"2024-03-24T09:20:03.904231+0000","last_clean":"2024-03-24T09:20:03.904231+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:20:03.904231+0000","last_undegraded":"2024-03-24T09:20:03.904231+0000","last_fullsized":"2024-03-24T09:20:03.904231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211456+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:19:44.211456+0000","last_peered":"2024-03-24T09:19:44.211456+0000","last_clean":"2024-03-24T09:19:44.211456+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:19:44.211456+0000","last_undegraded":"2024-03-24T09:19:44.211456+0000","last_fullsized":"2024-03-24T09:19:44.211456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211343+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:19:44.211343+0000","last_peered":"2024-03-24T09:19:44.211343+0000","last_clean":"2024-03-24T09:19:44.211343+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:19:44.211343+0000","last_undegraded":"2024-03-24T09:19:44.211343+0000","last_fullsized":"2024-03-24T09:19:44.211343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.030554+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:20:03.030554+0000","last_peered":"2024-03-24T09:20:03.030554+0000","last_clean":"2024-03-24T09:20:03.030554+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:20:03.030554+0000","last_undegraded":"2024-03-24T09:20:03.030554+0000","last_fullsized":"2024-03-24T09:20:03.030554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040220+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:19:44.040220+0000","last_peered":"2024-03-24T09:19:44.040220+0000","last_clean":"2024-03-24T09:19:44.040220+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:19:44.040220+0000","last_undegraded":"2024-03-24T09:19:44.040220+0000","last_fullsized":"2024-03-24T09:19:44.040220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726821+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:19:43.726821+0000","last_peered":"2024-03-24T09:19:43.726821+0000","last_clean":"2024-03-24T09:19:43.726821+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:19:43.726821+0000","last_undegraded":"2024-03-24T09:19:43.726821+0000","last_fullsized":"2024-03-24T09:19:43.726821+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.905791+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:20:03.905791+0000","last_peered":"2024-03-24T09:20:03.905791+0000","last_clean":"2024-03-24T09:20:03.905791+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:20:03.905791+0000","last_undegraded":"2024-03-24T09:20:03.905791+0000","last_fullsized":"2024-03-24T09:20:03.905791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040149+0000","last_change":"2024-03-24T09:19:38.730496+0000","last_active":"2024-03-24T09:19:44.040149+0000","last_peered":"2024-03-24T09:19:44.040149+0000","last_clean":"2024-03-24T09:19:44.040149+0000","last_became_active":"2024-03-24T09:19:38.730164+0000","last_became_peered":"2024-03-24T09:19:38.730164+0000","last_unstale":"2024-03-24T09:19:44.040149+0000","last_undegraded":"2024-03-24T09:19:44.040149+0000","last_fullsized":"2024-03-24T09:19:44.040149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:51.277493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211773+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:19:44.211773+0000","last_peered":"2024-03-24T09:19:44.211773+0000","last_clean":"2024-03-24T09:19:44.211773+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:19:44.211773+0000","last_undegraded":"2024-03-24T09:19:44.211773+0000","last_fullsized":"2024-03-24T09:19:44.211773+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.905845+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:20:03.905845+0000","last_peered":"2024-03-24T09:20:03.905845+0000","last_clean":"2024-03-24T09:20:03.905845+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:20:03.905845+0000","last_undegraded":"2024-03-24T09:20:03.905845+0000","last_fullsized":"2024-03-24T09:20:03.905845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212426+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:19:44.212426+0000","last_peered":"2024-03-24T09:19:44.212426+0000","last_clean":"2024-03-24T09:19:44.212426+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:19:44.212426+0000","last_undegraded":"2024-03-24T09:19:44.212426+0000","last_fullsized":"2024-03-24T09:19:44.212426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212194+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:19:44.212194+0000","last_peered":"2024-03-24T09:19:44.212194+0000","last_clean":"2024-03-24T09:19:44.212194+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:19:44.212194+0000","last_undegraded":"2024-03-24T09:19:44.212194+0000","last_fullsized":"2024-03-24T09:19:44.212194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211796+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:19:44.211796+0000","last_peered":"2024-03-24T09:19:44.211796+0000","last_clean":"2024-03-24T09:19:44.211796+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:19:44.211796+0000","last_undegraded":"2024-03-24T09:19:44.211796+0000","last_fullsized":"2024-03-24T09:19:44.211796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210992+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:19:44.210992+0000","last_peered":"2024-03-24T09:19:44.210992+0000","last_clean":"2024-03-24T09:19:44.210992+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:19:44.210992+0000","last_undegraded":"2024-03-24T09:19:44.210992+0000","last_fullsized":"2024-03-24T09:19:44.210992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211385+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:19:44.211385+0000","last_peered":"2024-03-24T09:19:44.211385+0000","last_clean":"2024-03-24T09:19:44.211385+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:19:44.211385+0000","last_undegraded":"2024-03-24T09:19:44.211385+0000","last_fullsized":"2024-03-24T09:19:44.211385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726715+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:19:43.726715+0000","last_peered":"2024-03-24T09:19:43.726715+0000","last_clean":"2024-03-24T09:19:43.726715+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:19:43.726715+0000","last_undegraded":"2024-03-24T09:19:43.726715+0000","last_fullsized":"2024-03-24T09:19:43.726715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726308+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:19:43.726308+0000","last_peered":"2024-03-24T09:19:43.726308+0000","last_clean":"2024-03-24T09:19:43.726308+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:19:43.726308+0000","last_undegraded":"2024-03-24T09:19:43.726308+0000","last_fullsized":"2024-03-24T09:19:43.726308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211871+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:19:44.211871+0000","last_peered":"2024-03-24T09:19:44.211871+0000","last_clean":"2024-03-24T09:19:44.211871+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:19:44.211871+0000","last_undegraded":"2024-03-24T09:19:44.211871+0000","last_fullsized":"2024-03-24T09:19:44.211871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040208+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:19:44.040208+0000","last_peered":"2024-03-24T09:19:44.040208+0000","last_clean":"2024-03-24T09:19:44.040208+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:19:44.040208+0000","last_undegraded":"2024-03-24T09:19:44.040208+0000","last_fullsized":"2024-03-24T09:19:44.040208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726128+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:19:43.726128+0000","last_peered":"2024-03-24T09:19:43.726128+0000","last_clean":"2024-03-24T09:19:43.726128+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:19:43.726128+0000","last_undegraded":"2024-03-24T09:19:43.726128+0000","last_fullsized":"2024-03-24T09:19:43.726128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211282+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:19:44.211282+0000","last_peered":"2024-03-24T09:19:44.211282+0000","last_clean":"2024-03-24T09:19:44.211282+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:19:44.211282+0000","last_undegraded":"2024-03-24T09:19:44.211282+0000","last_fullsized":"2024-03-24T09:19:44.211282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212258+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:19:44.212258+0000","last_peered":"2024-03-24T09:19:44.212258+0000","last_clean":"2024-03-24T09:19:44.212258+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:19:44.212258+0000","last_undegraded":"2024-03-24T09:19:44.212258+0000","last_fullsized":"2024-03-24T09:19:44.212258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.903195+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:20:03.903195+0000","last_peered":"2024-03-24T09:20:03.903195+0000","last_clean":"2024-03-24T09:20:03.903195+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:20:03.903195+0000","last_undegraded":"2024-03-24T09:20:03.903195+0000","last_fullsized":"2024-03-24T09:20:03.903195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040409+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:19:44.040409+0000","last_peered":"2024-03-24T09:19:44.040409+0000","last_clean":"2024-03-24T09:19:44.040409+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:19:44.040409+0000","last_undegraded":"2024-03-24T09:19:44.040409+0000","last_fullsized":"2024-03-24T09:19:44.040409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211556+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:19:44.211556+0000","last_peered":"2024-03-24T09:19:44.211556+0000","last_clean":"2024-03-24T09:19:44.211556+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:19:44.211556+0000","last_undegraded":"2024-03-24T09:19:44.211556+0000","last_fullsized":"2024-03-24T09:19:44.211556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.893846+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:20:03.893846+0000","last_peered":"2024-03-24T09:20:03.893846+0000","last_clean":"2024-03-24T09:20:03.893846+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:20:03.893846+0000","last_undegraded":"2024-03-24T09:20:03.893846+0000","last_fullsized":"2024-03-24T09:20:03.893846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.039985+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:19:44.039985+0000","last_peered":"2024-03-24T09:19:44.039985+0000","last_clean":"2024-03-24T09:19:44.039985+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:19:44.039985+0000","last_undegraded":"2024-03-24T09:19:44.039985+0000","last_fullsized":"2024-03-24T09:19:44.039985+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212483+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:19:44.212483+0000","last_peered":"2024-03-24T09:19:44.212483+0000","last_clean":"2024-03-24T09:19:44.212483+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:19:44.212483+0000","last_undegraded":"2024-03-24T09:19:44.212483+0000","last_fullsized":"2024-03-24T09:19:44.212483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210865+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:19:44.210865+0000","last_peered":"2024-03-24T09:19:44.210865+0000","last_clean":"2024-03-24T09:19:44.210865+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:19:44.210865+0000","last_undegraded":"2024-03-24T09:19:44.210865+0000","last_fullsized":"2024-03-24T09:19:44.210865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.905065+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:20:03.905065+0000","last_peered":"2024-03-24T09:20:03.905065+0000","last_clean":"2024-03-24T09:20:03.905065+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:20:03.905065+0000","last_undegraded":"2024-03-24T09:20:03.905065+0000","last_fullsized":"2024-03-24T09:20:03.905065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210800+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:19:44.210800+0000","last_peered":"2024-03-24T09:19:44.210800+0000","last_clean":"2024-03-24T09:19:44.210800+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:19:44.210800+0000","last_undegraded":"2024-03-24T09:19:44.210800+0000","last_fullsized":"2024-03-24T09:19:44.210800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029322+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:20:03.029322+0000","last_peered":"2024-03-24T09:20:03.029322+0000","last_clean":"2024-03-24T09:20:03.029322+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:20:03.029322+0000","last_undegraded":"2024-03-24T09:20:03.029322+0000","last_fullsized":"2024-03-24T09:20:03.029322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212123+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:19:44.212123+0000","last_peered":"2024-03-24T09:19:44.212123+0000","last_clean":"2024-03-24T09:19:44.212123+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:19:44.212123+0000","last_undegraded":"2024-03-24T09:19:44.212123+0000","last_fullsized":"2024-03-24T09:19:44.212123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211279+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:19:44.211279+0000","last_peered":"2024-03-24T09:19:44.211279+0000","last_clean":"2024-03-24T09:19:44.211279+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:19:44.211279+0000","last_undegraded":"2024-03-24T09:19:44.211279+0000","last_fullsized":"2024-03-24T09:19:44.211279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.912972+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:20:04.912972+0000","last_peered":"2024-03-24T09:20:04.912972+0000","last_clean":"2024-03-24T09:20:04.912972+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:20:04.912972+0000","last_undegraded":"2024-03-24T09:20:04.912972+0000","last_fullsized":"2024-03-24T09:20:04.912972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212582+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:19:44.212582+0000","last_peered":"2024-03-24T09:19:44.212582+0000","last_clean":"2024-03-24T09:19:44.212582+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:19:44.212582+0000","last_undegraded":"2024-03-24T09:19:44.212582+0000","last_fullsized":"2024-03-24T09:19:44.212582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726107+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:19:43.726107+0000","last_peered":"2024-03-24T09:19:43.726107+0000","last_clean":"2024-03-24T09:19:43.726107+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:19:43.726107+0000","last_undegraded":"2024-03-24T09:19:43.726107+0000","last_fullsized":"2024-03-24T09:19:43.726107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029183+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:20:03.029183+0000","last_peered":"2024-03-24T09:20:03.029183+0000","last_clean":"2024-03-24T09:20:03.029183+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:20:03.029183+0000","last_undegraded":"2024-03-24T09:20:03.029183+0000","last_fullsized":"2024-03-24T09:20:03.029183+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212293+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:19:44.212293+0000","last_peered":"2024-03-24T09:19:44.212293+0000","last_clean":"2024-03-24T09:19:44.212293+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:19:44.212293+0000","last_undegraded":"2024-03-24T09:19:44.212293+0000","last_fullsized":"2024-03-24T09:19:44.212293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211587+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:19:44.211587+0000","last_peered":"2024-03-24T09:19:44.211587+0000","last_clean":"2024-03-24T09:19:44.211587+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:19:44.211587+0000","last_undegraded":"2024-03-24T09:19:44.211587+0000","last_fullsized":"2024-03-24T09:19:44.211587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029752+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:20:03.029752+0000","last_peered":"2024-03-24T09:20:03.029752+0000","last_clean":"2024-03-24T09:20:03.029752+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:20:03.029752+0000","last_undegraded":"2024-03-24T09:20:03.029752+0000","last_fullsized":"2024-03-24T09:20:03.029752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210964+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:19:44.210964+0000","last_peered":"2024-03-24T09:19:44.210964+0000","last_clean":"2024-03-24T09:19:44.210964+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:19:44.210964+0000","last_undegraded":"2024-03-24T09:19:44.210964+0000","last_fullsized":"2024-03-24T09:19:44.210964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.727002+0000","last_change":"2024-03-24T09:19:42.743462+0000","last_active":"2024-03-24T09:19:43.727002+0000","last_peered":"2024-03-24T09:19:43.727002+0000","last_clean":"2024-03-24T09:19:43.727002+0000","last_became_active":"2024-03-24T09:19:42.743078+0000","last_became_peered":"2024-03-24T09:19:42.743078+0000","last_unstale":"2024-03-24T09:19:43.727002+0000","last_undegraded":"2024-03-24T09:19:43.727002+0000","last_fullsized":"2024-03-24T09:19:43.727002+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:49.958507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.030464+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:20:03.030464+0000","last_peered":"2024-03-24T09:20:03.030464+0000","last_clean":"2024-03-24T09:20:03.030464+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:20:03.030464+0000","last_undegraded":"2024-03-24T09:20:03.030464+0000","last_fullsized":"2024-03-24T09:20:03.030464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210725+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:19:44.210725+0000","last_peered":"2024-03-24T09:19:44.210725+0000","last_clean":"2024-03-24T09:19:44.210725+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:19:44.210725+0000","last_undegraded":"2024-03-24T09:19:44.210725+0000","last_fullsized":"2024-03-24T09:19:44.210725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211583+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:19:44.211583+0000","last_peered":"2024-03-24T09:19:44.211583+0000","last_clean":"2024-03-24T09:19:44.211583+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:19:44.211583+0000","last_undegraded":"2024-03-24T09:19:44.211583+0000","last_fullsized":"2024-03-24T09:19:44.211583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.024425+0000","last_change":"2024-03-24T09:19:38.100940+0000","last_active":"2024-03-24T09:20:04.024425+0000","last_peered":"2024-03-24T09:20:04.024425+0000","last_clean":"2024-03-24T09:20:04.024425+0000","last_became_active":"2024-03-24T09:19:38.100613+0000","last_became_peered":"2024-03-24T09:19:38.100613+0000","last_unstale":"2024-03-24T09:20:04.024425+0000","last_undegraded":"2024-03-24T09:20:04.024425+0000","last_fullsized":"2024-03-24T09:20:04.024425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:53.483332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212013+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:19:44.212013+0000","last_peered":"2024-03-24T09:19:44.212013+0000","last_clean":"2024-03-24T09:19:44.212013+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:19:44.212013+0000","last_undegraded":"2024-03-24T09:19:44.212013+0000","last_fullsized":"2024-03-24T09:19:44.212013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211384+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:19:44.211384+0000","last_peered":"2024-03-24T09:19:44.211384+0000","last_clean":"2024-03-24T09:19:44.211384+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:19:44.211384+0000","last_undegraded":"2024-03-24T09:19:44.211384+0000","last_fullsized":"2024-03-24T09:19:44.211384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.030033+0000","last_change":"2024-03-24T09:19:38.105336+0000","last_active":"2024-03-24T09:20:03.030033+0000","last_peered":"2024-03-24T09:20:03.030033+0000","last_clean":"2024-03-24T09:20:03.030033+0000","last_became_active":"2024-03-24T09:19:38.103695+0000","last_became_peered":"2024-03-24T09:19:38.103695+0000","last_unstale":"2024-03-24T09:20:03.030033+0000","last_undegraded":"2024-03-24T09:20:03.030033+0000","last_fullsized":"2024-03-24T09:20:03.030033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:11:41.784391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040201+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:19:44.040201+0000","last_peered":"2024-03-24T09:19:44.040201+0000","last_clean":"2024-03-24T09:19:44.040201+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:19:44.040201+0000","last_undegraded":"2024-03-24T09:19:44.040201+0000","last_fullsized":"2024-03-24T09:19:44.040201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210863+0000","last_change":"2024-03-24T09:19:42.744163+0000","last_active":"2024-03-24T09:19:44.210863+0000","last_peered":"2024-03-24T09:19:44.210863+0000","last_clean":"2024-03-24T09:19:44.210863+0000","last_became_active":"2024-03-24T09:19:42.743754+0000","last_became_peered":"2024-03-24T09:19:42.743754+0000","last_unstale":"2024-03-24T09:19:44.210863+0000","last_undegraded":"2024-03-24T09:19:44.210863+0000","last_fullsized":"2024-03-24T09:19:44.210863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:07.082490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.908680+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:20:04.908680+0000","last_peered":"2024-03-24T09:20:04.908680+0000","last_clean":"2024-03-24T09:20:04.908680+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:20:04.908680+0000","last_undegraded":"2024-03-24T09:20:04.908680+0000","last_fullsized":"2024-03-24T09:20:04.908680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040563+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:19:44.040563+0000","last_peered":"2024-03-24T09:19:44.040563+0000","last_clean":"2024-03-24T09:19:44.040563+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:19:44.040563+0000","last_undegraded":"2024-03-24T09:19:44.040563+0000","last_fullsized":"2024-03-24T09:19:44.040563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212044+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:19:44.212044+0000","last_peered":"2024-03-24T09:19:44.212044+0000","last_clean":"2024-03-24T09:19:44.212044+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:19:44.212044+0000","last_undegraded":"2024-03-24T09:19:44.212044+0000","last_fullsized":"2024-03-24T09:19:44.212044+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212590+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:19:44.212590+0000","last_peered":"2024-03-24T09:19:44.212590+0000","last_clean":"2024-03-24T09:19:44.212590+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:19:44.212590+0000","last_undegraded":"2024-03-24T09:19:44.212590+0000","last_fullsized":"2024-03-24T09:19:44.212590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040346+0000","last_change":"2024-03-24T09:19:38.730518+0000","last_active":"2024-03-24T09:19:44.040346+0000","last_peered":"2024-03-24T09:19:44.040346+0000","last_clean":"2024-03-24T09:19:44.040346+0000","last_became_active":"2024-03-24T09:19:38.730162+0000","last_became_peered":"2024-03-24T09:19:38.730162+0000","last_unstale":"2024-03-24T09:19:44.040346+0000","last_undegraded":"2024-03-24T09:19:44.040346+0000","last_fullsized":"2024-03-24T09:19:44.040346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:22.380328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211860+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:19:44.211860+0000","last_peered":"2024-03-24T09:19:44.211860+0000","last_clean":"2024-03-24T09:19:44.211860+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:19:44.211860+0000","last_undegraded":"2024-03-24T09:19:44.211860+0000","last_fullsized":"2024-03-24T09:19:44.211860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211640+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:19:44.211640+0000","last_peered":"2024-03-24T09:19:44.211640+0000","last_clean":"2024-03-24T09:19:44.211640+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:19:44.211640+0000","last_undegraded":"2024-03-24T09:19:44.211640+0000","last_fullsized":"2024-03-24T09:19:44.211640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040473+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:19:44.040473+0000","last_peered":"2024-03-24T09:19:44.040473+0000","last_clean":"2024-03-24T09:19:44.040473+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:19:44.040473+0000","last_undegraded":"2024-03-24T09:19:44.040473+0000","last_fullsized":"2024-03-24T09:19:44.040473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210774+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:19:44.210774+0000","last_peered":"2024-03-24T09:19:44.210774+0000","last_clean":"2024-03-24T09:19:44.210774+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:19:44.210774+0000","last_undegraded":"2024-03-24T09:19:44.210774+0000","last_fullsized":"2024-03-24T09:19:44.210774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726377+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:19:43.726377+0000","last_peered":"2024-03-24T09:19:43.726377+0000","last_clean":"2024-03-24T09:19:43.726377+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:19:43.726377+0000","last_undegraded":"2024-03-24T09:19:43.726377+0000","last_fullsized":"2024-03-24T09:19:43.726377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.725975+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:19:43.725975+0000","last_peered":"2024-03-24T09:19:43.725975+0000","last_clean":"2024-03-24T09:19:43.725975+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:19:43.725975+0000","last_undegraded":"2024-03-24T09:19:43.725975+0000","last_fullsized":"2024-03-24T09:19:43.725975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212206+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:19:44.212206+0000","last_peered":"2024-03-24T09:19:44.212206+0000","last_clean":"2024-03-24T09:19:44.212206+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:19:44.212206+0000","last_undegraded":"2024-03-24T09:19:44.212206+0000","last_fullsized":"2024-03-24T09:19:44.212206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212034+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:19:44.212034+0000","last_peered":"2024-03-24T09:19:44.212034+0000","last_clean":"2024-03-24T09:19:44.212034+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:19:44.212034+0000","last_undegraded":"2024-03-24T09:19:44.212034+0000","last_fullsized":"2024-03-24T09:19:44.212034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211288+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:19:44.211288+0000","last_peered":"2024-03-24T09:19:44.211288+0000","last_clean":"2024-03-24T09:19:44.211288+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:19:44.211288+0000","last_undegraded":"2024-03-24T09:19:44.211288+0000","last_fullsized":"2024-03-24T09:19:44.211288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726361+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:19:43.726361+0000","last_peered":"2024-03-24T09:19:43.726361+0000","last_clean":"2024-03-24T09:19:43.726361+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:19:43.726361+0000","last_undegraded":"2024-03-24T09:19:43.726361+0000","last_fullsized":"2024-03-24T09:19:43.726361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212665+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:19:44.212665+0000","last_peered":"2024-03-24T09:19:44.212665+0000","last_clean":"2024-03-24T09:19:44.212665+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:19:44.212665+0000","last_undegraded":"2024-03-24T09:19:44.212665+0000","last_fullsized":"2024-03-24T09:19:44.212665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212168+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:19:44.212168+0000","last_peered":"2024-03-24T09:19:44.212168+0000","last_clean":"2024-03-24T09:19:44.212168+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:19:44.212168+0000","last_undegraded":"2024-03-24T09:19:44.212168+0000","last_fullsized":"2024-03-24T09:19:44.212168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212236+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:19:44.212236+0000","last_peered":"2024-03-24T09:19:44.212236+0000","last_clean":"2024-03-24T09:19:44.212236+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:19:44.212236+0000","last_undegraded":"2024-03-24T09:19:44.212236+0000","last_fullsized":"2024-03-24T09:19:44.212236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210987+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:19:44.210987+0000","last_peered":"2024-03-24T09:19:44.210987+0000","last_clean":"2024-03-24T09:19:44.210987+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:19:44.210987+0000","last_undegraded":"2024-03-24T09:19:44.210987+0000","last_fullsized":"2024-03-24T09:19:44.210987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211236+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:19:44.211236+0000","last_peered":"2024-03-24T09:19:44.211236+0000","last_clean":"2024-03-24T09:19:44.211236+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:19:44.211236+0000","last_undegraded":"2024-03-24T09:19:44.211236+0000","last_fullsized":"2024-03-24T09:19:44.211236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211503+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:19:44.211503+0000","last_peered":"2024-03-24T09:19:44.211503+0000","last_clean":"2024-03-24T09:19:44.211503+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:19:44.211503+0000","last_undegraded":"2024-03-24T09:19:44.211503+0000","last_fullsized":"2024-03-24T09:19:44.211503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280427,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7984,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":13,"num_read_kb":12,"num_write":129,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":39,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":40177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50370,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50370,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":481639,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477409,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477859,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50370,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":23723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:20:07.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T09:20:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 -- 172.21.15.116:0/2129885440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 msgr2=0x7f83b0106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 --2- 172.21.15.116:0/2129885440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0106ed0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8398009900 tx=0x7f839802bc30 comp rx=0 tx=0).stop 2024-03-24T09:20:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 -- 172.21.15.116:0/2129885440 shutdown_connections 2024-03-24T09:20:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 --2- 172.21.15.116:0/2129885440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b0107410 0x7f83b0109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 --2- 172.21.15.116:0/2129885440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0106ed0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 --2- 172.21.15.116:0/2129885440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b01021b0 0x7f83b01045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 -- 172.21.15.116:0/2129885440 >> 172.21.15.116:0/2129885440 conn(0x7f83b00fbbc0 msgr2=0x7f83b00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 -- 172.21.15.116:0/2129885440 shutdown_connections 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 -- 172.21.15.116:0/2129885440 wait complete. 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 Processor -- start 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 -- start start 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.752+0000 7f83b583b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b01021b0 0x7f83b0197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b0107410 0x7f83b019ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83b019f200 con 0x7f83b0104ae0 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f83b019f370 con 0x7f83b0107410 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f83b019f530 con 0x7f83b01021b0 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83aeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b01021b0 0x7f83b0197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34508/0 (socket says 172.21.15.116:34508) 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 -- 172.21.15.116:0/1328217500 learned_addr learned my addr 172.21.15.116:0/1328217500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83af7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b0107410 0x7f83b019ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 -- 172.21.15.116:0/1328217500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b01021b0 msgr2=0x7f83b0197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b01021b0 0x7f83b0197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 -- 172.21.15.116:0/1328217500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b0107410 msgr2=0x7f83b019ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b0107410 0x7f83b019ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 -- 172.21.15.116:0/1328217500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8398009560 con 0x7f83b0104ae0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83af7fe640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b0107410 0x7f83b019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83a7fff640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0197c50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f839802c7d0 tx=0x7f8398004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398004500 con 0x7f83b0104ae0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b019f7b0 con 0x7f83b0104ae0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f839802de90 con 0x7f83b0104ae0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398005350 con 0x7f83b0104ae0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.753+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b019fcc0 con 0x7f83b0104ae0 2024-03-24T09:20:07.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.754+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b00fd6b0 con 0x7f83b0104ae0 2024-03-24T09:20:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.755+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f83980054b0 con 0x7f83b0104ae0 2024-03-24T09:20:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.756+0000 7f83b4839640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f839c0a2150 0x7f839c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.756+0000 7f83aeffd640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f839c0a2150 0x7f839c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.756+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f83980f0580 con 0x7f83b0104ae0 2024-03-24T09:20:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.756+0000 7f83aeffd640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f839c0a2150 0x7f839c0a4610 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009bb0 tx=0x7f83a0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:07.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.759+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f83980b92c0 con 0x7f83b0104ae0 2024-03-24T09:20:07.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.932+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f83b002d240 con 0x7f839c0a2150 2024-03-24T09:20:07.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.941+0000 7f83b4839640 1 -- 172.21.15.116:0/1328217500 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340640 (secure 0 0 0) 0x7f83b002d240 con 0x7f839c0a2150 2024-03-24T09:20:07.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:20:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.944+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f839c0a2150 msgr2=0x7f839c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.944+0000 7f83b583b640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f839c0a2150 0x7f839c0a4610 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009bb0 tx=0x7f83a0005f90 comp rx=0 tx=0).stop 2024-03-24T09:20:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.944+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 msgr2=0x7f83b0197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0197c50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f839802c7d0 tx=0x7f8398004040 comp rx=0 tx=0).stop 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 shutdown_connections 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b0107410 0x7f83b019ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b0104ae0 0x7f83b0197c50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f839c0a2150 0x7f839c0a4610 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 --2- 172.21.15.116:0/1328217500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b01021b0 0x7f83b0197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 >> 172.21.15.116:0/1328217500 conn(0x7f83b00fbbc0 msgr2=0x7f83b0102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 shutdown_connections 2024-03-24T09:20:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:07.945+0000 7f83b583b640 1 -- 172.21.15.116:0/1328217500 wait complete. 2024-03-24T09:20:07.956 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-03-24T09:20:07.926656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":635161,"num_objects":70,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":0,"num_read":166,"num_read_kb":181,"num_write":341,"num_write_kb":3024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":377504,"kb_used_data":11704,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124564320,"statfs":{"total":1151940427776,"available":1151553863680,"internally_reserved":0,"allocated":11984896,"data_stored":7556528,"data_compressed":283428,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":40294,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":30,"num_read_kb":62,"num_write":113,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003962"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211311+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:19:44.211311+0000","last_peered":"2024-03-24T09:19:44.211311+0000","last_clean":"2024-03-24T09:19:44.211311+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:19:44.211311+0000","last_undegraded":"2024-03-24T09:19:44.211311+0000","last_fullsized":"2024-03-24T09:19:44.211311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212391+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:19:44.212391+0000","last_peered":"2024-03-24T09:19:44.212391+0000","last_clean":"2024-03-24T09:19:44.212391+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:19:44.212391+0000","last_undegraded":"2024-03-24T09:19:44.212391+0000","last_fullsized":"2024-03-24T09:19:44.212391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211636+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:19:44.211636+0000","last_peered":"2024-03-24T09:19:44.211636+0000","last_clean":"2024-03-24T09:19:44.211636+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:19:44.211636+0000","last_undegraded":"2024-03-24T09:19:44.211636+0000","last_fullsized":"2024-03-24T09:19:44.211636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211599+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:19:44.211599+0000","last_peered":"2024-03-24T09:19:44.211599+0000","last_clean":"2024-03-24T09:19:44.211599+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:19:44.211599+0000","last_undegraded":"2024-03-24T09:19:44.211599+0000","last_fullsized":"2024-03-24T09:19:44.211599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211092+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:19:44.211092+0000","last_peered":"2024-03-24T09:19:44.211092+0000","last_clean":"2024-03-24T09:19:44.211092+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:19:44.211092+0000","last_undegraded":"2024-03-24T09:19:44.211092+0000","last_fullsized":"2024-03-24T09:19:44.211092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726099+0000","last_change":"2024-03-24T09:19:38.724773+0000","last_active":"2024-03-24T09:19:43.726099+0000","last_peered":"2024-03-24T09:19:43.726099+0000","last_clean":"2024-03-24T09:19:43.726099+0000","last_became_active":"2024-03-24T09:19:38.724398+0000","last_became_peered":"2024-03-24T09:19:38.724398+0000","last_unstale":"2024-03-24T09:19:43.726099+0000","last_undegraded":"2024-03-24T09:19:43.726099+0000","last_fullsized":"2024-03-24T09:19:43.726099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:45:28.883052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040130+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:19:44.040130+0000","last_peered":"2024-03-24T09:19:44.040130+0000","last_clean":"2024-03-24T09:19:44.040130+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:19:44.040130+0000","last_undegraded":"2024-03-24T09:19:44.040130+0000","last_fullsized":"2024-03-24T09:19:44.040130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211002+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:19:44.211002+0000","last_peered":"2024-03-24T09:19:44.211002+0000","last_clean":"2024-03-24T09:19:44.211002+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:19:44.211002+0000","last_undegraded":"2024-03-24T09:19:44.211002+0000","last_fullsized":"2024-03-24T09:19:44.211002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211221+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:19:44.211221+0000","last_peered":"2024-03-24T09:19:44.211221+0000","last_clean":"2024-03-24T09:19:44.211221+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:19:44.211221+0000","last_undegraded":"2024-03-24T09:19:44.211221+0000","last_fullsized":"2024-03-24T09:19:44.211221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726444+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:19:43.726444+0000","last_peered":"2024-03-24T09:19:43.726444+0000","last_clean":"2024-03-24T09:19:43.726444+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:19:43.726444+0000","last_undegraded":"2024-03-24T09:19:43.726444+0000","last_fullsized":"2024-03-24T09:19:43.726444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040468+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:19:44.040468+0000","last_peered":"2024-03-24T09:19:44.040468+0000","last_clean":"2024-03-24T09:19:44.040468+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:19:44.040468+0000","last_undegraded":"2024-03-24T09:19:44.040468+0000","last_fullsized":"2024-03-24T09:19:44.040468+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211494+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:19:44.211494+0000","last_peered":"2024-03-24T09:19:44.211494+0000","last_clean":"2024-03-24T09:19:44.211494+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:19:44.211494+0000","last_undegraded":"2024-03-24T09:19:44.211494+0000","last_fullsized":"2024-03-24T09:19:44.211494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211138+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:19:44.211138+0000","last_peered":"2024-03-24T09:19:44.211138+0000","last_clean":"2024-03-24T09:19:44.211138+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:19:44.211138+0000","last_undegraded":"2024-03-24T09:19:44.211138+0000","last_fullsized":"2024-03-24T09:19:44.211138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211141+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:19:44.211141+0000","last_peered":"2024-03-24T09:19:44.211141+0000","last_clean":"2024-03-24T09:19:44.211141+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:19:44.211141+0000","last_undegraded":"2024-03-24T09:19:44.211141+0000","last_fullsized":"2024-03-24T09:19:44.211141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211879+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:19:44.211879+0000","last_peered":"2024-03-24T09:19:44.211879+0000","last_clean":"2024-03-24T09:19:44.211879+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:19:44.211879+0000","last_undegraded":"2024-03-24T09:19:44.211879+0000","last_fullsized":"2024-03-24T09:19:44.211879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210533+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:19:44.210533+0000","last_peered":"2024-03-24T09:19:44.210533+0000","last_clean":"2024-03-24T09:19:44.210533+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:19:44.210533+0000","last_undegraded":"2024-03-24T09:19:44.210533+0000","last_fullsized":"2024-03-24T09:19:44.210533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211087+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:19:44.211087+0000","last_peered":"2024-03-24T09:19:44.211087+0000","last_clean":"2024-03-24T09:19:44.211087+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:19:44.211087+0000","last_undegraded":"2024-03-24T09:19:44.211087+0000","last_fullsized":"2024-03-24T09:19:44.211087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726691+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:19:43.726691+0000","last_peered":"2024-03-24T09:19:43.726691+0000","last_clean":"2024-03-24T09:19:43.726691+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:19:43.726691+0000","last_undegraded":"2024-03-24T09:19:43.726691+0000","last_fullsized":"2024-03-24T09:19:43.726691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726452+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:19:43.726452+0000","last_peered":"2024-03-24T09:19:43.726452+0000","last_clean":"2024-03-24T09:19:43.726452+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:19:43.726452+0000","last_undegraded":"2024-03-24T09:19:43.726452+0000","last_fullsized":"2024-03-24T09:19:43.726452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726075+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:19:43.726075+0000","last_peered":"2024-03-24T09:19:43.726075+0000","last_clean":"2024-03-24T09:19:43.726075+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:19:43.726075+0000","last_undegraded":"2024-03-24T09:19:43.726075+0000","last_fullsized":"2024-03-24T09:19:43.726075+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'11","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.912517+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:20:04.912517+0000","last_peered":"2024-03-24T09:20:04.912517+0000","last_clean":"2024-03-24T09:20:04.912517+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:20:04.912517+0000","last_undegraded":"2024-03-24T09:20:04.912517+0000","last_fullsized":"2024-03-24T09:20:04.912517+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212660+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:19:44.212660+0000","last_peered":"2024-03-24T09:19:44.212660+0000","last_clean":"2024-03-24T09:19:44.212660+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:19:44.212660+0000","last_undegraded":"2024-03-24T09:19:44.212660+0000","last_fullsized":"2024-03-24T09:19:44.212660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210728+0000","last_change":"2024-03-24T09:19:42.746142+0000","last_active":"2024-03-24T09:19:44.210728+0000","last_peered":"2024-03-24T09:19:44.210728+0000","last_clean":"2024-03-24T09:19:44.210728+0000","last_became_active":"2024-03-24T09:19:42.745735+0000","last_became_peered":"2024-03-24T09:19:42.745735+0000","last_unstale":"2024-03-24T09:19:44.210728+0000","last_undegraded":"2024-03-24T09:19:44.210728+0000","last_fullsized":"2024-03-24T09:19:44.210728+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:33:28.908417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212800+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:19:44.212800+0000","last_peered":"2024-03-24T09:19:44.212800+0000","last_clean":"2024-03-24T09:19:44.212800+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:19:44.212800+0000","last_undegraded":"2024-03-24T09:19:44.212800+0000","last_fullsized":"2024-03-24T09:19:44.212800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211742+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:19:44.211742+0000","last_peered":"2024-03-24T09:19:44.211742+0000","last_clean":"2024-03-24T09:19:44.211742+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:19:44.211742+0000","last_undegraded":"2024-03-24T09:19:44.211742+0000","last_fullsized":"2024-03-24T09:19:44.211742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211284+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:19:44.211284+0000","last_peered":"2024-03-24T09:19:44.211284+0000","last_clean":"2024-03-24T09:19:44.211284+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:19:44.211284+0000","last_undegraded":"2024-03-24T09:19:44.211284+0000","last_fullsized":"2024-03-24T09:19:44.211284+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.903185+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:20:03.903185+0000","last_peered":"2024-03-24T09:20:03.903185+0000","last_clean":"2024-03-24T09:20:03.903185+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:20:03.903185+0000","last_undegraded":"2024-03-24T09:20:03.903185+0000","last_fullsized":"2024-03-24T09:20:03.903185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211002+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:19:44.211002+0000","last_peered":"2024-03-24T09:19:44.211002+0000","last_clean":"2024-03-24T09:19:44.211002+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:19:44.211002+0000","last_undegraded":"2024-03-24T09:19:44.211002+0000","last_fullsized":"2024-03-24T09:19:44.211002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211581+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:19:44.211581+0000","last_peered":"2024-03-24T09:19:44.211581+0000","last_clean":"2024-03-24T09:19:44.211581+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:19:44.211581+0000","last_undegraded":"2024-03-24T09:19:44.211581+0000","last_fullsized":"2024-03-24T09:19:44.211581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211383+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:19:44.211383+0000","last_peered":"2024-03-24T09:19:44.211383+0000","last_clean":"2024-03-24T09:19:44.211383+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:19:44.211383+0000","last_undegraded":"2024-03-24T09:19:44.211383+0000","last_fullsized":"2024-03-24T09:19:44.211383+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212492+0000","last_change":"2024-03-24T09:19:38.733015+0000","last_active":"2024-03-24T09:19:44.212492+0000","last_peered":"2024-03-24T09:19:44.212492+0000","last_clean":"2024-03-24T09:19:44.212492+0000","last_became_active":"2024-03-24T09:19:38.732691+0000","last_became_peered":"2024-03-24T09:19:38.732691+0000","last_unstale":"2024-03-24T09:19:44.212492+0000","last_undegraded":"2024-03-24T09:19:44.212492+0000","last_fullsized":"2024-03-24T09:19:44.212492+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:50:02.190295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726450+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:19:43.726450+0000","last_peered":"2024-03-24T09:19:43.726450+0000","last_clean":"2024-03-24T09:19:43.726450+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:19:43.726450+0000","last_undegraded":"2024-03-24T09:19:43.726450+0000","last_fullsized":"2024-03-24T09:19:43.726450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.895768+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:20:03.895768+0000","last_peered":"2024-03-24T09:20:03.895768+0000","last_clean":"2024-03-24T09:20:03.895768+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:20:03.895768+0000","last_undegraded":"2024-03-24T09:20:03.895768+0000","last_fullsized":"2024-03-24T09:20:03.895768+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211405+0000","last_change":"2024-03-24T09:19:38.725875+0000","last_active":"2024-03-24T09:19:44.211405+0000","last_peered":"2024-03-24T09:19:44.211405+0000","last_clean":"2024-03-24T09:19:44.211405+0000","last_became_active":"2024-03-24T09:19:38.725536+0000","last_became_peered":"2024-03-24T09:19:38.725536+0000","last_unstale":"2024-03-24T09:19:44.211405+0000","last_undegraded":"2024-03-24T09:19:44.211405+0000","last_fullsized":"2024-03-24T09:19:44.211405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:20:36.250703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726250+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:19:43.726250+0000","last_peered":"2024-03-24T09:19:43.726250+0000","last_clean":"2024-03-24T09:19:43.726250+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:19:43.726250+0000","last_undegraded":"2024-03-24T09:19:43.726250+0000","last_fullsized":"2024-03-24T09:19:43.726250+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.911679+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:20:04.911679+0000","last_peered":"2024-03-24T09:20:04.911679+0000","last_clean":"2024-03-24T09:20:04.911679+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:20:04.911679+0000","last_undegraded":"2024-03-24T09:20:04.911679+0000","last_fullsized":"2024-03-24T09:20:04.911679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212684+0000","last_change":"2024-03-24T09:19:38.734122+0000","last_active":"2024-03-24T09:19:44.212684+0000","last_peered":"2024-03-24T09:19:44.212684+0000","last_clean":"2024-03-24T09:19:44.212684+0000","last_became_active":"2024-03-24T09:19:38.733805+0000","last_became_peered":"2024-03-24T09:19:38.733805+0000","last_unstale":"2024-03-24T09:19:44.212684+0000","last_undegraded":"2024-03-24T09:19:44.212684+0000","last_fullsized":"2024-03-24T09:19:44.212684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:57.423822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210944+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:19:44.210944+0000","last_peered":"2024-03-24T09:19:44.210944+0000","last_clean":"2024-03-24T09:19:44.210944+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:19:44.210944+0000","last_undegraded":"2024-03-24T09:19:44.210944+0000","last_fullsized":"2024-03-24T09:19:44.210944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.904008+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:20:03.904008+0000","last_peered":"2024-03-24T09:20:03.904008+0000","last_clean":"2024-03-24T09:20:03.904008+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:20:03.904008+0000","last_undegraded":"2024-03-24T09:20:03.904008+0000","last_fullsized":"2024-03-24T09:20:03.904008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211329+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:19:44.211329+0000","last_peered":"2024-03-24T09:19:44.211329+0000","last_clean":"2024-03-24T09:19:44.211329+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:19:44.211329+0000","last_undegraded":"2024-03-24T09:19:44.211329+0000","last_fullsized":"2024-03-24T09:19:44.211329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212319+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:19:44.212319+0000","last_peered":"2024-03-24T09:19:44.212319+0000","last_clean":"2024-03-24T09:19:44.212319+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:19:44.212319+0000","last_undegraded":"2024-03-24T09:19:44.212319+0000","last_fullsized":"2024-03-24T09:19:44.212319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212118+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:19:44.212118+0000","last_peered":"2024-03-24T09:19:44.212118+0000","last_clean":"2024-03-24T09:19:44.212118+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:19:44.212118+0000","last_undegraded":"2024-03-24T09:19:44.212118+0000","last_fullsized":"2024-03-24T09:19:44.212118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726332+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:19:43.726332+0000","last_peered":"2024-03-24T09:19:43.726332+0000","last_clean":"2024-03-24T09:19:43.726332+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:19:43.726332+0000","last_undegraded":"2024-03-24T09:19:43.726332+0000","last_fullsized":"2024-03-24T09:19:43.726332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212523+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:19:44.212523+0000","last_peered":"2024-03-24T09:19:44.212523+0000","last_clean":"2024-03-24T09:19:44.212523+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:19:44.212523+0000","last_undegraded":"2024-03-24T09:19:44.212523+0000","last_fullsized":"2024-03-24T09:19:44.212523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:06.900830+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:20:06.900830+0000","last_peered":"2024-03-24T09:20:06.900830+0000","last_clean":"2024-03-24T09:20:06.900830+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:20:06.900830+0000","last_undegraded":"2024-03-24T09:20:06.900830+0000","last_fullsized":"2024-03-24T09:20:06.900830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210962+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:19:44.210962+0000","last_peered":"2024-03-24T09:19:44.210962+0000","last_clean":"2024-03-24T09:19:44.210962+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:19:44.210962+0000","last_undegraded":"2024-03-24T09:19:44.210962+0000","last_fullsized":"2024-03-24T09:19:44.210962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726574+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:19:43.726574+0000","last_peered":"2024-03-24T09:19:43.726574+0000","last_clean":"2024-03-24T09:19:43.726574+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:19:43.726574+0000","last_undegraded":"2024-03-24T09:19:43.726574+0000","last_fullsized":"2024-03-24T09:19:43.726574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029691+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:20:03.029691+0000","last_peered":"2024-03-24T09:20:03.029691+0000","last_clean":"2024-03-24T09:20:03.029691+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:20:03.029691+0000","last_undegraded":"2024-03-24T09:20:03.029691+0000","last_fullsized":"2024-03-24T09:20:03.029691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211830+0000","last_change":"2024-03-24T09:19:38.730068+0000","last_active":"2024-03-24T09:19:44.211830+0000","last_peered":"2024-03-24T09:19:44.211830+0000","last_clean":"2024-03-24T09:19:44.211830+0000","last_became_active":"2024-03-24T09:19:38.729747+0000","last_became_peered":"2024-03-24T09:19:38.729747+0000","last_unstale":"2024-03-24T09:19:44.211830+0000","last_undegraded":"2024-03-24T09:19:44.211830+0000","last_fullsized":"2024-03-24T09:19:44.211830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:44:48.285768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211573+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:19:44.211573+0000","last_peered":"2024-03-24T09:19:44.211573+0000","last_clean":"2024-03-24T09:19:44.211573+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:19:44.211573+0000","last_undegraded":"2024-03-24T09:19:44.211573+0000","last_fullsized":"2024-03-24T09:19:44.211573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.031031+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:20:03.031031+0000","last_peered":"2024-03-24T09:20:03.031031+0000","last_clean":"2024-03-24T09:20:03.031031+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:20:03.031031+0000","last_undegraded":"2024-03-24T09:20:03.031031+0000","last_fullsized":"2024-03-24T09:20:03.031031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211115+0000","last_change":"2024-03-24T09:19:38.736495+0000","last_active":"2024-03-24T09:19:44.211115+0000","last_peered":"2024-03-24T09:19:44.211115+0000","last_clean":"2024-03-24T09:19:44.211115+0000","last_became_active":"2024-03-24T09:19:38.736113+0000","last_became_peered":"2024-03-24T09:19:38.736113+0000","last_unstale":"2024-03-24T09:19:44.211115+0000","last_undegraded":"2024-03-24T09:19:44.211115+0000","last_fullsized":"2024-03-24T09:19:44.211115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:32:45.254878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726606+0000","last_change":"2024-03-24T09:19:42.742649+0000","last_active":"2024-03-24T09:19:43.726606+0000","last_peered":"2024-03-24T09:19:43.726606+0000","last_clean":"2024-03-24T09:19:43.726606+0000","last_became_active":"2024-03-24T09:19:42.742256+0000","last_became_peered":"2024-03-24T09:19:42.742256+0000","last_unstale":"2024-03-24T09:19:43.726606+0000","last_undegraded":"2024-03-24T09:19:43.726606+0000","last_fullsized":"2024-03-24T09:19:43.726606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:46:59.749664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'25","reported_seq":60,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:06.899325+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:20:06.899325+0000","last_peered":"2024-03-24T09:20:06.899325+0000","last_clean":"2024-03-24T09:20:06.899325+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:20:06.899325+0000","last_undegraded":"2024-03-24T09:20:06.899325+0000","last_fullsized":"2024-03-24T09:20:06.899325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40471,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":45,"num_write":27,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211656+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:19:44.211656+0000","last_peered":"2024-03-24T09:19:44.211656+0000","last_clean":"2024-03-24T09:19:44.211656+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:19:44.211656+0000","last_undegraded":"2024-03-24T09:19:44.211656+0000","last_fullsized":"2024-03-24T09:19:44.211656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212796+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:19:44.212796+0000","last_peered":"2024-03-24T09:19:44.212796+0000","last_clean":"2024-03-24T09:19:44.212796+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:19:44.212796+0000","last_undegraded":"2024-03-24T09:19:44.212796+0000","last_fullsized":"2024-03-24T09:19:44.212796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'7","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.904231+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:20:03.904231+0000","last_peered":"2024-03-24T09:20:03.904231+0000","last_clean":"2024-03-24T09:20:03.904231+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:20:03.904231+0000","last_undegraded":"2024-03-24T09:20:03.904231+0000","last_fullsized":"2024-03-24T09:20:03.904231+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211456+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:19:44.211456+0000","last_peered":"2024-03-24T09:19:44.211456+0000","last_clean":"2024-03-24T09:19:44.211456+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:19:44.211456+0000","last_undegraded":"2024-03-24T09:19:44.211456+0000","last_fullsized":"2024-03-24T09:19:44.211456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211343+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:19:44.211343+0000","last_peered":"2024-03-24T09:19:44.211343+0000","last_clean":"2024-03-24T09:19:44.211343+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:19:44.211343+0000","last_undegraded":"2024-03-24T09:19:44.211343+0000","last_fullsized":"2024-03-24T09:19:44.211343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.030554+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:20:03.030554+0000","last_peered":"2024-03-24T09:20:03.030554+0000","last_clean":"2024-03-24T09:20:03.030554+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:20:03.030554+0000","last_undegraded":"2024-03-24T09:20:03.030554+0000","last_fullsized":"2024-03-24T09:20:03.030554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040220+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:19:44.040220+0000","last_peered":"2024-03-24T09:19:44.040220+0000","last_clean":"2024-03-24T09:19:44.040220+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:19:44.040220+0000","last_undegraded":"2024-03-24T09:19:44.040220+0000","last_fullsized":"2024-03-24T09:19:44.040220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726821+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:19:43.726821+0000","last_peered":"2024-03-24T09:19:43.726821+0000","last_clean":"2024-03-24T09:19:43.726821+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:19:43.726821+0000","last_undegraded":"2024-03-24T09:19:43.726821+0000","last_fullsized":"2024-03-24T09:19:43.726821+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.905791+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:20:03.905791+0000","last_peered":"2024-03-24T09:20:03.905791+0000","last_clean":"2024-03-24T09:20:03.905791+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:20:03.905791+0000","last_undegraded":"2024-03-24T09:20:03.905791+0000","last_fullsized":"2024-03-24T09:20:03.905791+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040149+0000","last_change":"2024-03-24T09:19:38.730496+0000","last_active":"2024-03-24T09:19:44.040149+0000","last_peered":"2024-03-24T09:19:44.040149+0000","last_clean":"2024-03-24T09:19:44.040149+0000","last_became_active":"2024-03-24T09:19:38.730164+0000","last_became_peered":"2024-03-24T09:19:38.730164+0000","last_unstale":"2024-03-24T09:19:44.040149+0000","last_undegraded":"2024-03-24T09:19:44.040149+0000","last_fullsized":"2024-03-24T09:19:44.040149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:51.277493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211773+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:19:44.211773+0000","last_peered":"2024-03-24T09:19:44.211773+0000","last_clean":"2024-03-24T09:19:44.211773+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:19:44.211773+0000","last_undegraded":"2024-03-24T09:19:44.211773+0000","last_fullsized":"2024-03-24T09:19:44.211773+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.905845+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:20:03.905845+0000","last_peered":"2024-03-24T09:20:03.905845+0000","last_clean":"2024-03-24T09:20:03.905845+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:20:03.905845+0000","last_undegraded":"2024-03-24T09:20:03.905845+0000","last_fullsized":"2024-03-24T09:20:03.905845+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212426+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:19:44.212426+0000","last_peered":"2024-03-24T09:19:44.212426+0000","last_clean":"2024-03-24T09:19:44.212426+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:19:44.212426+0000","last_undegraded":"2024-03-24T09:19:44.212426+0000","last_fullsized":"2024-03-24T09:19:44.212426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212194+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:19:44.212194+0000","last_peered":"2024-03-24T09:19:44.212194+0000","last_clean":"2024-03-24T09:19:44.212194+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:19:44.212194+0000","last_undegraded":"2024-03-24T09:19:44.212194+0000","last_fullsized":"2024-03-24T09:19:44.212194+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211796+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:19:44.211796+0000","last_peered":"2024-03-24T09:19:44.211796+0000","last_clean":"2024-03-24T09:19:44.211796+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:19:44.211796+0000","last_undegraded":"2024-03-24T09:19:44.211796+0000","last_fullsized":"2024-03-24T09:19:44.211796+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210992+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:19:44.210992+0000","last_peered":"2024-03-24T09:19:44.210992+0000","last_clean":"2024-03-24T09:19:44.210992+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:19:44.210992+0000","last_undegraded":"2024-03-24T09:19:44.210992+0000","last_fullsized":"2024-03-24T09:19:44.210992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211385+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:19:44.211385+0000","last_peered":"2024-03-24T09:19:44.211385+0000","last_clean":"2024-03-24T09:19:44.211385+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:19:44.211385+0000","last_undegraded":"2024-03-24T09:19:44.211385+0000","last_fullsized":"2024-03-24T09:19:44.211385+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726715+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:19:43.726715+0000","last_peered":"2024-03-24T09:19:43.726715+0000","last_clean":"2024-03-24T09:19:43.726715+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:19:43.726715+0000","last_undegraded":"2024-03-24T09:19:43.726715+0000","last_fullsized":"2024-03-24T09:19:43.726715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726308+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:19:43.726308+0000","last_peered":"2024-03-24T09:19:43.726308+0000","last_clean":"2024-03-24T09:19:43.726308+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:19:43.726308+0000","last_undegraded":"2024-03-24T09:19:43.726308+0000","last_fullsized":"2024-03-24T09:19:43.726308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211871+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:19:44.211871+0000","last_peered":"2024-03-24T09:19:44.211871+0000","last_clean":"2024-03-24T09:19:44.211871+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:19:44.211871+0000","last_undegraded":"2024-03-24T09:19:44.211871+0000","last_fullsized":"2024-03-24T09:19:44.211871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040208+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:19:44.040208+0000","last_peered":"2024-03-24T09:19:44.040208+0000","last_clean":"2024-03-24T09:19:44.040208+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:19:44.040208+0000","last_undegraded":"2024-03-24T09:19:44.040208+0000","last_fullsized":"2024-03-24T09:19:44.040208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726128+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:19:43.726128+0000","last_peered":"2024-03-24T09:19:43.726128+0000","last_clean":"2024-03-24T09:19:43.726128+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:19:43.726128+0000","last_undegraded":"2024-03-24T09:19:43.726128+0000","last_fullsized":"2024-03-24T09:19:43.726128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211282+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:19:44.211282+0000","last_peered":"2024-03-24T09:19:44.211282+0000","last_clean":"2024-03-24T09:19:44.211282+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:19:44.211282+0000","last_undegraded":"2024-03-24T09:19:44.211282+0000","last_fullsized":"2024-03-24T09:19:44.211282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212258+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:19:44.212258+0000","last_peered":"2024-03-24T09:19:44.212258+0000","last_clean":"2024-03-24T09:19:44.212258+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:19:44.212258+0000","last_undegraded":"2024-03-24T09:19:44.212258+0000","last_fullsized":"2024-03-24T09:19:44.212258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.903195+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:20:03.903195+0000","last_peered":"2024-03-24T09:20:03.903195+0000","last_clean":"2024-03-24T09:20:03.903195+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:20:03.903195+0000","last_undegraded":"2024-03-24T09:20:03.903195+0000","last_fullsized":"2024-03-24T09:20:03.903195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040409+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:19:44.040409+0000","last_peered":"2024-03-24T09:19:44.040409+0000","last_clean":"2024-03-24T09:19:44.040409+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:19:44.040409+0000","last_undegraded":"2024-03-24T09:19:44.040409+0000","last_fullsized":"2024-03-24T09:19:44.040409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211556+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:19:44.211556+0000","last_peered":"2024-03-24T09:19:44.211556+0000","last_clean":"2024-03-24T09:19:44.211556+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:19:44.211556+0000","last_undegraded":"2024-03-24T09:19:44.211556+0000","last_fullsized":"2024-03-24T09:19:44.211556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.893846+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:20:03.893846+0000","last_peered":"2024-03-24T09:20:03.893846+0000","last_clean":"2024-03-24T09:20:03.893846+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:20:03.893846+0000","last_undegraded":"2024-03-24T09:20:03.893846+0000","last_fullsized":"2024-03-24T09:20:03.893846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":357,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.039985+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:19:44.039985+0000","last_peered":"2024-03-24T09:19:44.039985+0000","last_clean":"2024-03-24T09:19:44.039985+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:19:44.039985+0000","last_undegraded":"2024-03-24T09:19:44.039985+0000","last_fullsized":"2024-03-24T09:19:44.039985+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212483+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:19:44.212483+0000","last_peered":"2024-03-24T09:19:44.212483+0000","last_clean":"2024-03-24T09:19:44.212483+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:19:44.212483+0000","last_undegraded":"2024-03-24T09:19:44.212483+0000","last_fullsized":"2024-03-24T09:19:44.212483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210865+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:19:44.210865+0000","last_peered":"2024-03-24T09:19:44.210865+0000","last_clean":"2024-03-24T09:19:44.210865+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:19:44.210865+0000","last_undegraded":"2024-03-24T09:19:44.210865+0000","last_fullsized":"2024-03-24T09:19:44.210865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.905065+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:20:03.905065+0000","last_peered":"2024-03-24T09:20:03.905065+0000","last_clean":"2024-03-24T09:20:03.905065+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:20:03.905065+0000","last_undegraded":"2024-03-24T09:20:03.905065+0000","last_fullsized":"2024-03-24T09:20:03.905065+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210800+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:19:44.210800+0000","last_peered":"2024-03-24T09:19:44.210800+0000","last_clean":"2024-03-24T09:19:44.210800+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:19:44.210800+0000","last_undegraded":"2024-03-24T09:19:44.210800+0000","last_fullsized":"2024-03-24T09:19:44.210800+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029322+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:20:03.029322+0000","last_peered":"2024-03-24T09:20:03.029322+0000","last_clean":"2024-03-24T09:20:03.029322+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:20:03.029322+0000","last_undegraded":"2024-03-24T09:20:03.029322+0000","last_fullsized":"2024-03-24T09:20:03.029322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212123+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:19:44.212123+0000","last_peered":"2024-03-24T09:19:44.212123+0000","last_clean":"2024-03-24T09:19:44.212123+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:19:44.212123+0000","last_undegraded":"2024-03-24T09:19:44.212123+0000","last_fullsized":"2024-03-24T09:19:44.212123+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211279+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:19:44.211279+0000","last_peered":"2024-03-24T09:19:44.211279+0000","last_clean":"2024-03-24T09:19:44.211279+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:19:44.211279+0000","last_undegraded":"2024-03-24T09:19:44.211279+0000","last_fullsized":"2024-03-24T09:19:44.211279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.912972+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:20:04.912972+0000","last_peered":"2024-03-24T09:20:04.912972+0000","last_clean":"2024-03-24T09:20:04.912972+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:20:04.912972+0000","last_undegraded":"2024-03-24T09:20:04.912972+0000","last_fullsized":"2024-03-24T09:20:04.912972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212582+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:19:44.212582+0000","last_peered":"2024-03-24T09:19:44.212582+0000","last_clean":"2024-03-24T09:19:44.212582+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:19:44.212582+0000","last_undegraded":"2024-03-24T09:19:44.212582+0000","last_fullsized":"2024-03-24T09:19:44.212582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726107+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:19:43.726107+0000","last_peered":"2024-03-24T09:19:43.726107+0000","last_clean":"2024-03-24T09:19:43.726107+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:19:43.726107+0000","last_undegraded":"2024-03-24T09:19:43.726107+0000","last_fullsized":"2024-03-24T09:19:43.726107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029183+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:20:03.029183+0000","last_peered":"2024-03-24T09:20:03.029183+0000","last_clean":"2024-03-24T09:20:03.029183+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:20:03.029183+0000","last_undegraded":"2024-03-24T09:20:03.029183+0000","last_fullsized":"2024-03-24T09:20:03.029183+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212293+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:19:44.212293+0000","last_peered":"2024-03-24T09:19:44.212293+0000","last_clean":"2024-03-24T09:19:44.212293+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:19:44.212293+0000","last_undegraded":"2024-03-24T09:19:44.212293+0000","last_fullsized":"2024-03-24T09:19:44.212293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211587+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:19:44.211587+0000","last_peered":"2024-03-24T09:19:44.211587+0000","last_clean":"2024-03-24T09:19:44.211587+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:19:44.211587+0000","last_undegraded":"2024-03-24T09:19:44.211587+0000","last_fullsized":"2024-03-24T09:19:44.211587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.029752+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:20:03.029752+0000","last_peered":"2024-03-24T09:20:03.029752+0000","last_clean":"2024-03-24T09:20:03.029752+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:20:03.029752+0000","last_undegraded":"2024-03-24T09:20:03.029752+0000","last_fullsized":"2024-03-24T09:20:03.029752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210964+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:19:44.210964+0000","last_peered":"2024-03-24T09:19:44.210964+0000","last_clean":"2024-03-24T09:19:44.210964+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:19:44.210964+0000","last_undegraded":"2024-03-24T09:19:44.210964+0000","last_fullsized":"2024-03-24T09:19:44.210964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.727002+0000","last_change":"2024-03-24T09:19:42.743462+0000","last_active":"2024-03-24T09:19:43.727002+0000","last_peered":"2024-03-24T09:19:43.727002+0000","last_clean":"2024-03-24T09:19:43.727002+0000","last_became_active":"2024-03-24T09:19:42.743078+0000","last_became_peered":"2024-03-24T09:19:42.743078+0000","last_unstale":"2024-03-24T09:19:43.727002+0000","last_undegraded":"2024-03-24T09:19:43.727002+0000","last_fullsized":"2024-03-24T09:19:43.727002+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:49.958507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.030464+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:20:03.030464+0000","last_peered":"2024-03-24T09:20:03.030464+0000","last_clean":"2024-03-24T09:20:03.030464+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:20:03.030464+0000","last_undegraded":"2024-03-24T09:20:03.030464+0000","last_fullsized":"2024-03-24T09:20:03.030464+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210725+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:19:44.210725+0000","last_peered":"2024-03-24T09:19:44.210725+0000","last_clean":"2024-03-24T09:19:44.210725+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:19:44.210725+0000","last_undegraded":"2024-03-24T09:19:44.210725+0000","last_fullsized":"2024-03-24T09:19:44.210725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211583+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:19:44.211583+0000","last_peered":"2024-03-24T09:19:44.211583+0000","last_clean":"2024-03-24T09:19:44.211583+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:19:44.211583+0000","last_undegraded":"2024-03-24T09:19:44.211583+0000","last_fullsized":"2024-03-24T09:19:44.211583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.024425+0000","last_change":"2024-03-24T09:19:38.100940+0000","last_active":"2024-03-24T09:20:04.024425+0000","last_peered":"2024-03-24T09:20:04.024425+0000","last_clean":"2024-03-24T09:20:04.024425+0000","last_became_active":"2024-03-24T09:19:38.100613+0000","last_became_peered":"2024-03-24T09:19:38.100613+0000","last_unstale":"2024-03-24T09:20:04.024425+0000","last_undegraded":"2024-03-24T09:20:04.024425+0000","last_fullsized":"2024-03-24T09:20:04.024425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:53.483332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212013+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:19:44.212013+0000","last_peered":"2024-03-24T09:19:44.212013+0000","last_clean":"2024-03-24T09:19:44.212013+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:19:44.212013+0000","last_undegraded":"2024-03-24T09:19:44.212013+0000","last_fullsized":"2024-03-24T09:19:44.212013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211384+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:19:44.211384+0000","last_peered":"2024-03-24T09:19:44.211384+0000","last_clean":"2024-03-24T09:19:44.211384+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:19:44.211384+0000","last_undegraded":"2024-03-24T09:19:44.211384+0000","last_fullsized":"2024-03-24T09:19:44.211384+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:03.030033+0000","last_change":"2024-03-24T09:19:38.105336+0000","last_active":"2024-03-24T09:20:03.030033+0000","last_peered":"2024-03-24T09:20:03.030033+0000","last_clean":"2024-03-24T09:20:03.030033+0000","last_became_active":"2024-03-24T09:19:38.103695+0000","last_became_peered":"2024-03-24T09:19:38.103695+0000","last_unstale":"2024-03-24T09:20:03.030033+0000","last_undegraded":"2024-03-24T09:20:03.030033+0000","last_fullsized":"2024-03-24T09:20:03.030033+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:11:41.784391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040201+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:19:44.040201+0000","last_peered":"2024-03-24T09:19:44.040201+0000","last_clean":"2024-03-24T09:19:44.040201+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:19:44.040201+0000","last_undegraded":"2024-03-24T09:19:44.040201+0000","last_fullsized":"2024-03-24T09:19:44.040201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210863+0000","last_change":"2024-03-24T09:19:42.744163+0000","last_active":"2024-03-24T09:19:44.210863+0000","last_peered":"2024-03-24T09:19:44.210863+0000","last_clean":"2024-03-24T09:19:44.210863+0000","last_became_active":"2024-03-24T09:19:42.743754+0000","last_became_peered":"2024-03-24T09:19:42.743754+0000","last_unstale":"2024-03-24T09:19:44.210863+0000","last_undegraded":"2024-03-24T09:19:44.210863+0000","last_fullsized":"2024-03-24T09:19:44.210863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:07.082490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:20:04.908680+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:20:04.908680+0000","last_peered":"2024-03-24T09:20:04.908680+0000","last_clean":"2024-03-24T09:20:04.908680+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:20:04.908680+0000","last_undegraded":"2024-03-24T09:20:04.908680+0000","last_fullsized":"2024-03-24T09:20:04.908680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040563+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:19:44.040563+0000","last_peered":"2024-03-24T09:19:44.040563+0000","last_clean":"2024-03-24T09:19:44.040563+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:19:44.040563+0000","last_undegraded":"2024-03-24T09:19:44.040563+0000","last_fullsized":"2024-03-24T09:19:44.040563+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212044+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:19:44.212044+0000","last_peered":"2024-03-24T09:19:44.212044+0000","last_clean":"2024-03-24T09:19:44.212044+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:19:44.212044+0000","last_undegraded":"2024-03-24T09:19:44.212044+0000","last_fullsized":"2024-03-24T09:19:44.212044+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212590+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:19:44.212590+0000","last_peered":"2024-03-24T09:19:44.212590+0000","last_clean":"2024-03-24T09:19:44.212590+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:19:44.212590+0000","last_undegraded":"2024-03-24T09:19:44.212590+0000","last_fullsized":"2024-03-24T09:19:44.212590+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040346+0000","last_change":"2024-03-24T09:19:38.730518+0000","last_active":"2024-03-24T09:19:44.040346+0000","last_peered":"2024-03-24T09:19:44.040346+0000","last_clean":"2024-03-24T09:19:44.040346+0000","last_became_active":"2024-03-24T09:19:38.730162+0000","last_became_peered":"2024-03-24T09:19:38.730162+0000","last_unstale":"2024-03-24T09:19:44.040346+0000","last_undegraded":"2024-03-24T09:19:44.040346+0000","last_fullsized":"2024-03-24T09:19:44.040346+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:22.380328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211860+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:19:44.211860+0000","last_peered":"2024-03-24T09:19:44.211860+0000","last_clean":"2024-03-24T09:19:44.211860+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:19:44.211860+0000","last_undegraded":"2024-03-24T09:19:44.211860+0000","last_fullsized":"2024-03-24T09:19:44.211860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211640+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:19:44.211640+0000","last_peered":"2024-03-24T09:19:44.211640+0000","last_clean":"2024-03-24T09:19:44.211640+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:19:44.211640+0000","last_undegraded":"2024-03-24T09:19:44.211640+0000","last_fullsized":"2024-03-24T09:19:44.211640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.040473+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:19:44.040473+0000","last_peered":"2024-03-24T09:19:44.040473+0000","last_clean":"2024-03-24T09:19:44.040473+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:19:44.040473+0000","last_undegraded":"2024-03-24T09:19:44.040473+0000","last_fullsized":"2024-03-24T09:19:44.040473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210774+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:19:44.210774+0000","last_peered":"2024-03-24T09:19:44.210774+0000","last_clean":"2024-03-24T09:19:44.210774+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:19:44.210774+0000","last_undegraded":"2024-03-24T09:19:44.210774+0000","last_fullsized":"2024-03-24T09:19:44.210774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726377+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:19:43.726377+0000","last_peered":"2024-03-24T09:19:43.726377+0000","last_clean":"2024-03-24T09:19:43.726377+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:19:43.726377+0000","last_undegraded":"2024-03-24T09:19:43.726377+0000","last_fullsized":"2024-03-24T09:19:43.726377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.725975+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:19:43.725975+0000","last_peered":"2024-03-24T09:19:43.725975+0000","last_clean":"2024-03-24T09:19:43.725975+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:19:43.725975+0000","last_undegraded":"2024-03-24T09:19:43.725975+0000","last_fullsized":"2024-03-24T09:19:43.725975+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212206+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:19:44.212206+0000","last_peered":"2024-03-24T09:19:44.212206+0000","last_clean":"2024-03-24T09:19:44.212206+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:19:44.212206+0000","last_undegraded":"2024-03-24T09:19:44.212206+0000","last_fullsized":"2024-03-24T09:19:44.212206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212034+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:19:44.212034+0000","last_peered":"2024-03-24T09:19:44.212034+0000","last_clean":"2024-03-24T09:19:44.212034+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:19:44.212034+0000","last_undegraded":"2024-03-24T09:19:44.212034+0000","last_fullsized":"2024-03-24T09:19:44.212034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211288+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:19:44.211288+0000","last_peered":"2024-03-24T09:19:44.211288+0000","last_clean":"2024-03-24T09:19:44.211288+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:19:44.211288+0000","last_undegraded":"2024-03-24T09:19:44.211288+0000","last_fullsized":"2024-03-24T09:19:44.211288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:43.726361+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:19:43.726361+0000","last_peered":"2024-03-24T09:19:43.726361+0000","last_clean":"2024-03-24T09:19:43.726361+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:19:43.726361+0000","last_undegraded":"2024-03-24T09:19:43.726361+0000","last_fullsized":"2024-03-24T09:19:43.726361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212665+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:19:44.212665+0000","last_peered":"2024-03-24T09:19:44.212665+0000","last_clean":"2024-03-24T09:19:44.212665+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:19:44.212665+0000","last_undegraded":"2024-03-24T09:19:44.212665+0000","last_fullsized":"2024-03-24T09:19:44.212665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212168+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:19:44.212168+0000","last_peered":"2024-03-24T09:19:44.212168+0000","last_clean":"2024-03-24T09:19:44.212168+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:19:44.212168+0000","last_undegraded":"2024-03-24T09:19:44.212168+0000","last_fullsized":"2024-03-24T09:19:44.212168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.212236+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:19:44.212236+0000","last_peered":"2024-03-24T09:19:44.212236+0000","last_clean":"2024-03-24T09:19:44.212236+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:19:44.212236+0000","last_undegraded":"2024-03-24T09:19:44.212236+0000","last_fullsized":"2024-03-24T09:19:44.212236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.210987+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:19:44.210987+0000","last_peered":"2024-03-24T09:19:44.210987+0000","last_clean":"2024-03-24T09:19:44.210987+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:19:44.210987+0000","last_undegraded":"2024-03-24T09:19:44.210987+0000","last_fullsized":"2024-03-24T09:19:44.210987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211236+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:19:44.211236+0000","last_peered":"2024-03-24T09:19:44.211236+0000","last_clean":"2024-03-24T09:19:44.211236+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:19:44.211236+0000","last_undegraded":"2024-03-24T09:19:44.211236+0000","last_fullsized":"2024-03-24T09:19:44.211236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-24T09:19:44.211503+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:19:44.211503+0000","last_peered":"2024-03-24T09:19:44.211503+0000","last_clean":"2024-03-24T09:19:44.211503+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:19:44.211503+0000","last_undegraded":"2024-03-24T09:19:44.211503+0000","last_fullsized":"2024-03-24T09:19:44.211503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280427,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":44793,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":30,"num_read_kb":62,"num_write":160,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":344064,"data_stored":80827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32256,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712896,"statfs":{"total":95995035648,"available":95962005504,"internally_reserved":0,"allocated":1105920,"data_stored":1110181,"data_compressed":50370,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":478033,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":1068165,"data_compressed":50370,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":477215,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":481639,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":479393,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":482407,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477409,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477859,"data_compressed":14702,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":1069059,"data_compressed":50370,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":23723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":42598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:20:07.961 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T09:20:07.962 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T09:20:07.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T09:20:08.235 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:08.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:08.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:08.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] up:active 2024-03-24T09:20:08.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-24T09:20:08.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-24T09:20:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:20:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1716264360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:20:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:07 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3219627815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] up:active 2024-03-24T09:20:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-24T09:20:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1716264360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3219627815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: 2024-03-24T09:20:07.989+0000 7f12dea0a700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:20:07 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: 2024-03-24T09:20:07.989+0000 7f12dea0a700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:08.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:20:08.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] up:active 2024-03-24T09:20:08.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-24T09:20:08.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-24T09:20:08.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:20:08.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1716264360' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:20:08.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:07 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3219627815' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:08 smithi143 ceph-mon[35965]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:20:09.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:08 smithi143 ceph-mon[35965]: pgmap v306: 129 pgs: 129 active+clean; 620 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-24T09:20:09.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:08 smithi143 ceph-mon[35965]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:20:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:08 smithi117 ceph-mon[35788]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:20:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:08 smithi117 ceph-mon[35788]: pgmap v306: 129 pgs: 129 active+clean; 620 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-24T09:20:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:08 smithi117 ceph-mon[35788]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:20:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:08 smithi116 ceph-mon[32207]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:20:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:08 smithi116 ceph-mon[32207]: pgmap v306: 129 pgs: 129 active+clean; 620 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-24T09:20:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:08 smithi116 ceph-mon[32207]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 -- 172.21.15.116:0/1264632661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c074c00 msgr2=0x7fac3c075080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 --2- 172.21.15.116:0/1264632661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c074c00 0x7fac3c075080 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fac2c009990 tx=0x7fac2c009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 -- 172.21.15.116:0/1264632661 shutdown_connections 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 --2- 172.21.15.116:0/1264632661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac3c10a130 0x7fac3c10c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 --2- 172.21.15.116:0/1264632661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac3c0761d0 0x7fac3c074630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 --2- 172.21.15.116:0/1264632661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c074c00 0x7fac3c075080 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 -- 172.21.15.116:0/1264632661 >> 172.21.15.116:0/1264632661 conn(0x7fac3c0fd810 msgr2=0x7fac3c0ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 -- 172.21.15.116:0/1264632661 shutdown_connections 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.147+0000 7fac441b6700 1 -- 172.21.15.116:0/1264632661 wait complete. 2024-03-24T09:20:10.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 Processor -- start 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 -- start start 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac3c074c00 0x7fac3c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 0x7fac3c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac3c10a130 0x7fac3c1a1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1a1dd0 con 0x7fac3c0761d0 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1a1f10 con 0x7fac3c074c00 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac441b6700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fac3c1a20a0 con 0x7fac3c10a130 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 0x7fac3c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 0x7fac3c19a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34522/0 (socket says 172.21.15.116:34522) 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 -- 172.21.15.116:0/1492056014 learned_addr learned my addr 172.21.15.116:0/1492056014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41f52700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac3c074c00 0x7fac3c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac42753700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac3c10a130 0x7fac3c1a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 -- 172.21.15.116:0/1492056014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac3c10a130 msgr2=0x7fac3c1a1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac3c10a130 0x7fac3c1a1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 -- 172.21.15.116:0/1492056014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac3c074c00 msgr2=0x7fac3c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac3c074c00 0x7fac3c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.148+0000 7fac41751700 1 -- 172.21.15.116:0/1492056014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2c0095f0 con 0x7fac3c0761d0 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.149+0000 7fac41751700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 0x7fac3c19a780 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fac2c004710 tx=0x7fac2c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.149+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c01d070 con 0x7fac3c0761d0 2024-03-24T09:20:10.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.149+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fac2c021410 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.149+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3c1a2320 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.149+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c017650 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.149+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3c1a2730 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.150+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac3c108f40 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.151+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fac2c021580 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.151+0000 7fac32ffd700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac280a2290 0x7fac280a4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.151+0000 7fac41f52700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac280a2290 0x7fac280a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.151+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fac2c0cf300 con 0x7fac3c0761d0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.152+0000 7fac41f52700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac280a2290 0x7fac280a4750 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fac38009c60 tx=0x7fac38009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:10.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.156+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fac2c097fc0 con 0x7fac3c0761d0 2024-03-24T09:20:10.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.426+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fac3c10fd60 con 0x7fac3c0761d0 2024-03-24T09:20:10.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.428+0000 7fac32ffd700 1 -- 172.21.15.116:0/1492056014 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fac2c09b680 con 0x7fac3c0761d0 2024-03-24T09:20:10.429 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-24T09:20:10.429 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBK8P9l3KV9GRAAnp4jtsab2g3TDsKw0YVCvw== 2024-03-24T09:20:10.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac280a2290 msgr2=0x7fac280a4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac280a2290 0x7fac280a4750 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fac38009c60 tx=0x7fac38009260 comp rx=0 tx=0).stop 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 msgr2=0x7fac3c19a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 0x7fac3c19a780 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fac2c004710 tx=0x7fac2c009580 comp rx=0 tx=0).stop 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 shutdown_connections 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac3c10a130 0x7fac3c1a1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac3c074c00 0x7fac3c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fac280a2290 0x7fac280a4750 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 --2- 172.21.15.116:0/1492056014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fac3c0761d0 0x7fac3c19a780 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:10.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.430+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 >> 172.21.15.116:0/1492056014 conn(0x7fac3c0fd810 msgr2=0x7fac3c106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:10.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.431+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 shutdown_connections 2024-03-24T09:20:10.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:10.431+0000 7fac441b6700 1 -- 172.21.15.116:0/1492056014 wait complete. 2024-03-24T09:20:11.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:10 smithi116 ceph-mon[32207]: pgmap v307: 129 pgs: 129 active+clean; 623 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.6 KiB/s wr, 19 op/s 2024-03-24T09:20:11.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:10 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1492056014' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:20:11.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:10 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1492056014' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:20:11.253 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:20:11.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T09:20:11.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T09:20:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:10 smithi143 ceph-mon[35965]: pgmap v307: 129 pgs: 129 active+clean; 623 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.6 KiB/s wr, 19 op/s 2024-03-24T09:20:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:10 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1492056014' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:20:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:10 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1492056014' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:20:11.297 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T09:20:11.327 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T09:20:11.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T09:20:11.328 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:10 smithi117 ceph-mon[35788]: pgmap v307: 129 pgs: 129 active+clean; 623 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.6 KiB/s wr, 19 op/s 2024-03-24T09:20:11.328 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:10 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1492056014' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T09:20:11.328 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:10 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1492056014' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T09:20:11.575 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:13.236 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:12 smithi116 ceph-mon[32207]: pgmap v308: 129 pgs: 129 active+clean; 623 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-03-24T09:20:13.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:12 smithi143 ceph-mon[35965]: pgmap v308: 129 pgs: 129 active+clean; 623 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-03-24T09:20:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:12 smithi117 ceph-mon[35788]: pgmap v308: 129 pgs: 129 active+clean; 623 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.9 KiB/s wr, 20 op/s 2024-03-24T09:20:13.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/807019197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88102c10 msgr2=0x7f2d8810f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/807019197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88102c10 0x7f2d8810f050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000b0d0 tx=0x7f2d8000b3e0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/807019197 shutdown_connections 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/807019197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88102c10 0x7f2d8810f050 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/807019197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88108230 0x7f2d88108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/807019197 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102250 0x7f2d881026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/807019197 >> 172.21.15.116:0/807019197 conn(0x7f2d88079330 msgr2=0x7f2d88079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/807019197 shutdown_connections 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.469+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/807019197 wait complete. 2024-03-24T09:20:13.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 Processor -- start 2024-03-24T09:20:13.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 -- start start 2024-03-24T09:20:13.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88102250 0x7f2d88198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:13.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102c10 0x7f2d88198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:13.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 0x7f2d8819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:13.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8819d630 con 0x7f2d88108230 2024-03-24T09:20:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8819d7a0 con 0x7f2d88102250 2024-03-24T09:20:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.470+0000 7f2d8d0d0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8819d960 con 0x7f2d88102c10 2024-03-24T09:20:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 0x7f2d8819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:13.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d86d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88102250 0x7f2d88198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8759e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 0x7f2d8819d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34538/0 (socket says 172.21.15.116:34538) 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8759e700 1 -- 172.21.15.116:0/1789133193 learned_addr learned my addr 172.21.15.116:0/1789133193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d86d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88102250 0x7f2d88198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:38260/0 (socket says 172.21.15.116:38260) 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102c10 0x7f2d88198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 -- 172.21.15.116:0/1789133193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88102250 msgr2=0x7f2d88198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88102250 0x7f2d88198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 -- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 msgr2=0x7f2d8819d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 0x7f2d8819d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 -- 172.21.15.116:0/1789133193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d80009d00 con 0x7f2d88102c10 2024-03-24T09:20:13.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8759e700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 0x7f2d8819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8659c700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102c10 0x7f2d88198c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800e9f0 tx=0x7f2d7800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800c9c0 con 0x7f2d88102c10 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8819dc40 con 0x7f2d88102c10 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2d78010430 con 0x7f2d88102c10 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78018680 con 0x7f2d88102c10 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.471+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d881a54a0 con 0x7f2d88102c10 2024-03-24T09:20:13.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.472+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d8810a710 con 0x7f2d88102c10 2024-03-24T09:20:13.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.473+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f2d780105a0 con 0x7f2d88102c10 2024-03-24T09:20:13.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.474+0000 7f2d6ffff700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2d700a21e0 0x7f2d700a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:13.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.474+0000 7f2d86d9d700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2d700a21e0 0x7f2d700a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:13.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.474+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2d78014070 con 0x7f2d88102c10 2024-03-24T09:20:13.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.474+0000 7f2d86d9d700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2d700a21e0 0x7f2d700a46a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009be0 tx=0x7f2d7c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:13.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.482+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2d78096fd0 con 0x7f2d88102c10 2024-03-24T09:20:13.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.730+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2d88199ab0 con 0x7f2d88102c10 2024-03-24T09:20:13.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.735+0000 7f2d6ffff700 1 -- 172.21.15.116:0/1789133193 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+261477 (secure 0 0 0) 0x7f2d78026090 con 0x7f2d88102c10 2024-03-24T09:20:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:13.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2d700a21e0 msgr2=0x7f2d700a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2d700a21e0 0x7f2d700a46a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009be0 tx=0x7f2d7c009260 comp rx=0 tx=0).stop 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102c10 msgr2=0x7f2d88198c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102c10 0x7f2d88198c90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800e9f0 tx=0x7f2d7800ed00 comp rx=0 tx=0).stop 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 shutdown_connections 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2d700a21e0 0x7f2d700a46a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d88108230 0x7f2d8819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d88102250 0x7f2d88198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 --2- 172.21.15.116:0/1789133193 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2d88102c10 0x7f2d88198c90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:13.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 >> 172.21.15.116:0/1789133193 conn(0x7f2d88079330 msgr2=0x7f2d88100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:13.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 shutdown_connections 2024-03-24T09:20:13.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:13.738+0000 7f2d8d0d0700 1 -- 172.21.15.116:0/1789133193 wait complete. 2024-03-24T09:20:14.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1789133193' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T09:20:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1789133193' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T09:20:14.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1789133193' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T09:20:14.558 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2363601294},{"type":"v1","addr":"172.21.15.116:6801","nonce":2363601294}]},"active_addr":"172.21.15.116:6801/2363601294","active_change":"2024-03-24T09:12:33.839962+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.116:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":664840167}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1503975320}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":227139610}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":4025940557}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":2244931405}]}]} 2024-03-24T09:20:14.563 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T09:20:14.563 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T09:20:14.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T09:20:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:15.192 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:14 smithi116 ceph-mon[32207]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-24T09:20:15.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:14 smithi143 ceph-mon[35965]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-24T09:20:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:14 smithi117 ceph-mon[35788]: pgmap v309: 129 pgs: 129 active+clean; 628 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 -- 172.21.15.116:0/774364507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898101100 msgr2=0x7f4898101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 --2- 172.21.15.116:0/774364507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898101100 0x7f4898101560 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4894009990 tx=0x7f4894009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 -- 172.21.15.116:0/774364507 shutdown_connections 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 --2- 172.21.15.116:0/774364507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 0x7f489810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 --2- 172.21.15.116:0/774364507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898107140 0x7f4898107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 --2- 172.21.15.116:0/774364507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898101100 0x7f4898101560 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:16.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.835+0000 7f489f9f0700 1 -- 172.21.15.116:0/774364507 >> 172.21.15.116:0/774364507 conn(0x7f4898078030 msgr2=0x7f4898078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 -- 172.21.15.116:0/774364507 shutdown_connections 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 -- 172.21.15.116:0/774364507 wait complete. 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 Processor -- start 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 -- start start 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898101100 0x7f4898198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 0x7f4898198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898107140 0x7f489819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f489819d4e0 con 0x7f4898107140 2024-03-24T09:20:16.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f489819d650 con 0x7f4898101aa0 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489f9f0700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f489819d810 con 0x7f4898101100 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489df8d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898107140 0x7f489819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.836+0000 7f489d78c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898101100 0x7f4898198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 0x7f4898198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489d78c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898101100 0x7f4898198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:58656/0 (socket says 172.21.15.116:58656) 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489d78c700 1 -- 172.21.15.116:0/2115838069 learned_addr learned my addr 172.21.15.116:0/2115838069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 -- 172.21.15.116:0/2115838069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898101100 msgr2=0x7f4898198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898101100 0x7f4898198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 -- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898107140 msgr2=0x7f489819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:16.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898107140 0x7f489819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 -- 172.21.15.116:0/2115838069 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48940095f0 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489df8d700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898107140 0x7f489819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489cf8b700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 0x7f4898198b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48940059d0 tx=0x7f4894004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489401d070 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f489819da90 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4894022470 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4894017650 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.837+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48981a5350 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.838+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48981027f0 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.839+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f48940225e0 con 0x7f4898101aa0 2024-03-24T09:20:16.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.840+0000 7f488e7fc700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48840a21e0 0x7f48840a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.840+0000 7f489d78c700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48840a21e0 0x7f48840a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.840+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f48940cf0c0 con 0x7f4898101aa0 2024-03-24T09:20:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.840+0000 7f489d78c700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48840a21e0 0x7f48840a46a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4888009ca0 tx=0x7f4888009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:16.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:16.848+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4894097d80 con 0x7f4898101aa0 2024-03-24T09:20:17.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.051+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48981999c0 con 0x7f4898101aa0 2024-03-24T09:20:17.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.052+0000 7f488e7fc700 1 -- 172.21.15.116:0/2115838069 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7f489409b440 con 0x7f4898101aa0 2024-03-24T09:20:17.053 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:17.054 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:20:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.054+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48840a21e0 msgr2=0x7f48840a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:17.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.054+0000 7f489f9f0700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48840a21e0 0x7f48840a46a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4888009ca0 tx=0x7f4888009260 comp rx=0 tx=0).stop 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.054+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 msgr2=0x7f4898198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.054+0000 7f489f9f0700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 0x7f4898198b40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f48940059d0 tx=0x7f4894004730 comp rx=0 tx=0).stop 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 shutdown_connections 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48840a21e0 0x7f48840a46a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4898107140 0x7f489819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4898101100 0x7f4898198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 --2- 172.21.15.116:0/2115838069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4898101aa0 0x7f4898198b40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 >> 172.21.15.116:0/2115838069 conn(0x7f4898078030 msgr2=0x7f489806a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 shutdown_connections 2024-03-24T09:20:17.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:17.055+0000 7f489f9f0700 1 -- 172.21.15.116:0/2115838069 wait complete. 2024-03-24T09:20:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:16 smithi143 ceph-mon[35965]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.4 KiB/s wr, 25 op/s 2024-03-24T09:20:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:16 smithi117 ceph-mon[35788]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.4 KiB/s wr, 25 op/s 2024-03-24T09:20:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:16 smithi116 ceph-mon[32207]: pgmap v310: 129 pgs: 129 active+clean; 628 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 8.4 KiB/s wr, 25 op/s 2024-03-24T09:20:17.830 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T09:20:17.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T09:20:18.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:17 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2115838069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:18.077 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:18.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:17 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2115838069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:18.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:17 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2115838069' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:18 smithi143 ceph-mon[35965]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 19 op/s 2024-03-24T09:20:19.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:18 smithi116 ceph-mon[32207]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 19 op/s 2024-03-24T09:20:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:18 smithi117 ceph-mon[35788]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 19 op/s 2024-03-24T09:20:19.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 -- 172.21.15.116:0/305352890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4102c40 msgr2=0x7f0cb410f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 --2- 172.21.15.116:0/305352890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4102c40 0x7f0cb410f0b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00b0d0 tx=0x7f0cac00b3e0 comp rx=0 tx=0).stop 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 -- 172.21.15.116:0/305352890 shutdown_connections 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 --2- 172.21.15.116:0/305352890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4102c40 0x7f0cb410f0b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 --2- 172.21.15.116:0/305352890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4108290 0x7f0cb4108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 --2- 172.21.15.116:0/305352890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb4102280 0x7f0cb4102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 -- 172.21.15.116:0/305352890 >> 172.21.15.116:0/305352890 conn(0x7f0cb4079360 msgr2=0x7f0cb4079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 -- 172.21.15.116:0/305352890 shutdown_connections 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.869+0000 7f0cba65e700 1 -- 172.21.15.116:0/305352890 wait complete. 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 Processor -- start 2024-03-24T09:20:19.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 -- start start 2024-03-24T09:20:19.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4102280 0x7f0cb4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:19.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb4102c40 0x7f0cb4198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:19.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 0x7f0cb419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb419d560 con 0x7f0cb4108290 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb419d6d0 con 0x7f0cb4102280 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.870+0000 7f0cba65e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb419d890 con 0x7f0cb4102c40 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 0x7f0cb419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4102280 0x7f0cb4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 0x7f0cb419cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54388/0 (socket says 172.21.15.116:54388) 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 -- 172.21.15.116:0/639955735 learned_addr learned my addr 172.21.15.116:0/639955735 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb37fe700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb4102c40 0x7f0cb4198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 -- 172.21.15.116:0/639955735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb4102c40 msgr2=0x7f0cb4198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb4102c40 0x7f0cb4198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:19.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 -- 172.21.15.116:0/639955735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4102280 msgr2=0x7f0cb4198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4102280 0x7f0cb4198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 -- 172.21.15.116:0/639955735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cac009d00 con 0x7f0cb4108290 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb3fff700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4102280 0x7f0cb4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb8bfb700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 0x7f0cb419cf50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00bd50 tx=0x7f0cac00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac00e040 con 0x7f0cb4108290 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb419db10 con 0x7f0cb4108290 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0cac01b460 con 0x7f0cb4108290 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0052b0 con 0x7f0cb4108290 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.871+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb41a53d0 con 0x7f0cb4108290 2024-03-24T09:20:19.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.872+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb4103940 con 0x7f0cb4108290 2024-03-24T09:20:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.873+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f0cac01b5d0 con 0x7f0cb4108290 2024-03-24T09:20:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.874+0000 7f0cb17fa700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0c9c0a21e0 0x7f0c9c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.874+0000 7f0cb3fff700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0c9c0a21e0 0x7f0c9c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.874+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0cac0cee60 con 0x7f0cb4108290 2024-03-24T09:20:19.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.874+0000 7f0cb3fff700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0c9c0a21e0 0x7f0c9c0a46a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8011490 tx=0x7f0ca8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:19.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:19.879+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0cac097b20 con 0x7f0cb4108290 2024-03-24T09:20:20.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.079+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0cb4112170 con 0x7f0cb4108290 2024-03-24T09:20:20.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.081+0000 7f0cb17fa700 1 -- 172.21.15.116:0/639955735 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24197 (secure 0 0 0) 0x7f0cac09b1e0 con 0x7f0cb4108290 2024-03-24T09:20:20.083 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:20:20.083 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:19:43.082734+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:20:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0c9c0a21e0 msgr2=0x7f0c9c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0c9c0a21e0 0x7f0c9c0a46a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8011490 tx=0x7f0ca8009260 comp rx=0 tx=0).stop 2024-03-24T09:20:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 msgr2=0x7f0cb419cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:20.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 0x7f0cb419cf50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00bd50 tx=0x7f0cac00bd80 comp rx=0 tx=0).stop 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 shutdown_connections 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0c9c0a21e0 0x7f0c9c0a46a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cb4108290 0x7f0cb419cf50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cb4102280 0x7f0cb4198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 --2- 172.21.15.116:0/639955735 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cb4102c40 0x7f0cb4198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 >> 172.21.15.116:0/639955735 conn(0x7f0cb4079360 msgr2=0x7f0cb4100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 shutdown_connections 2024-03-24T09:20:20.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:20.083+0000 7f0cba65e700 1 -- 172.21.15.116:0/639955735 wait complete. 2024-03-24T09:20:20.879 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T09:20:20.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T09:20:20.880 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T09:20:20.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T09:20:20.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T09:20:20.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T09:20:20.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T09:20:20.883 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T09:20:20.884 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-24T09:20:20.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-24T09:20:20.885 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-24T09:20:20.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-24T09:20:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:20 smithi143 ceph-mon[35965]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-24T09:20:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:20 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/639955735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:20 smithi117 ceph-mon[35788]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-24T09:20:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:20 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/639955735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:21.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:20 smithi116 ceph-mon[32207]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 4.2 KiB/s wr, 18 op/s 2024-03-24T09:20:21.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:20 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/639955735' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:20:21.551 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:21.564 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:21.565 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:22.620 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:22.621 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:22.624 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:22.626 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:23.054 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:23.258 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:23 smithi116 ceph-mon[32207]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-24T09:20:23.272 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:23 smithi143 ceph-mon[35965]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-24T09:20:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:23 smithi117 ceph-mon[35788]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-03-24T09:20:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:25 smithi143 ceph-mon[35965]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-24T09:20:25.299 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:25 smithi116 ceph-mon[32207]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-24T09:20:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:25 smithi117 ceph-mon[35788]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-24T09:20:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:27 smithi143 ceph-mon[35965]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-24T09:20:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:27 smithi117 ceph-mon[35788]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-24T09:20:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:27 smithi116 ceph-mon[32207]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-24T09:20:29.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:29 smithi116 ceph-mon[32207]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-24T09:20:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:29 smithi143 ceph-mon[35965]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-24T09:20:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:29 smithi117 ceph-mon[35788]: pgmap v316: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-03-24T09:20:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.953+0000 7f13aad8c700 1 -- 172.21.15.116:0/2445111873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41075d0 msgr2=0x7f13a410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.953+0000 7f13aad8c700 1 --2- 172.21.15.116:0/2445111873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41075d0 0x7f13a410f1e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f139c00b0a0 tx=0x7f139c00b3b0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- 172.21.15.116:0/2445111873 shutdown_connections 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 --2- 172.21.15.116:0/2445111873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41075d0 0x7f13a410f1e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 --2- 172.21.15.116:0/2445111873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a4100e00 0x7f13a41011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 --2- 172.21.15.116:0/2445111873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a41017b0 0x7f13a4106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- 172.21.15.116:0/2445111873 >> 172.21.15.116:0/2445111873 conn(0x7f13a4076480 msgr2=0x7f13a4076890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- 172.21.15.116:0/2445111873 shutdown_connections 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- 172.21.15.116:0/2445111873 wait complete. 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 Processor -- start 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- start start 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a4100e00 0x7f13a419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41017b0 0x7f13a419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a41075d0 0x7f13a4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4198e00 con 0x7f13a41017b0 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4198f70 con 0x7f13a4100e00 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13aad8c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f13a4199130 con 0x7f13a41075d0 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.954+0000 7f13a3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41017b0 0x7f13a419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a8b28700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a4100e00 0x7f13a419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41017b0 0x7f13a419eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32770/0 (socket says 172.21.15.116:32770) 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 -- 172.21.15.116:0/621498140 learned_addr learned my addr 172.21.15.116:0/621498140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a9329700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a41075d0 0x7f13a4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 -- 172.21.15.116:0/621498140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a41075d0 msgr2=0x7f13a4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:29.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a41075d0 0x7f13a4198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 -- 172.21.15.116:0/621498140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a4100e00 msgr2=0x7f13a419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a4100e00 0x7f13a419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 -- 172.21.15.116:0/621498140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c009d00 con 0x7f13a41017b0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a3fff700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41017b0 0x7f13a419eac0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f139000e970 tx=0x7f139000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390009790 con 0x7f13a41017b0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f139000c7e0 con 0x7f13a41017b0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a4199410 con 0x7f13a41017b0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13900054e0 con 0x7f13a41017b0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.955+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a41a5550 con 0x7f13a41017b0 2024-03-24T09:20:29.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.956+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13a4112500 con 0x7f13a41017b0 2024-03-24T09:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.957+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f139000c950 con 0x7f13a41017b0 2024-03-24T09:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a1ffb700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13940a2260 0x7f13940a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a8b28700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13940a2260 0x7f13940a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1390014070 con 0x7f13a41017b0 2024-03-24T09:20:29.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a1ffb700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] conn(0x7f13940acf20 0x7f13940af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 --> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13940afa30 con 0x7f13940acf20 2024-03-24T09:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f13900ce740 con 0x7f13a41017b0 2024-03-24T09:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a9329700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] conn(0x7f13940acf20 0x7f13940af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.958+0000 7f13a8b28700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13940a2260 0x7f13940a4720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1398009c60 tx=0x7f1398009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.959+0000 7f13a9329700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] conn(0x7f13940acf20 0x7f13940af360 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.960+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== osd.3 v2:172.21.15.116:6810/2981524673 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13940afa30 con 0x7f13940acf20 2024-03-24T09:20:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.976+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 --> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f13a4062890 con 0x7f13940acf20 2024-03-24T09:20:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.976+0000 7f13a1ffb700 1 -- 172.21.15.116:0/621498140 <== osd.3 v2:172.21.15.116:6810/2981524673 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f13a4062890 con 0x7f13940acf20 2024-03-24T09:20:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] conn(0x7f13940acf20 msgr2=0x7f13940af360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] conn(0x7f13940acf20 0x7f13940af360 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13940a2260 msgr2=0x7f13940a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13940a2260 0x7f13940a4720 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1398009c60 tx=0x7f1398009260 comp rx=0 tx=0).stop 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41017b0 msgr2=0x7f13a419eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 --2- 172.21.15.116:0/621498140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a41017b0 0x7f13a419eac0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f139000e970 tx=0x7f139000ec80 comp rx=0 tx=0).stop 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13a9329700 1 -- 172.21.15.116:0/621498140 reap_dead start 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 shutdown_connections 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 >> 172.21.15.116:0/621498140 conn(0x7f13a4076480 msgr2=0x7f13a40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 shutdown_connections 2024-03-24T09:20:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:29.977+0000 7f13aad8c700 1 -- 172.21.15.116:0/621498140 wait complete. 2024-03-24T09:20:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:31 smithi143 ceph-mon[35965]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T09:20:31.306 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:31 smithi116 ceph-mon[32207]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T09:20:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:31 smithi117 ceph-mon[35788]: pgmap v317: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T09:20:32.538 INFO:teuthology.orchestra.run.smithi116.stdout:107374182465 2024-03-24T09:20:32.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T09:20:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:33 smithi143 ceph-mon[35965]: pgmap v318: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-24T09:20:33.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:33 smithi116 ceph-mon[32207]: pgmap v318: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-24T09:20:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:33 smithi117 ceph-mon[35788]: pgmap v318: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-24T09:20:34.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-24T09:20:34.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-24T09:20:34.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:34 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-24T09:20:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-24T09:20:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-24T09:20:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-24T09:20:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-24T09:20:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-03-24T09:20:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]: dispatch 2024-03-24T09:20:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-24T09:20:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-24T09:20:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-24T09:20:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:34 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-24T09:20:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-24T09:20:34.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:34 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-03-24T09:20:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: pgmap v319: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-24T09:20:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:35 smithi117 ceph-mon[35788]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: pgmap v319: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-24T09:20:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-03-24T09:20:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]': finished 2024-03-24T09:20:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-24T09:20:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-24T09:20:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-24T09:20:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-24T09:20:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:35 smithi116 ceph-mon[32207]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T09:20:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: pgmap v319: 129 pgs: 129 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-24T09:20:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [1, 9]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 6]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-03-24T09:20:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:35 smithi143 ceph-mon[35965]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T09:20:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:36 smithi117 ceph-mon[35788]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T09:20:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:36 smithi117 ceph-mon[35788]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-24T09:20:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:36 smithi116 ceph-mon[32207]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T09:20:36.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:36 smithi116 ceph-mon[32207]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-24T09:20:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:36 smithi143 ceph-mon[35965]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T09:20:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:36 smithi143 ceph-mon[35965]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 644 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-24T09:20:38.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:37 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:20:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 -- 172.21.15.116:0/3349184906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c101aa0 msgr2=0x7fd08c10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 --2- 172.21.15.116:0/3349184906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c101aa0 0x7fd08c10df80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd08400b0a0 tx=0x7fd08400b3b0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 -- 172.21.15.116:0/3349184906 shutdown_connections 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 --2- 172.21.15.116:0/3349184906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c101aa0 0x7fd08c10df80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 --2- 172.21.15.116:0/3349184906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c107140 0x7fd08c107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 --2- 172.21.15.116:0/3349184906 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101100 0x7fd08c101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 -- 172.21.15.116:0/3349184906 >> 172.21.15.116:0/3349184906 conn(0x7fd08c078030 msgr2=0x7fd08c078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 -- 172.21.15.116:0/3349184906 shutdown_connections 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.113+0000 7fd0909dd700 1 -- 172.21.15.116:0/3349184906 wait complete. 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 Processor -- start 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 -- start start 2024-03-24T09:20:38.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c101100 0x7fd08c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101aa0 0x7fd08c198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c107140 0x7fd08c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c19d5e0 con 0x7fd08c107140 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c19d750 con 0x7fd08c101100 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd0909dd700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c19d910 con 0x7fd08c101aa0 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd08a59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c101100 0x7fd08c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd08ad9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c107140 0x7fd08c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd089d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101aa0 0x7fd08c198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd08a59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c101100 0x7fd08c198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46072/0 (socket says 172.21.15.116:46072) 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd089d9b700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101aa0 0x7fd08c198c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41732/0 (socket says 172.21.15.116:41732) 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.114+0000 7fd08a59c700 1 -- 172.21.15.116:0/196958711 learned_addr learned my addr 172.21.15.116:0/196958711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:38.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08ad9d700 1 -- 172.21.15.116:0/196958711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101aa0 msgr2=0x7fd08c198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08ad9d700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101aa0 0x7fd08c198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08ad9d700 1 -- 172.21.15.116:0/196958711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c101100 msgr2=0x7fd08c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08ad9d700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c101100 0x7fd08c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08ad9d700 1 -- 172.21.15.116:0/196958711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd084009d00 con 0x7fd08c107140 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08a59c700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd08c101100 0x7fd08c198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd089d9b700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd08c101aa0 0x7fd08c198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd08ad9d700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c107140 0x7fd08c19cfd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd08400b0a0 tx=0x7fd08400bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08400e040 con 0x7fd08c107140 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd084005040 con 0x7fd08c107140 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08c19db90 con 0x7fd08c107140 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08401b680 con 0x7fd08c107140 2024-03-24T09:20:38.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.115+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd08c1a5450 con 0x7fd08c107140 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.116+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd08c1112d0 con 0x7fd08c107140 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.117+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fd0840051b0 con 0x7fd08c107140 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd07b7fe700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0740a21e0 0x7fd0740a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd08a59c700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0740a21e0 0x7fd0740a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd0840cf320 con 0x7fd08c107140 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd07b7fe700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] conn(0x7fd0740ad340 0x7fd0740af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 --> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0740afeb0 con 0x7fd0740ad340 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fd0840d1a30 con 0x7fd08c107140 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd089d9b700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] conn(0x7fd0740ad340 0x7fd0740af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.118+0000 7fd08a59c700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0740a21e0 0x7fd0740a46a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0103d0 tx=0x7fd07c007480 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.119+0000 7fd089d9b700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] conn(0x7fd0740ad340 0x7fd0740af7e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.119+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== osd.6 v2:172.21.15.116:6818/254462190 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd0740afeb0 con 0x7fd0740ad340 2024-03-24T09:20:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 --> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd08c1051f0 con 0x7fd0740ad340 2024-03-24T09:20:38.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd07b7fe700 1 -- 172.21.15.116:0/196958711 <== osd.6 v2:172.21.15.116:6818/254462190 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd08c1051f0 con 0x7fd0740ad340 2024-03-24T09:20:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] conn(0x7fd0740ad340 msgr2=0x7fd0740af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] conn(0x7fd0740ad340 0x7fd0740af7e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0740a21e0 msgr2=0x7fd0740a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd0740a21e0 0x7fd0740a46a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0103d0 tx=0x7fd07c007480 comp rx=0 tx=0).stop 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c107140 msgr2=0x7fd08c19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd0909dd700 1 --2- 172.21.15.116:0/196958711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c107140 0x7fd08c19cfd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd08400b0a0 tx=0x7fd08400bdb0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.136+0000 7fd08ad9d700 1 -- 172.21.15.116:0/196958711 reap_dead start 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.137+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 shutdown_connections 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.137+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 >> 172.21.15.116:0/196958711 conn(0x7fd08c078030 msgr2=0x7fd08c1004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.137+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 shutdown_connections 2024-03-24T09:20:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.137+0000 7fd0909dd700 1 -- 172.21.15.116:0/196958711 wait complete. 2024-03-24T09:20:38.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:37 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:20:38.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:37 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 -- 172.21.15.116:0/2568426713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 msgr2=0x7f8e68100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 --2- 172.21.15.116:0/2568426713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e68100c70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8e58009990 tx=0x7f8e58009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 -- 172.21.15.116:0/2568426713 shutdown_connections 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 --2- 172.21.15.116:0/2568426713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e681011b0 0x7f8e6810f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 --2- 172.21.15.116:0/2568426713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e68107270 0x7f8e68107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 --2- 172.21.15.116:0/2568426713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e68100c70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 -- 172.21.15.116:0/2568426713 >> 172.21.15.116:0/2568426713 conn(0x7f8e68077700 msgr2=0x7f8e680feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 -- 172.21.15.116:0/2568426713 shutdown_connections 2024-03-24T09:20:38.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.458+0000 7f8e6f399700 1 -- 172.21.15.116:0/2568426713 wait complete. 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 Processor -- start 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 -- start start 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e6819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e681011b0 0x7f8e6819e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e68107270 0x7f8e68198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68198ca0 con 0x7f8e681007f0 2024-03-24T09:20:38.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68198de0 con 0x7f8e68107270 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6f399700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8e68198fa0 con 0x7f8e681011b0 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6d135700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e6819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6d135700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e6819e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59370/0 (socket says 172.21.15.116:59370) 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6d135700 1 -- 172.21.15.116:0/3696802661 learned_addr learned my addr 172.21.15.116:0/3696802661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.459+0000 7f8e6c934700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e681011b0 0x7f8e6819e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d135700 1 -- 172.21.15.116:0/3696802661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e681011b0 msgr2=0x7f8e6819e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d135700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8e681011b0 0x7f8e6819e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d135700 1 -- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e68107270 msgr2=0x7f8e68198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d936700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e68107270 0x7f8e68198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d135700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e68107270 0x7f8e68198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d135700 1 -- 172.21.15.116:0/3696802661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e580095f0 con 0x7f8e681007f0 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d936700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e68107270 0x7f8e68198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:20:38.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6d135700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e6819e430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000d980 tx=0x7f8e6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60004220 con 0x7f8e681007f0 2024-03-24T09:20:38.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8e60007640 con 0x7f8e681007f0 2024-03-24T09:20:38.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60005680 con 0x7f8e681007f0 2024-03-24T09:20:38.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e68199280 con 0x7f8e681007f0 2024-03-24T09:20:38.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.460+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e681a54d0 con 0x7f8e681007f0 2024-03-24T09:20:38.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.461+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e681125d0 con 0x7f8e681007f0 2024-03-24T09:20:38.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.463+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f8e60004750 con 0x7f8e681007f0 2024-03-24T09:20:38.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.463+0000 7f8e5e7fc700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8e540a2260 0x7f8e540a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.463+0000 7f8e6c934700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8e540a2260 0x7f8e540a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8e600ce5c0 con 0x7f8e681007f0 2024-03-24T09:20:38.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e5e7fc700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] conn(0x7f8e540ad3c0 0x7f8e540af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 --> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e540aff30 con 0x7f8e540ad3c0 2024-03-24T09:20:38.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8e600ce9d0 con 0x7f8e681007f0 2024-03-24T09:20:38.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e6c934700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8e540a2260 0x7f8e540a4720 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800b5c0 tx=0x7f8e58005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e6d936700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] conn(0x7f8e540ad3c0 0x7f8e540af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.464+0000 7f8e6d936700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] conn(0x7f8e540ad3c0 0x7f8e540af860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.465+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== osd.7 v2:172.21.15.117:6816/3523258939 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e540aff30 con 0x7f8e540ad3c0 2024-03-24T09:20:38.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.482+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 --> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e68062890 con 0x7f8e540ad3c0 2024-03-24T09:20:38.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e5e7fc700 1 -- 172.21.15.116:0/3696802661 <== osd.7 v2:172.21.15.117:6816/3523258939 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8e68062890 con 0x7f8e540ad3c0 2024-03-24T09:20:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] conn(0x7f8e540ad3c0 msgr2=0x7f8e540af860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] conn(0x7f8e540ad3c0 0x7f8e540af860 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8e540a2260 msgr2=0x7f8e540a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8e540a2260 0x7f8e540a4720 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800b5c0 tx=0x7f8e58005960 comp rx=0 tx=0).stop 2024-03-24T09:20:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 msgr2=0x7f8e6819e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 --2- 172.21.15.116:0/3696802661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e681007f0 0x7f8e6819e430 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8e6000d980 tx=0x7f8e6000dc90 comp rx=0 tx=0).stop 2024-03-24T09:20:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6d936700 1 -- 172.21.15.116:0/3696802661 reap_dead start 2024-03-24T09:20:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 shutdown_connections 2024-03-24T09:20:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 >> 172.21.15.116:0/3696802661 conn(0x7f8e68077700 msgr2=0x7f8e681053b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 shutdown_connections 2024-03-24T09:20:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:38.483+0000 7f8e6f399700 1 -- 172.21.15.116:0/3696802661 wait complete. 2024-03-24T09:20:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 -- 172.21.15.116:0/4038804390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 msgr2=0x7f78b4105da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 --2- 172.21.15.116:0/4038804390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 0x7f78b4105da0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f78a8009990 tx=0x7f78a8009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 -- 172.21.15.116:0/4038804390 shutdown_connections 2024-03-24T09:20:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 --2- 172.21.15.116:0/4038804390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4106480 0x7f78b410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 --2- 172.21.15.116:0/4038804390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78b4101b60 0x7f78b4101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 --2- 172.21.15.116:0/4038804390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 0x7f78b4105da0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 -- 172.21.15.116:0/4038804390 >> 172.21.15.116:0/4038804390 conn(0x7f78b4077700 msgr2=0x7f78b40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 -- 172.21.15.116:0/4038804390 shutdown_connections 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.719+0000 7f78ba927700 1 -- 172.21.15.116:0/4038804390 wait complete. 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.720+0000 7f78ba927700 1 Processor -- start 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.720+0000 7f78ba927700 1 -- start start 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78ba927700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4101b60 0x7f78b4197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78ba927700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 0x7f78b4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78ba927700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78b4106480 0x7f78b419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78ba927700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78b419c4a0 con 0x7f78b4102510 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78ba927700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f78b419c610 con 0x7f78b4106480 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78ba927700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f78b419c7d0 con 0x7f78b4101b60 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4101b60 0x7f78b4197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b37fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 0x7f78b4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4101b60 0x7f78b4197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:46538/0 (socket says 172.21.15.116:46538) 2024-03-24T09:20:38.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 -- 172.21.15.116:0/2933985245 learned_addr learned my addr 172.21.15.116:0/2933985245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b8ec4700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78b4106480 0x7f78b419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 -- 172.21.15.116:0/2933985245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78b4106480 msgr2=0x7f78b419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78b4106480 0x7f78b419be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 -- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 msgr2=0x7f78b4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78b4102510 0x7f78b4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.721+0000 7f78b3fff700 1 -- 172.21.15.116:0/2933985245 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a80095f0 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.722+0000 7f78b3fff700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4101b60 0x7f78b4197610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78a400e9a0 tx=0x7f78a400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.722+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4009790 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.722+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78b419cab0 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.722+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78a4010430 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.722+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78a4018680 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.722+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78b41a4310 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.723+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f78b4111330 con 0x7f78b4101b60 2024-03-24T09:20:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.724+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f78a40105a0 con 0x7f78b4101b60 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b17fa700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f789c0a21e0 0x7f789c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b37fe700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f789c0a21e0 0x7f789c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f78a4014070 con 0x7f78b4101b60 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b17fa700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] conn(0x7f789c0acea0 0x7f789c0af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 --> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f789c0af9b0 con 0x7f789c0acea0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f78a40ce950 con 0x7f78b4101b60 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b8ec4700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] conn(0x7f789c0acea0 0x7f789c0af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.725+0000 7f78b37fe700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f789c0a21e0 0x7f789c0a46a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006010 tx=0x7f78a8005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.726+0000 7f78b8ec4700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] conn(0x7f789c0acea0 0x7f789c0af2e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.726+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== osd.9 v2:172.21.15.116:6826/2856079084 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f789c0af9b0 con 0x7f789c0acea0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.742+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 --> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f78b4062910 con 0x7f789c0acea0 2024-03-24T09:20:38.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78b17fa700 1 -- 172.21.15.116:0/2933985245 <== osd.9 v2:172.21.15.116:6826/2856079084 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f78b4062910 con 0x7f789c0acea0 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] conn(0x7f789c0acea0 msgr2=0x7f789c0af2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] conn(0x7f789c0acea0 0x7f789c0af2e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f789c0a21e0 msgr2=0x7f789c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f789c0a21e0 0x7f789c0a46a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f78a8006010 tx=0x7f78a8005960 comp rx=0 tx=0).stop 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4101b60 msgr2=0x7f78b4197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 --2- 172.21.15.116:0/2933985245 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78b4101b60 0x7f78b4197610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78a400e9a0 tx=0x7f78a400ecb0 comp rx=0 tx=0).stop 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78b8ec4700 1 -- 172.21.15.116:0/2933985245 reap_dead start 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 shutdown_connections 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 >> 172.21.15.116:0/2933985245 conn(0x7f78b4077700 msgr2=0x7f78b40feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 shutdown_connections 2024-03-24T09:20:38.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:32.743+0000 7f78ba927700 1 -- 172.21.15.116:0/2933985245 wait complete. 2024-03-24T09:20:39.024 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:38 smithi143 ceph-mon[35965]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T09:20:39.043 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:38 smithi117 ceph-mon[35788]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T09:20:39.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:38 smithi116 ceph-mon[32207]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T09:20:39.220 INFO:teuthology.orchestra.run.smithi116.stdout:287762808867 2024-03-24T09:20:39.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-24T09:20:39.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 -- 172.21.15.116:0/2981724872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 msgr2=0x7fb430107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 --2- 172.21.15.116:0/2981724872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 0x7fb430107520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb424009990 tx=0x7fb424009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 -- 172.21.15.116:0/2981724872 shutdown_connections 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 --2- 172.21.15.116:0/2981724872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb430101aa0 0x7fb43010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 --2- 172.21.15.116:0/2981724872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 0x7fb430107520 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 --2- 172.21.15.116:0/2981724872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 0x7fb430101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 -- 172.21.15.116:0/2981724872 >> 172.21.15.116:0/2981724872 conn(0x7fb430078030 msgr2=0x7fb430078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 -- 172.21.15.116:0/2981724872 shutdown_connections 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.876+0000 7fb437015700 1 -- 172.21.15.116:0/2981724872 wait complete. 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 Processor -- start 2024-03-24T09:20:39.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 -- start start 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 0x7fb430199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb430101aa0 0x7fb430199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 0x7fb4301a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb43019a250 con 0x7fb430107140 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb4301a4a10 con 0x7fb430101aa0 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb437015700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb4301a4b50 con 0x7fb430101100 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 0x7fb430199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 0x7fb430199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41708/0 (socket says 172.21.15.116:41708) 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 -- 172.21.15.116:0/1153714850 learned_addr learned my addr 172.21.15.116:0/1153714850 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb42ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb430101aa0 0x7fb430199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:39.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb4355b2700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 0x7fb4301a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 -- 172.21.15.116:0/1153714850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb430101aa0 msgr2=0x7fb430199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb430101aa0 0x7fb430199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 -- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 msgr2=0x7fb4301a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:39.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb430107140 0x7fb4301a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.877+0000 7fb434db1700 1 -- 172.21.15.116:0/1153714850 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4240095f0 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.878+0000 7fb434db1700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 0x7fb430199450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb424000c00 tx=0x7fb424004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.878+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42401d070 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.878+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4301a4dd0 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.878+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb424022470 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.878+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424017650 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.878+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4301a51e0 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.879+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb430111240 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.880+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fb4240225e0 con 0x7fb430101100 2024-03-24T09:20:39.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42dffb700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb4200a21e0 0x7fb4200a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb4240cfae0 con 0x7fb430101100 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42ffff700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb4200a21e0 0x7fb4200a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42dffb700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] conn(0x7fb4200ad340 0x7fb4200af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 --> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4200afeb0 con 0x7fb4200ad340 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb4240cfef0 con 0x7fb430101100 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb4355b2700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] conn(0x7fb4200ad340 0x7fb4200af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.881+0000 7fb42ffff700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb4200a21e0 0x7fb4200a46a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb430106230 tx=0x7fb41c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.882+0000 7fb4355b2700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] conn(0x7fb4200ad340 0x7fb4200af7e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.883+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== osd.5 v2:172.21.15.143:6808/1015586525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb4200afeb0 con 0x7fb4200ad340 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.900+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 --> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb4300628c0 con 0x7fb4200ad340 2024-03-24T09:20:39.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.900+0000 7fb42dffb700 1 -- 172.21.15.116:0/1153714850 <== osd.5 v2:172.21.15.143:6808/1015586525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb4300628c0 con 0x7fb4200ad340 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] conn(0x7fb4200ad340 msgr2=0x7fb4200af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] conn(0x7fb4200ad340 0x7fb4200af7e0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb4200a21e0 msgr2=0x7fb4200a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb4200a21e0 0x7fb4200a46a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb430106230 tx=0x7fb41c009260 comp rx=0 tx=0).stop 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 msgr2=0x7fb430199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 --2- 172.21.15.116:0/1153714850 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb430101100 0x7fb430199450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb424000c00 tx=0x7fb424004730 comp rx=0 tx=0).stop 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb4355b2700 1 -- 172.21.15.116:0/1153714850 reap_dead start 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 shutdown_connections 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 >> 172.21.15.116:0/1153714850 conn(0x7fb430078030 msgr2=0x7fb4300fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 shutdown_connections 2024-03-24T09:20:39.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.901+0000 7fb437015700 1 -- 172.21.15.116:0/1153714850 wait complete. 2024-03-24T09:20:40.033 INFO:teuthology.orchestra.run.smithi116.stdout:163208757304 2024-03-24T09:20:40.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T09:20:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/2243964840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 msgr2=0x7f8f7410f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/2243964840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 0x7f8f7410f120 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c00b0a0 tx=0x7f8f6c00b3b0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/2243964840 shutdown_connections 2024-03-24T09:20:40.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/2243964840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 0x7f8f7410f120 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/2243964840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f7406ac10 0x7f8f7406aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/2243964840 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406b5c0 0x7f8f74106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/2243964840 >> 172.21.15.116:0/2243964840 conn(0x7f8f740781d0 msgr2=0x7f8f740785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/2243964840 shutdown_connections 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/2243964840 wait complete. 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 Processor -- start 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.351+0000 7f8f7b2a5700 1 -- start start 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406ac10 0x7f8f74101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f7406b5c0 0x7f8f74101670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:40.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 0x7f8f74101bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a6b60 con 0x7f8f74107550 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a6cd0 con 0x7f8f7406b5c0 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a6e90 con 0x7f8f7406ac10 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406ac10 0x7f8f74101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79842700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 0x7f8f74101bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f78840700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f7406b5c0 0x7f8f74101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406ac10 0x7f8f74101130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41788/0 (socket says 172.21.15.116:41788) 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 -- 172.21.15.116:0/1991725933 learned_addr learned my addr 172.21.15.116:0/1991725933 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:40.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 -- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f7406b5c0 msgr2=0x7f8f74101670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f7406b5c0 0x7f8f74101670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 -- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 msgr2=0x7f8f74101bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 0x7f8f74101bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 -- 172.21.15.116:0/1991725933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6c009d00 con 0x7f8f7406ac10 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79842700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f74107550 0x7f8f74101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f79041700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406ac10 0x7f8f74101130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000d980 tx=0x7f8f7000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f70004220 con 0x7f8f7406ac10 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f70007640 con 0x7f8f7406ac10 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f700056c0 con 0x7f8f7406ac10 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.352+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f741a7170 con 0x7f8f7406ac10 2024-03-24T09:20:40.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.353+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f741a7670 con 0x7f8f7406ac10 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.353+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f74112440 con 0x7f8f7406ac10 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.354+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f8f70004750 con 0x7f8f7406ac10 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.355+0000 7f8f6a7fc700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8f600a21e0 0x7f8f600a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.355+0000 7f8f78840700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8f600a21e0 0x7f8f600a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.355+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8f700ce550 con 0x7f8f7406ac10 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.355+0000 7f8f6a7fc700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] conn(0x7f8f600ad340 0x7f8f600af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.355+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 --> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f600afeb0 con 0x7f8f600ad340 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.355+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f8f700ce960 con 0x7f8f7406ac10 2024-03-24T09:20:40.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.356+0000 7f8f78840700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8f600a21e0 0x7f8f600a46a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8f74108990 tx=0x7f8f64009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.356+0000 7f8f79842700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] conn(0x7f8f600ad340 0x7f8f600af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.356+0000 7f8f79842700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] conn(0x7f8f600ad340 0x7f8f600af7e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.357+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== osd.1 v2:172.21.15.117:6800/2635729600 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f600afeb0 con 0x7f8f600ad340 2024-03-24T09:20:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.373+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 --> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8f74062910 con 0x7f8f600ad340 2024-03-24T09:20:40.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.373+0000 7f8f6a7fc700 1 -- 172.21.15.116:0/1991725933 <== osd.1 v2:172.21.15.117:6800/2635729600 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8f74062910 con 0x7f8f600ad340 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.373+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] conn(0x7f8f600ad340 msgr2=0x7f8f600af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.373+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] conn(0x7f8f600ad340 0x7f8f600af7e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8f600a21e0 msgr2=0x7f8f600a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8f600a21e0 0x7f8f600a46a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8f74108990 tx=0x7f8f64009260 comp rx=0 tx=0).stop 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406ac10 msgr2=0x7f8f74101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 --2- 172.21.15.116:0/1991725933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f7406ac10 0x7f8f74101130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000d980 tx=0x7f8f7000dc90 comp rx=0 tx=0).stop 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f79842700 1 -- 172.21.15.116:0/1991725933 reap_dead start 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 shutdown_connections 2024-03-24T09:20:40.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 >> 172.21.15.116:0/1991725933 conn(0x7f8f740781d0 msgr2=0x7f8f740ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 shutdown_connections 2024-03-24T09:20:40.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.374+0000 7f8f7b2a5700 1 -- 172.21.15.116:0/1991725933 wait complete. 2024-03-24T09:20:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:40 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:40 smithi116 ceph-mon[32207]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 12 op/s 2024-03-24T09:20:40.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:40 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:40 smithi143 ceph-mon[35965]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 12 op/s 2024-03-24T09:20:40.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:40 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:20:40.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:40 smithi117 ceph-mon[35788]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 383 B/s wr, 12 op/s 2024-03-24T09:20:41.275 INFO:teuthology.orchestra.run.smithi116.stdout:55834574923 2024-03-24T09:20:41.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T09:20:41.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 -- 172.21.15.116:0/3522491596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0107140 msgr2=0x7f7cb0107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/3522491596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0107140 0x7f7cb0107520 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0009990 tx=0x7f7ca0009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 -- 172.21.15.116:0/3522491596 shutdown_connections 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/3522491596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cb0101aa0 0x7f7cb010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/3522491596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0107140 0x7f7cb0107520 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/3522491596 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb0101100 0x7f7cb0101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 -- 172.21.15.116:0/3522491596 >> 172.21.15.116:0/3522491596 conn(0x7f7cb0078030 msgr2=0x7f7cb0078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 -- 172.21.15.116:0/3522491596 shutdown_connections 2024-03-24T09:20:41.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.222+0000 7f7cb52c4700 1 -- 172.21.15.116:0/3522491596 wait complete. 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 Processor -- start 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 -- start start 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0101100 0x7f7cb0198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb0101aa0 0x7f7cb0198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cb0107140 0x7f7cb019cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb019d520 con 0x7f7cb0101100 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb019d690 con 0x7f7cb0107140 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cb52c4700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb019d850 con 0x7f7cb0101aa0 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7caeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0101100 0x7f7cb0198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:41.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7caeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0101100 0x7f7cb0198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59384/0 (socket says 172.21.15.116:59384) 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7caeffd700 1 -- 172.21.15.116:0/2175869683 learned_addr learned my addr 172.21.15.116:0/2175869683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.223+0000 7f7cae7fc700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb0101aa0 0x7f7cb0198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caf7fe700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cb0107140 0x7f7cb019cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caeffd700 1 -- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb0101aa0 msgr2=0x7f7cb0198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caeffd700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7cb0101aa0 0x7f7cb0198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caeffd700 1 -- 172.21.15.116:0/2175869683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cb0107140 msgr2=0x7f7cb019cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caeffd700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cb0107140 0x7f7cb019cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caeffd700 1 -- 172.21.15.116:0/2175869683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca00095f0 con 0x7f7cb0101100 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7caeffd700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0101100 0x7f7cb0198640 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0000c00 tx=0x7f7ca0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:41.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca001d070 con 0x7f7cb0101100 2024-03-24T09:20:41.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ca0022470 con 0x7f7cb0101100 2024-03-24T09:20:41.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb019dad0 con 0x7f7cb0101100 2024-03-24T09:20:41.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0017650 con 0x7f7cb0101100 2024-03-24T09:20:41.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.224+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb01a5390 con 0x7f7cb0101100 2024-03-24T09:20:41.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.225+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7cb019dda0 con 0x7f7cb0101100 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.226+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f7ca00225e0 con 0x7f7cb0101100 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7c97fff700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7c980a21e0 0x7f7c980a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7cae7fc700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7c980a21e0 0x7f7c980a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7ca00cfc50 con 0x7f7cb0101100 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7c97fff700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] conn(0x7f7c980ad340 0x7f7c980af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 --> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c980afeb0 con 0x7f7c980ad340 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7ca00d0060 con 0x7f7cb0101100 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7cae7fc700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7c980a21e0 0x7f7c980a46a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0078770 tx=0x7f7ca4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:41.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7caf7fe700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] conn(0x7f7c980ad340 0x7f7c980af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.227+0000 7f7caf7fe700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] conn(0x7f7c980ad340 0x7f7c980af7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.228+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== osd.8 v2:172.21.15.143:6816/3224917064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c980afeb0 con 0x7f7c980ad340 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.245+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 --> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7cb0112440 con 0x7f7c980ad340 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.245+0000 7f7c97fff700 1 -- 172.21.15.116:0/2175869683 <== osd.8 v2:172.21.15.143:6816/3224917064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7cb0112440 con 0x7f7c980ad340 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] conn(0x7f7c980ad340 msgr2=0x7f7c980af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] conn(0x7f7c980ad340 0x7f7c980af7e0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7c980a21e0 msgr2=0x7f7c980a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f7c980a21e0 0x7f7c980a46a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0078770 tx=0x7f7ca4009260 comp rx=0 tx=0).stop 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0101100 msgr2=0x7f7cb0198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 --2- 172.21.15.116:0/2175869683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cb0101100 0x7f7cb0198640 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0000c00 tx=0x7f7ca0004730 comp rx=0 tx=0).stop 2024-03-24T09:20:41.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7caf7fe700 1 -- 172.21.15.116:0/2175869683 reap_dead start 2024-03-24T09:20:41.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 shutdown_connections 2024-03-24T09:20:41.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 >> 172.21.15.116:0/2175869683 conn(0x7f7cb0078030 msgr2=0x7f7cb0100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:41.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 shutdown_connections 2024-03-24T09:20:41.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:40.246+0000 7f7cb52c4700 1 -- 172.21.15.116:0/2175869683 wait complete. 2024-03-24T09:20:42.031 INFO:teuthology.orchestra.run.smithi116.stdout:253403070506 2024-03-24T09:20:42.031 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-24T09:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 -- 172.21.15.116:0/2966903365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0101100 msgr2=0x7f19b0101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 --2- 172.21.15.116:0/2966903365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0101100 0x7f19b0101560 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1998009990 tx=0x7f1998009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 -- 172.21.15.116:0/2966903365 shutdown_connections 2024-03-24T09:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 --2- 172.21.15.116:0/2966903365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 0x7f19b010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 --2- 172.21.15.116:0/2966903365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b0107140 0x7f19b0107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 --2- 172.21.15.116:0/2966903365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0101100 0x7f19b0101560 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 -- 172.21.15.116:0/2966903365 >> 172.21.15.116:0/2966903365 conn(0x7f19b0078030 msgr2=0x7f19b0078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 -- 172.21.15.116:0/2966903365 shutdown_connections 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.765+0000 7f19b6739700 1 -- 172.21.15.116:0/2966903365 wait complete. 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.766+0000 7f19b6739700 1 Processor -- start 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.766+0000 7f19b6739700 1 -- start start 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.766+0000 7f19b6739700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b0101100 0x7f19b01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.766+0000 7f19b6739700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 0x7f19b0198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.766+0000 7f19b6739700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0107140 0x7f19b019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.766+0000 7f19b6739700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19b019d4c0 con 0x7f19b0107140 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b6739700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f19b019d630 con 0x7f19b0101aa0 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b6739700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f19b019d7f0 con 0x7f19b0101100 2024-03-24T09:20:42.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0107140 0x7f19b019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19af7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 0x7f19b0198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19af7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 0x7f19b0198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46042/0 (socket says 172.21.15.116:46042) 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0107140 0x7f19b019ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59308/0 (socket says 172.21.15.116:59308) 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19af7fe700 1 -- 172.21.15.116:0/3673374059 learned_addr learned my addr 172.21.15.116:0/3673374059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19affff700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b0101100 0x7f19b01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 -- 172.21.15.116:0/3673374059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b0101100 msgr2=0x7f19b01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19b0101100 0x7f19b01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 -- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 msgr2=0x7f19b0198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 0x7f19b0198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 -- 172.21.15.116:0/3673374059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19980095f0 con 0x7f19b0107140 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19af7fe700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19b0101aa0 0x7f19b0198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:42.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b4cd6700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0107140 0x7f19b019ceb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f19a800efc0 tx=0x7f19a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8018070 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b019da70 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f19a800f040 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a800cc40 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.767+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b01a5330 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.768+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f19b01027f0 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.769+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f19a8007500 con 0x7f19b0107140 2024-03-24T09:20:42.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.770+0000 7f19ad7fa700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f199c0a21e0 0x7f199c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.770+0000 7f19affff700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f199c0a21e0 0x7f199c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.770+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f19a80ce9a0 con 0x7f19b0107140 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.770+0000 7f19ad7fa700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] conn(0x7f199c0ad340 0x7f199c0af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.770+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 --> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f199c0afeb0 con 0x7f199c0ad340 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.770+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f19a80cedb0 con 0x7f19b0107140 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.771+0000 7f19af7fe700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] conn(0x7f199c0ad340 0x7f199c0af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.771+0000 7f19affff700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f199c0a21e0 0x7f199c0a46a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009be0 tx=0x7f19a0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.771+0000 7f19af7fe700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] conn(0x7f199c0ad340 0x7f199c0af7e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.772+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== osd.10 v2:172.21.15.117:6824/3860527713 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f199c0afeb0 con 0x7f199c0ad340 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.788+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 --> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f19b01112e0 con 0x7f199c0ad340 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.788+0000 7f19ad7fa700 1 -- 172.21.15.116:0/3673374059 <== osd.10 v2:172.21.15.117:6824/3860527713 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f19b01112e0 con 0x7f199c0ad340 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.788+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] conn(0x7f199c0ad340 msgr2=0x7f199c0af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.788+0000 7f19b6739700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] conn(0x7f199c0ad340 0x7f199c0af7e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f199c0a21e0 msgr2=0x7f199c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f199c0a21e0 0x7f199c0a46a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f19a0009be0 tx=0x7f19a0009260 comp rx=0 tx=0).stop 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0107140 msgr2=0x7f19b019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 --2- 172.21.15.116:0/3673374059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b0107140 0x7f19b019ceb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f19a800efc0 tx=0x7f19a800c3a0 comp rx=0 tx=0).stop 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b4cd6700 1 -- 172.21.15.116:0/3673374059 reap_dead start 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 shutdown_connections 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 >> 172.21.15.116:0/3673374059 conn(0x7f19b0078030 msgr2=0x7f19b006a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 shutdown_connections 2024-03-24T09:20:42.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:37.789+0000 7f19b6739700 1 -- 172.21.15.116:0/3673374059 wait complete. 2024-03-24T09:20:42.792 INFO:teuthology.orchestra.run.smithi116.stdout:309237645342 2024-03-24T09:20:42.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-24T09:20:43.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:42 smithi116 ceph-mon[32207]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T09:20:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:42 smithi143 ceph-mon[35965]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T09:20:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:42 smithi117 ceph-mon[35788]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 383 B/s wr, 10 op/s 2024-03-24T09:20:43.669 INFO:teuthology.orchestra.run.smithi116.stdout:197568495667 2024-03-24T09:20:43.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T09:20:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 -- 172.21.15.116:0/1313223215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 msgr2=0x7f1724101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 --2- 172.21.15.116:0/1313223215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 0x7f1724101770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1718009990 tx=0x7f1718009ca0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 -- 172.21.15.116:0/1313223215 shutdown_connections 2024-03-24T09:20:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 --2- 172.21.15.116:0/1313223215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241063e0 0x7f172410dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 --2- 172.21.15.116:0/1313223215 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 0x7f1724101770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 --2- 172.21.15.116:0/1313223215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 0x7f1724105d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.890+0000 7f172abed700 1 -- 172.21.15.116:0/1313223215 >> 172.21.15.116:0/1313223215 conn(0x7f1724077150 msgr2=0x7f1724077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.891+0000 7f172abed700 1 -- 172.21.15.116:0/1313223215 shutdown_connections 2024-03-24T09:20:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.891+0000 7f172abed700 1 -- 172.21.15.116:0/1313223215 wait complete. 2024-03-24T09:20:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.891+0000 7f172abed700 1 Processor -- start 2024-03-24T09:20:43.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 -- start start 2024-03-24T09:20:43.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 0x7f17241994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 0x7f17241999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241063e0 0x7f17241a4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f1723fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 0x7f17241999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f1723fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 0x7f17241999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46134/0 (socket says 172.21.15.116:46134) 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f1723fff700 1 -- 172.21.15.116:0/2041055352 learned_addr learned my addr 172.21.15.116:0/2041055352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f172419a2a0 con 0x7f17241063e0 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f17241a4a60 con 0x7f1724101d40 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.892+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f17241a4ba0 con 0x7f1724101390 2024-03-24T09:20:43.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1728989700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 0x7f17241994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1723fff700 1 -- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 msgr2=0x7f17241994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1723fff700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 0x7f17241994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1723fff700 1 -- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241063e0 msgr2=0x7f17241a4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1723fff700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17241063e0 0x7f17241a4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1723fff700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1710009520 con 0x7f1724101d40 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1728989700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1724101390 0x7f17241994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1723fff700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 0x7f17241999e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f171000ea30 tx=0x7f171000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710009820 con 0x7f1724101d40 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.893+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f17100044d0 con 0x7f1724101d40 2024-03-24T09:20:43.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.894+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1710003680 con 0x7f1724101d40 2024-03-24T09:20:43.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.894+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17180095f0 con 0x7f1724101d40 2024-03-24T09:20:43.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.894+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1724197660 con 0x7f1724101d40 2024-03-24T09:20:43.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.895+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f17241112b0 con 0x7f1724101d40 2024-03-24T09:20:43.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.896+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f1710024450 con 0x7f1724101d40 2024-03-24T09:20:43.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f1721ffb700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17140a21e0 0x7f17140a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f17100d2b40 con 0x7f1724101d40 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f1728989700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17140a21e0 0x7f17140a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f1721ffb700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] conn(0x7f17140ad340 0x7f17140af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17140afeb0 con 0x7f17140ad340 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f17100d2f50 con 0x7f1724101d40 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.897+0000 7f172918a700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] conn(0x7f17140ad340 0x7f17140af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.898+0000 7f172918a700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] conn(0x7f17140ad340 0x7f17140af7e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.898+0000 7f1728989700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17140a21e0 0x7f17140a46a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1718000c00 tx=0x7f1718005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:43.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.899+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== osd.2 v2:172.21.15.143:6800/541711081 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f17140afeb0 con 0x7f17140ad340 2024-03-24T09:20:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.916+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 --> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1724062910 con 0x7f17140ad340 2024-03-24T09:20:43.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.916+0000 7f1721ffb700 1 -- 172.21.15.116:0/2041055352 <== osd.2 v2:172.21.15.143:6800/541711081 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1724062910 con 0x7f17140ad340 2024-03-24T09:20:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] conn(0x7f17140ad340 msgr2=0x7f17140af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] conn(0x7f17140ad340 0x7f17140af7e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17140a21e0 msgr2=0x7f17140a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f17140a21e0 0x7f17140a46a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1718000c00 tx=0x7f1718005fb0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 msgr2=0x7f17241999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 --2- 172.21.15.116:0/2041055352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1724101d40 0x7f17241999e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f171000ea30 tx=0x7f171000edf0 comp rx=0 tx=0).stop 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172918a700 1 -- 172.21.15.116:0/2041055352 reap_dead start 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 shutdown_connections 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 >> 172.21.15.116:0/2041055352 conn(0x7f1724077150 msgr2=0x7f1724105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 shutdown_connections 2024-03-24T09:20:43.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:43.917+0000 7f172abed700 1 -- 172.21.15.116:0/2041055352 wait complete. 2024-03-24T09:20:44.326 INFO:teuthology.orchestra.run.smithi116.stdout:219043332141 2024-03-24T09:20:44.326 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T09:20:44.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 -- 172.21.15.116:0/1482544573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc06b310 msgr2=0x7fa6cc10f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 --2- 172.21.15.116:0/1482544573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc06b310 0x7fa6cc10f050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4009fc0 tx=0x7fa6c400b330 comp rx=0 tx=0).stop 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 -- 172.21.15.116:0/1482544573 shutdown_connections 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 --2- 172.21.15.116:0/1482544573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc06b310 0x7fa6cc10f050 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 --2- 172.21.15.116:0/1482544573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6cc108230 0x7fa6cc108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 --2- 172.21.15.116:0/1482544573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 0x7fa6cc06add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 -- 172.21.15.116:0/1482544573 >> 172.21.15.116:0/1482544573 conn(0x7fa6cc078110 msgr2=0x7fa6cc078520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.774+0000 7fa6d479f700 1 -- 172.21.15.116:0/1482544573 shutdown_connections 2024-03-24T09:20:44.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 -- 172.21.15.116:0/1482544573 wait complete. 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 Processor -- start 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 -- start start 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 0x7fa6cc1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc06b310 0x7fa6cc1016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6cc108230 0x7fa6cc101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6cc1a6ad0 con 0x7fa6cc06b310 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa6cc1a6c40 con 0x7fa6cc06a950 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.775+0000 7fa6d479f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa6cc1a6e00 con 0x7fa6cc108230 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 0x7fa6cc1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:44.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 0x7fa6cc1011b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46154/0 (socket says 172.21.15.116:46154) 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 -- 172.21.15.116:0/2039497569 learned_addr learned my addr 172.21.15.116:0/2039497569 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d2d3c700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6cc108230 0x7fa6cc101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 -- 172.21.15.116:0/2039497569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6cc108230 msgr2=0x7fa6cc101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6cc108230 0x7fa6cc101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 -- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc06b310 msgr2=0x7fa6cc1016f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6cc06b310 0x7fa6cc1016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 -- 172.21.15.116:0/2039497569 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c4009c40 con 0x7fa6cc06a950 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d2d3c700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa6cc108230 0x7fa6cc101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:20:44.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6d253b700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 0x7fa6cc1011b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800e9a0 tx=0x7fa6c800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.776+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c800cbd0 con 0x7fa6cc06a950 2024-03-24T09:20:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.777+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6cc1a70e0 con 0x7fa6cc06a950 2024-03-24T09:20:44.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.777+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6cc1a7630 con 0x7fa6cc06a950 2024-03-24T09:20:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.777+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa6c800cd30 con 0x7fa6cc06a950 2024-03-24T09:20:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.777+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8018680 con 0x7fa6cc06a950 2024-03-24T09:20:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.777+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6b0001030 con 0x7fa6cc06a950 2024-03-24T09:20:44.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.780+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fa6c80187e0 con 0x7fa6cc06a950 2024-03-24T09:20:44.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.780+0000 7fa6bf7fe700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa6b80a2240 0x7fa6b80a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:44.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa6c8014070 con 0x7fa6cc06a950 2024-03-24T09:20:44.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6d1d3a700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa6b80a2240 0x7fa6b80a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:44.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6bf7fe700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] conn(0x7fa6b80ad3a0 0x7fa6b80af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:44.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 --> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6b80aff10 con 0x7fa6b80ad3a0 2024-03-24T09:20:44.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa6c8018680 con 0x7fa6cc06a950 2024-03-24T09:20:44.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6d2d3c700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] conn(0x7fa6b80ad3a0 0x7fa6b80af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:44.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.781+0000 7fa6d1d3a700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa6b80a2240 0x7fa6b80a4700 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0005950 tx=0x7fa6c00058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:44.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.782+0000 7fa6d2d3c700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] conn(0x7fa6b80ad3a0 0x7fa6b80af840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:44.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.783+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== osd.4 v2:172.21.15.117:6808/3606274523 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa6b80aff10 con 0x7fa6b80ad3a0 2024-03-24T09:20:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.804+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 --> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6b0002dc0 con 0x7fa6b80ad3a0 2024-03-24T09:20:44.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6bf7fe700 1 -- 172.21.15.116:0/2039497569 <== osd.4 v2:172.21.15.117:6808/3606274523 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa6b0002dc0 con 0x7fa6b80ad3a0 2024-03-24T09:20:44.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] conn(0x7fa6b80ad3a0 msgr2=0x7fa6b80af840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] conn(0x7fa6b80ad3a0 0x7fa6b80af840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa6b80a2240 msgr2=0x7fa6b80a4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa6b80a2240 0x7fa6b80a4700 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0005950 tx=0x7fa6c00058e0 comp rx=0 tx=0).stop 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 msgr2=0x7fa6cc1011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 --2- 172.21.15.116:0/2039497569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6cc06a950 0x7fa6cc1011b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800e9a0 tx=0x7fa6c800ed60 comp rx=0 tx=0).stop 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d2d3c700 1 -- 172.21.15.116:0/2039497569 reap_dead start 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 shutdown_connections 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 >> 172.21.15.116:0/2039497569 conn(0x7fa6cc078110 msgr2=0x7fa6cc0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 shutdown_connections 2024-03-24T09:20:44.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:44.805+0000 7fa6d479f700 1 -- 172.21.15.116:0/2039497569 wait complete. 2024-03-24T09:20:45.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:44 smithi143 ceph-mon[35965]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 309 B/s wr, 9 op/s 2024-03-24T09:20:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:44 smithi117 ceph-mon[35788]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 309 B/s wr, 9 op/s 2024-03-24T09:20:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:44 smithi116 ceph-mon[32207]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 309 B/s wr, 9 op/s 2024-03-24T09:20:45.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.984+0000 7f3662491700 1 -- 172.21.15.116:0/134022743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c06a850 msgr2=0x7f365c10f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:45.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.984+0000 7f3662491700 1 --2- 172.21.15.116:0/134022743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c06a850 0x7f365c10f090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f365400b0d0 tx=0x7f365400b3e0 comp rx=0 tx=0).stop 2024-03-24T09:20:45.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 -- 172.21.15.116:0/134022743 shutdown_connections 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 --2- 172.21.15.116:0/134022743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c06a850 0x7f365c10f090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 --2- 172.21.15.116:0/134022743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c108250 0x7f365c108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 --2- 172.21.15.116:0/134022743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f365c069eb0 0x7f365c06a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 -- 172.21.15.116:0/134022743 >> 172.21.15.116:0/134022743 conn(0x7f365c078030 msgr2=0x7f365c078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 -- 172.21.15.116:0/134022743 shutdown_connections 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 -- 172.21.15.116:0/134022743 wait complete. 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 Processor -- start 2024-03-24T09:20:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.985+0000 7f3662491700 1 -- start start 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3662491700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c069eb0 0x7f365c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3662491700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c06a850 0x7f365c198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3662491700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f365c108250 0x7f365c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3662491700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f365c19d5f0 con 0x7f365c069eb0 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3662491700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f365c19d760 con 0x7f365c108250 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3662491700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f365c19d920 con 0x7f365c06a850 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c069eb0 0x7f365c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c069eb0 0x7f365c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59458/0 (socket says 172.21.15.116:59458) 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 -- 172.21.15.116:0/2633201522 learned_addr learned my addr 172.21.15.116:0/2633201522 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c06a850 0x7f365c198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3660a2e700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f365c108250 0x7f365c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 -- 172.21.15.116:0/2633201522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c06a850 msgr2=0x7f365c198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f365c06a850 0x7f365c198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 -- 172.21.15.116:0/2633201522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f365c108250 msgr2=0x7f365c19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f365c108250 0x7f365c19cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 -- 172.21.15.116:0/2633201522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3654009d00 con 0x7f365c069eb0 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f3660a2e700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f365c108250 0x7f365c19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f365bfff700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c069eb0 0x7f365c198760 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f364c00e9a0 tx=0x7f364c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.986+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c009790 con 0x7f365c069eb0 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.987+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f364c010430 con 0x7f365c069eb0 2024-03-24T09:20:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.987+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f365c19dc00 con 0x7f365c069eb0 2024-03-24T09:20:45.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.987+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f364c018680 con 0x7f365c069eb0 2024-03-24T09:20:45.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.987+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f365c1a5460 con 0x7f365c069eb0 2024-03-24T09:20:45.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.987+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f365c1123e0 con 0x7f365c069eb0 2024-03-24T09:20:45.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.989+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f364c0105a0 con 0x7f365c069eb0 2024-03-24T09:20:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.989+0000 7f36597fa700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f36440a21e0 0x7f36440a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.989+0000 7f365b7fe700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f36440a21e0 0x7f36440a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f364c014070 con 0x7f365c069eb0 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f36597fa700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] conn(0x7f36440ad340 0x7f36440af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 --> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36440afeb0 con 0x7f36440ad340 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f364c0ce9e0 con 0x7f365c069eb0 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f3660a2e700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] conn(0x7f36440ad340 0x7f36440af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f365b7fe700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f36440a21e0 0x7f36440a46a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f365c0786a0 tx=0x7f3650009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.990+0000 7f3660a2e700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] conn(0x7f36440ad340 0x7f36440af7e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:45.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:45.991+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== osd.0 v2:172.21.15.116:6802/2942581569 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36440afeb0 con 0x7f36440ad340 2024-03-24T09:20:46.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.010+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 --> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f365c062910 con 0x7f36440ad340 2024-03-24T09:20:46.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f36597fa700 1 -- 172.21.15.116:0/2633201522 <== osd.0 v2:172.21.15.116:6802/2942581569 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f365c062910 con 0x7f36440ad340 2024-03-24T09:20:46.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] conn(0x7f36440ad340 msgr2=0x7f36440af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] conn(0x7f36440ad340 0x7f36440af7e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f36440a21e0 msgr2=0x7f36440a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f36440a21e0 0x7f36440a46a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f365c0786a0 tx=0x7f3650009260 comp rx=0 tx=0).stop 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c069eb0 msgr2=0x7f365c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 --2- 172.21.15.116:0/2633201522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f365c069eb0 0x7f365c198760 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f364c00e9a0 tx=0x7f364c00ecb0 comp rx=0 tx=0).stop 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3660a2e700 1 -- 172.21.15.116:0/2633201522 reap_dead start 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 shutdown_connections 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 >> 172.21.15.116:0/2633201522 conn(0x7f365c078030 msgr2=0x7f365c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:46.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 shutdown_connections 2024-03-24T09:20:46.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:46.011+0000 7f3662491700 1 -- 172.21.15.116:0/2633201522 wait complete. 2024-03-24T09:20:46.471 INFO:teuthology.orchestra.run.smithi116.stdout:141733920831 2024-03-24T09:20:46.471 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T09:20:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:46 smithi143 ceph-mon[35965]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 281 B/s wr, 9 op/s 2024-03-24T09:20:47.280 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:46 smithi116 ceph-mon[32207]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 281 B/s wr, 9 op/s 2024-03-24T09:20:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:46 smithi117 ceph-mon[35788]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 281 B/s wr, 9 op/s 2024-03-24T09:20:47.329 INFO:teuthology.orchestra.run.smithi116.stdout:77309411400 2024-03-24T09:20:47.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T09:20:47.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 -- 172.21.15.116:0/891159653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e8100c00 msgr2=0x7f75e8101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 --2- 172.21.15.116:0/891159653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e8100c00 0x7f75e8101080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0099c0 tx=0x7f75dc009cd0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 -- 172.21.15.116:0/891159653 shutdown_connections 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 --2- 172.21.15.116:0/891159653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e81015c0 0x7f75e810f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 --2- 172.21.15.116:0/891159653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75e81072d0 0x7f75e81076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 --2- 172.21.15.116:0/891159653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e8100c00 0x7f75e8101080 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 -- 172.21.15.116:0/891159653 >> 172.21.15.116:0/891159653 conn(0x7f75e8076450 msgr2=0x7f75e8076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 -- 172.21.15.116:0/891159653 shutdown_connections 2024-03-24T09:20:47.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.852+0000 7f75ee340700 1 -- 172.21.15.116:0/891159653 wait complete. 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 Processor -- start 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 -- start start 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75e8100c00 0x7f75e8101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e81015c0 0x7f75e8102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e81072d0 0x7f75e8102860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75e81a6a30 con 0x7f75e81015c0 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f75e81a6ba0 con 0x7f75e8100c00 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75ee340700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f75e81a6d60 con 0x7f75e81072d0 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.853+0000 7f75e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e81015c0 0x7f75e8102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e81015c0 0x7f75e8102320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56166/0 (socket says 172.21.15.116:56166) 2024-03-24T09:20:47.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 -- 172.21.15.116:0/1952942724 learned_addr learned my addr 172.21.15.116:0/1952942724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e7fff700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75e8100c00 0x7f75e8101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75ec8dd700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e81072d0 0x7f75e8102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 -- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e81072d0 msgr2=0x7f75e8102860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e81072d0 0x7f75e8102860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 -- 172.21.15.116:0/1952942724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75e8100c00 msgr2=0x7f75e8101de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75e8100c00 0x7f75e8101de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 -- 172.21.15.116:0/1952942724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc0095f0 con 0x7f75e81015c0 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75ec8dd700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f75e81072d0 0x7f75e8102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e77fe700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e81015c0 0x7f75e8102320 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f75dc005910 tx=0x7f75dc0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:47.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc01d070 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e81a6fe0 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f75dc003e30 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc00f9f0 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.854+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e81a73a0 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.855+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f75e8112430 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.856+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f75dc00b730 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e57fa700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f75d00a21e0 0x7f75d00a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e7fff700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f75d00a21e0 0x7f75d00a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f75dc0cf2a0 con 0x7f75e81015c0 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e57fa700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] conn(0x7f75d00ad340 0x7f75d00af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:20:47.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 --> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75d00afeb0 con 0x7f75d00ad340 2024-03-24T09:20:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f75dc0cf6b0 con 0x7f75e81015c0 2024-03-24T09:20:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75e7fff700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f75d00a21e0 0x7f75d00a46a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009be0 tx=0x7f75d8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.857+0000 7f75ec8dd700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] conn(0x7f75d00ad340 0x7f75d00af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:20:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.858+0000 7f75ec8dd700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] conn(0x7f75d00ad340 0x7f75d00af7e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:20:47.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.858+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== osd.11 v2:172.21.15.143:6824/2884681125 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75d00afeb0 con 0x7f75d00ad340 2024-03-24T09:20:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 --> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f75e8062940 con 0x7f75d00ad340 2024-03-24T09:20:47.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75e57fa700 1 -- 172.21.15.116:0/1952942724 <== osd.11 v2:172.21.15.143:6824/2884681125 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f75e8062940 con 0x7f75d00ad340 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] conn(0x7f75d00ad340 msgr2=0x7f75d00af7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] conn(0x7f75d00ad340 0x7f75d00af7e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f75d00a21e0 msgr2=0x7f75d00a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f75d00a21e0 0x7f75d00a46a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009be0 tx=0x7f75d8009260 comp rx=0 tx=0).stop 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e81015c0 msgr2=0x7f75e8102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ee340700 1 --2- 172.21.15.116:0/1952942724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75e81015c0 0x7f75e8102320 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f75dc005910 tx=0x7f75dc0039f0 comp rx=0 tx=0).stop 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.877+0000 7f75ec8dd700 1 -- 172.21.15.116:0/1952942724 reap_dead start 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.878+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 shutdown_connections 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.878+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 >> 172.21.15.116:0/1952942724 conn(0x7f75e8076450 msgr2=0x7f75e80fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:20:47.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.878+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 shutdown_connections 2024-03-24T09:20:47.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:20:47.878+0000 7f75ee340700 1 -- 172.21.15.116:0/1952942724 wait complete. 2024-03-24T09:20:48.614 INFO:teuthology.orchestra.run.smithi116.stdout:343597383707 2024-03-24T09:20:48.614 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-24T09:20:49.145 INFO:teuthology.orchestra.run.smithi116.stdout:34359738450 2024-03-24T09:20:49.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T09:20:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.198 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.199 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.205 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.208 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.220 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.223 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:48 smithi143 ceph-mon[35965]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T09:20:49.277 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:48 smithi117 ceph-mon[35788]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T09:20:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:48 smithi116 ceph-mon[32207]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T09:20:50.251 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:50.252 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:50.252 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:20:51.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:50 smithi143 ceph-mon[35965]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-24T09:20:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:50 smithi117 ceph-mon[35788]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-24T09:20:51.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:50 smithi116 ceph-mon[32207]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-24T09:20:53.214 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:52 smithi116 ceph-mon[32207]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T09:20:53.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:52 smithi143 ceph-mon[35965]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T09:20:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:52 smithi117 ceph-mon[35788]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T09:20:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:55 smithi143 ceph-mon[35965]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:20:55.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:55 smithi116 ceph-mon[32207]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:20:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:55 smithi117 ceph-mon[35788]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:20:57.223 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:57 smithi116 ceph-mon[32207]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-24T09:20:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:57 smithi143 ceph-mon[35965]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-24T09:20:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:57 smithi117 ceph-mon[35788]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-24T09:20:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:20:59 smithi143 ceph-mon[35965]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:20:59.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:20:59 smithi116 ceph-mon[32207]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:20:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:20:59 smithi117 ceph-mon[35788]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/398863400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74107140 msgr2=0x7f4d74107520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/398863400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74107140 0x7f4d74107520 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4d68009990 tx=0x7f4d68009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/398863400 shutdown_connections 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/398863400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 0x7f4d7410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/398863400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74107140 0x7f4d74107520 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/398863400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d74101100 0x7f4d74101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/398863400 >> 172.21.15.116:0/398863400 conn(0x7f4d74078030 msgr2=0x7f4d74078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/398863400 shutdown_connections 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/398863400 wait complete. 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 Processor -- start 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.483+0000 7f4d7a4f3700 1 -- start start 2024-03-24T09:21:00.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d7a4f3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 0x7f4d7410e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d7a4f3700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 0x7f4d7410ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d7a4f3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d74107140 0x7f4d7410f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d7a4f3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d741a6a30 con 0x7f4d74101100 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d7a4f3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d741a6ba0 con 0x7f4d74107140 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d7a4f3700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4d741a6d60 con 0x7f4d74101aa0 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 0x7f4d7410e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 0x7f4d7410e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44124/0 (socket says 172.21.15.116:44124) 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 -- 172.21.15.116:0/2342431796 learned_addr learned my addr 172.21.15.116:0/2342431796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d737fe700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 0x7f4d7410ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d78a90700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d74107140 0x7f4d7410f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:00.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 -- 172.21.15.116:0/2342431796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 msgr2=0x7f4d7410ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 0x7f4d7410ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 -- 172.21.15.116:0/2342431796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d74107140 msgr2=0x7f4d7410f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d74107140 0x7f4d7410f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 -- 172.21.15.116:0/2342431796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d680095f0 con 0x7f4d74101100 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d737fe700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 0x7f4d7410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.484+0000 7f4d73fff700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 0x7f4d7410e810 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4d68000c00 tx=0x7f4d68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.485+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6801d070 con 0x7f4d74101100 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.485+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4d68022470 con 0x7f4d74101100 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.485+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d68017650 con 0x7f4d74101100 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.485+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d741a6fe0 con 0x7f4d74101100 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.485+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d741a7390 con 0x7f4d74101100 2024-03-24T09:21:00.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.486+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d74109520 con 0x7f4d74101100 2024-03-24T09:21:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.487+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f4d680225e0 con 0x7f4d74101100 2024-03-24T09:21:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.487+0000 7f4d717fa700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d5c0a21e0 0x7f4d5c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:00.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.487+0000 7f4d737fe700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d5c0a21e0 0x7f4d5c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.487+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4d680cf890 con 0x7f4d74101100 2024-03-24T09:21:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.488+0000 7f4d737fe700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d5c0a21e0 0x7f4d5c0a46a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4d74078900 tx=0x7f4d64009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:00.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.491+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4d68098460 con 0x7f4d74101100 2024-03-24T09:21:00.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.691+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f4d741121c0 con 0x7f4d74101100 2024-03-24T09:21:00.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.692+0000 7f4d717fa700 1 -- 172.21.15.116:0/2342431796 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4d6809bb20 con 0x7f4d74101100 2024-03-24T09:21:00.693 INFO:teuthology.orchestra.run.smithi116.stdout:287762808872 2024-03-24T09:21:00.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d5c0a21e0 msgr2=0x7f4d5c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d5c0a21e0 0x7f4d5c0a46a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4d74078900 tx=0x7f4d64009260 comp rx=0 tx=0).stop 2024-03-24T09:21:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 msgr2=0x7f4d7410e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 0x7f4d7410e810 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4d68000c00 tx=0x7f4d68004730 comp rx=0 tx=0).stop 2024-03-24T09:21:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 shutdown_connections 2024-03-24T09:21:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d74107140 0x7f4d7410f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4d5c0a21e0 0x7f4d5c0a46a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d74101100 0x7f4d7410e810 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 --2- 172.21.15.116:0/2342431796 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d74101aa0 0x7f4d7410ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 >> 172.21.15.116:0/2342431796 conn(0x7f4d74078030 msgr2=0x7f4d74100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 shutdown_connections 2024-03-24T09:21:00.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:00.695+0000 7f4d7a4f3700 1 -- 172.21.15.116:0/2342431796 wait complete. 2024-03-24T09:21:01.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:01 smithi116 ceph-mon[32207]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:01.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:01 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2342431796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T09:21:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:01 smithi143 ceph-mon[35965]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:01 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2342431796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T09:21:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:01 smithi117 ceph-mon[35788]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:01 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2342431796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T09:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 -- 172.21.15.116:0/5987664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 msgr2=0x7fe9441026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 --2- 172.21.15.116:0/5987664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe9441026d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe938009990 tx=0x7fe938009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 -- 172.21.15.116:0/5987664 shutdown_connections 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 --2- 172.21.15.116:0/5987664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944102c10 0x7fe94410f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 --2- 172.21.15.116:0/5987664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe944108300 0x7fe9441086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 --2- 172.21.15.116:0/5987664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe9441026d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 -- 172.21.15.116:0/5987664 >> 172.21.15.116:0/5987664 conn(0x7fe944079330 msgr2=0x7fe944079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 -- 172.21.15.116:0/5987664 shutdown_connections 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.783+0000 7fe94b564700 1 -- 172.21.15.116:0/5987664 wait complete. 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 Processor -- start 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 -- start start 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe944074500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe944102c10 0x7fe944074a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944108300 0x7fe944074f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:01.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9441a6a10 con 0x7fe944102250 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9441a6b80 con 0x7fe944102c10 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe94b564700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe9441a6d40 con 0x7fe944108300 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe944074500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe944074500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44140/0 (socket says 172.21.15.116:44140) 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 -- 172.21.15.116:0/16941487 learned_addr learned my addr 172.21.15.116:0/16941487 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe948aff700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe944102c10 0x7fe944074a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949b01700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944108300 0x7fe944074f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 -- 172.21.15.116:0/16941487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944108300 msgr2=0x7fe944074f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944108300 0x7fe944074f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 -- 172.21.15.116:0/16941487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe944102c10 msgr2=0x7fe944074a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe944102c10 0x7fe944074a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 -- 172.21.15.116:0/16941487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9380095f0 con 0x7fe944102250 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949b01700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944108300 0x7fe944074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.784+0000 7fe949300700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe944074500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00e9a0 tx=0x7fe93c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.785+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c009790 con 0x7fe944102250 2024-03-24T09:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.785+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe93c00cbd0 con 0x7fe944102250 2024-03-24T09:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.785+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9441a6fc0 con 0x7fe944102250 2024-03-24T09:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.785+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93c018680 con 0x7fe944102250 2024-03-24T09:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.785+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9441a73e0 con 0x7fe944102250 2024-03-24T09:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.786+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe94410a650 con 0x7fe944102250 2024-03-24T09:21:01.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.787+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fe93c00cd40 con 0x7fe944102250 2024-03-24T09:21:01.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.787+0000 7fe9427fc700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe92c0a21e0 0x7fe92c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:01.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.788+0000 7fe948aff700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe92c0a21e0 0x7fe92c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.788+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe93c014070 con 0x7fe944102250 2024-03-24T09:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.788+0000 7fe948aff700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe92c0a21e0 0x7fe92c0a46a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe93800b2e0 tx=0x7fe938005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:01.793+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe93c053280 con 0x7fe944102250 2024-03-24T09:21:02.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.000+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe944112140 con 0x7fe944102250 2024-03-24T09:21:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.000+0000 7fe9427fc700 1 -- 172.21.15.116:0/16941487 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe93c096e90 con 0x7fe944102250 2024-03-24T09:21:02.001 INFO:teuthology.orchestra.run.smithi116.stdout:219043332146 2024-03-24T09:21:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe92c0a21e0 msgr2=0x7fe92c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe92c0a21e0 0x7fe92c0a46a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe93800b2e0 tx=0x7fe938005960 comp rx=0 tx=0).stop 2024-03-24T09:21:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 msgr2=0x7fe944074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe944074500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00e9a0 tx=0x7fe93c00ecb0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 shutdown_connections 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe944108300 0x7fe944074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe92c0a21e0 0x7fe92c0a46a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe944102250 0x7fe944074500 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 --2- 172.21.15.116:0/16941487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe944102c10 0x7fe944074a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 >> 172.21.15.116:0/16941487 conn(0x7fe944079330 msgr2=0x7fe944101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 shutdown_connections 2024-03-24T09:21:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.002+0000 7fe94b564700 1 -- 172.21.15.116:0/16941487 wait complete. 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 -- 172.21.15.116:0/2013953327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30101aa0 msgr2=0x7fcc3010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2013953327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30101aa0 0x7fcc3010df80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800b0a0 tx=0x7fcc2800b3b0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 -- 172.21.15.116:0/2013953327 shutdown_connections 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2013953327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30101aa0 0x7fcc3010df80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2013953327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30107140 0x7fcc30107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2013953327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101100 0x7fcc30101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 -- 172.21.15.116:0/2013953327 >> 172.21.15.116:0/2013953327 conn(0x7fcc30078030 msgr2=0x7fcc30078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 -- 172.21.15.116:0/2013953327 shutdown_connections 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.022+0000 7fcc3482e700 1 -- 172.21.15.116:0/2013953327 wait complete. 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 Processor -- start 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 -- start start 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30101100 0x7fcc3019e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 0x7fcc3019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 0x7fcc301985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcc30198c00 con 0x7fcc30107140 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcc30198d40 con 0x7fcc30101aa0 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc3482e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcc30198f00 con 0x7fcc30101100 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2dd9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 0x7fcc3019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 0x7fcc301985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30101100 0x7fcc3019e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2dd9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 0x7fcc3019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44986/0 (socket says 172.21.15.116:44986) 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2e59c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30101100 0x7fcc3019e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:53644/0 (socket says 172.21.15.116:53644) 2024-03-24T09:21:02.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2ed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 0x7fcc301985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44154/0 (socket says 172.21.15.116:44154) 2024-03-24T09:21:02.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.023+0000 7fcc2dd9b700 1 -- 172.21.15.116:0/2099253079 learned_addr learned my addr 172.21.15.116:0/2099253079 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2dd9b700 1 -- 172.21.15.116:0/2099253079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30101100 msgr2=0x7fcc3019e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2dd9b700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30101100 0x7fcc3019e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2dd9b700 1 -- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 msgr2=0x7fcc301985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2dd9b700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 0x7fcc301985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2dd9b700 1 -- 172.21.15.116:0/2099253079 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc28009d00 con 0x7fcc30101aa0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2ed9d700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 0x7fcc301985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc2dd9b700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 0x7fcc3019e920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e9a0 tx=0x7fcc2400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24009790 con 0x7fcc30101aa0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcc24010430 con 0x7fcc30101aa0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc301991e0 con 0x7fcc30101aa0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24018680 con 0x7fcc30101aa0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.024+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc301a5480 con 0x7fcc30101aa0 2024-03-24T09:21:02.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.025+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc30068ec0 con 0x7fcc30101aa0 2024-03-24T09:21:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.026+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fcc2400ca10 con 0x7fcc30101aa0 2024-03-24T09:21:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.027+0000 7fcc1f7fe700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fcc180a21e0 0x7fcc180a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.027+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcc24014070 con 0x7fcc30101aa0 2024-03-24T09:21:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.027+0000 7fcc2e59c700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fcc180a21e0 0x7fcc180a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.028+0000 7fcc2e59c700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fcc180a21e0 0x7fcc180a46a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009be0 tx=0x7fcc20009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:02.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.032+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcc24053280 con 0x7fcc30101aa0 2024-03-24T09:21:02.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:02 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/16941487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T09:21:02.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.233+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcc30110ff0 con 0x7fcc30101aa0 2024-03-24T09:21:02.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.234+0000 7fcc1f7fe700 1 -- 172.21.15.116:0/2099253079 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcc24096e70 con 0x7fcc30101aa0 2024-03-24T09:21:02.235 INFO:teuthology.orchestra.run.smithi116.stdout:55834574928 2024-03-24T09:21:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fcc180a21e0 msgr2=0x7fcc180a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fcc180a21e0 0x7fcc180a46a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009be0 tx=0x7fcc20009260 comp rx=0 tx=0).stop 2024-03-24T09:21:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 msgr2=0x7fcc3019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 0x7fcc3019e920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400e9a0 tx=0x7fcc2400ecb0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 shutdown_connections 2024-03-24T09:21:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fcc180a21e0 0x7fcc180a46a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcc30107140 0x7fcc301985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcc30101100 0x7fcc3019e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 --2- 172.21.15.116:0/2099253079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcc30101aa0 0x7fcc3019e920 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 >> 172.21.15.116:0/2099253079 conn(0x7fcc30078030 msgr2=0x7fcc3006a050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 shutdown_connections 2024-03-24T09:21:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.236+0000 7fcc3482e700 1 -- 172.21.15.116:0/2099253079 wait complete. 2024-03-24T09:21:02.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:02 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/16941487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T09:21:02.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:02 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/16941487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T09:21:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 -- 172.21.15.116:0/519170561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c102250 msgr2=0x7fd77c1026d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 --2- 172.21.15.116:0/519170561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c102250 0x7fd77c1026d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd7640099c0 tx=0x7fd764009cd0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 -- 172.21.15.116:0/519170561 shutdown_connections 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 --2- 172.21.15.116:0/519170561 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102c10 0x7fd77c10f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 --2- 172.21.15.116:0/519170561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c108230 0x7fd77c108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 --2- 172.21.15.116:0/519170561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c102250 0x7fd77c1026d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 -- 172.21.15.116:0/519170561 >> 172.21.15.116:0/519170561 conn(0x7fd77c079330 msgr2=0x7fd77c079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 -- 172.21.15.116:0/519170561 shutdown_connections 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 -- 172.21.15.116:0/519170561 wait complete. 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 Processor -- start 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.704+0000 7fd782659700 1 -- start start 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 0x7fd77c1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c102c10 0x7fd77c1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c108230 0x7fd77c19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c19ae20 con 0x7fd77c108230 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c19af90 con 0x7fd77c102c10 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c19b150 con 0x7fd77c102250 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd780bf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c108230 0x7fd77c19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 0x7fd77c1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c102c10 0x7fd77c1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77bfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 0x7fd77c1a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:53674/0 (socket says 172.21.15.116:53674) 2024-03-24T09:21:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77bfff700 1 -- 172.21.15.116:0/3237053517 learned_addr learned my addr 172.21.15.116:0/3237053517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd780bf6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c108230 0x7fd77c19a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44174/0 (socket says 172.21.15.116:44174) 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 -- 172.21.15.116:0/3237053517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 msgr2=0x7fd77c1a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 0x7fd77c1a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 -- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c108230 msgr2=0x7fd77c19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c108230 0x7fd77c19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 -- 172.21.15.116:0/3237053517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76c009520 con 0x7fd77c102c10 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77bfff700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 0x7fd77c1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd77b7fe700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c102c10 0x7fd77c1a0ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd7640098e0 tx=0x7fd7640039d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76401d070 con 0x7fd77c102c10 2024-03-24T09:21:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd764003e10 con 0x7fd77c102c10 2024-03-24T09:21:02.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7640095f0 con 0x7fd77c102c10 2024-03-24T09:21:02.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.705+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd764005620 con 0x7fd77c102c10 2024-03-24T09:21:02.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.706+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c1a7350 con 0x7fd77c102c10 2024-03-24T09:21:02.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.706+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c1038e0 con 0x7fd77c102c10 2024-03-24T09:21:02.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.708+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fd764004d70 con 0x7fd77c102c10 2024-03-24T09:21:02.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.708+0000 7fd7797fa700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd7680a2260 0x7fd7680a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:02.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.708+0000 7fd77bfff700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd7680a2260 0x7fd7680a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:02.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.708+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd7640cea40 con 0x7fd77c102c10 2024-03-24T09:21:02.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.709+0000 7fd77bfff700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd7680a2260 0x7fd7680a4720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0116b0 tx=0x7fd76c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:02.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.712+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd7640975e0 con 0x7fd77c102c10 2024-03-24T09:21:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.916+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd77c1120e0 con 0x7fd77c102c10 2024-03-24T09:21:02.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.917+0000 7fd7797fa700 1 -- 172.21.15.116:0/3237053517 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd76409aca0 con 0x7fd77c102c10 2024-03-24T09:21:02.918 INFO:teuthology.orchestra.run.smithi116.stdout:141733920834 2024-03-24T09:21:02.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd7680a2260 msgr2=0x7fd7680a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd7680a2260 0x7fd7680a4720 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd76c0116b0 tx=0x7fd76c009260 comp rx=0 tx=0).stop 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c102c10 msgr2=0x7fd77c1a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c102c10 0x7fd77c1a0ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd7640098e0 tx=0x7fd7640039d0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 shutdown_connections 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd7680a2260 0x7fd7680a4720 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd77c108230 0x7fd77c19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd77c102250 0x7fd77c1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 --2- 172.21.15.116:0/3237053517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd77c102c10 0x7fd77c1a0ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 >> 172.21.15.116:0/3237053517 conn(0x7fd77c079330 msgr2=0x7fd77c074730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 shutdown_connections 2024-03-24T09:21:02.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:02.919+0000 7fd782659700 1 -- 172.21.15.116:0/3237053517 wait complete. 2024-03-24T09:21:03.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:03 smithi116 ceph-mon[32207]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:03.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:03 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2099253079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T09:21:03.216 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:03 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3237053517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T09:21:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:03 smithi143 ceph-mon[35965]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:03 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2099253079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T09:21:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:03 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3237053517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T09:21:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:03 smithi117 ceph-mon[35788]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:03 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2099253079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T09:21:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:03 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3237053517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T09:21:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 -- 172.21.15.116:0/925738167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8107520 msgr2=0x7fd9b810f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/925738167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8107520 0x7fd9b810f0f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000b0a0 tx=0x7fd9b000b3b0 comp rx=0 tx=0).stop 2024-03-24T09:21:04.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 -- 172.21.15.116:0/925738167 shutdown_connections 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/925738167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8107520 0x7fd9b810f0f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/925738167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8102440 0x7fd9b8102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/925738167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 0x7fd9b8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 -- 172.21.15.116:0/925738167 >> 172.21.15.116:0/925738167 conn(0x7fd9b80fdcb0 msgr2=0x7fd9b81000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 -- 172.21.15.116:0/925738167 shutdown_connections 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.980+0000 7fd9bf95f700 1 -- 172.21.15.116:0/925738167 wait complete. 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 Processor -- start 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 -- start start 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 0x7fd9b819e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 0x7fd9b819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8107520 0x7fd9b8198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:04.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8198bf0 con 0x7fd9b8102440 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8198d30 con 0x7fd9b8102df0 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bf95f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8198ef0 con 0x7fd9b8107520 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bd6fb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 0x7fd9b819e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bd6fb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 0x7fd9b819e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44228/0 (socket says 172.21.15.116:44228) 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bd6fb700 1 -- 172.21.15.116:0/1358532375 learned_addr learned my addr 172.21.15.116:0/1358532375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bcefa700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 0x7fd9b819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.981+0000 7fd9bdefc700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8107520 0x7fd9b8198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bd6fb700 1 -- 172.21.15.116:0/1358532375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8107520 msgr2=0x7fd9b8198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bd6fb700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8107520 0x7fd9b8198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bd6fb700 1 -- 172.21.15.116:0/1358532375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 msgr2=0x7fd9b819e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bd6fb700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 0x7fd9b819e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bd6fb700 1 -- 172.21.15.116:0/1358532375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a8009520 con 0x7fd9b8102440 2024-03-24T09:21:04.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bdefc700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8107520 0x7fd9b8198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bcefa700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 0x7fd9b819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bd6fb700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 0x7fd9b819e3d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800eb30 tx=0x7fd9a800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8009880 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b0009d00 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd9a80044d0 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8003d50 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.982+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b81a52a0 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.983+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b810a840 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.984+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fd9a8003ef0 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.985+0000 7fd9ae7fc700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd9a40a21e0 0x7fd9a40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.985+0000 7fd9bcefa700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd9a40a21e0 0x7fd9a40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.985+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd9a8014070 con 0x7fd9b8102440 2024-03-24T09:21:04.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.985+0000 7fd9bcefa700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd9a40a21e0 0x7fd9a40a46a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8108a90 tx=0x7fd9b4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.989+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd9a8097110 con 0x7fd9b8102440 2024-03-24T09:21:05.166 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:05 smithi116 ceph-mon[32207]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:05.166 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:05 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1698077620' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T09:21:05.166 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:05 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2095535115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T09:21:05.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.203+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd9b8199ac0 con 0x7fd9b8102440 2024-03-24T09:21:05.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.203+0000 7fd9ae7fc700 1 -- 172.21.15.116:0/1358532375 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd9a809a7d0 con 0x7fd9b8102440 2024-03-24T09:21:05.204 INFO:teuthology.orchestra.run.smithi116.stdout:197568495672 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd9a40a21e0 msgr2=0x7fd9a40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd9a40a21e0 0x7fd9a40a46a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8108a90 tx=0x7fd9b4009260 comp rx=0 tx=0).stop 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 msgr2=0x7fd9b819e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 0x7fd9b819e3d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800eb30 tx=0x7fd9a800ee40 comp rx=0 tx=0).stop 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 shutdown_connections 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd9b8107520 0x7fd9b8198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd9a40a21e0 0x7fd9a40a46a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9b8102440 0x7fd9b819e3d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 --2- 172.21.15.116:0/1358532375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9b8102df0 0x7fd9b819e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 >> 172.21.15.116:0/1358532375 conn(0x7fd9b80fdcb0 msgr2=0x7fd9b8100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 shutdown_connections 2024-03-24T09:21:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:05.205+0000 7fd9bf95f700 1 -- 172.21.15.116:0/1358532375 wait complete. 2024-03-24T09:21:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:05 smithi143 ceph-mon[35965]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:05 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1698077620' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T09:21:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:05 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2095535115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T09:21:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:05 smithi117 ceph-mon[35788]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:05 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1698077620' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T09:21:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:05 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2095535115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T09:21:05.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574928 for osd.1 2024-03-24T09:21:05.527 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:06.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:06 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1358532375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T09:21:06.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:06.189 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:06 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:06.213 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495667 got 197568495672 for osd.6 2024-03-24T09:21:06.214 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:06 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1358532375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T09:21:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:06.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:06 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:06 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1358532375' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T09:21:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:06.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:06 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:06.652 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808872 for osd.9 2024-03-24T09:21:06.653 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.058+0000 7f9d88e69700 1 -- 172.21.15.116:0/480032231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 msgr2=0x7f9d8406ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.058+0000 7f9d88e69700 1 --2- 172.21.15.116:0/480032231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d8406ad80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9d74009990 tx=0x7f9d74009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.058+0000 7f9d88e69700 1 -- 172.21.15.116:0/480032231 shutdown_connections 2024-03-24T09:21:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 --2- 172.21.15.116:0/480032231 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 0x7f9d8410f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 --2- 172.21.15.116:0/480032231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d8406ad80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 --2- 172.21.15.116:0/480032231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d8406b350 0x7f9d84106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 -- 172.21.15.116:0/480032231 >> 172.21.15.116:0/480032231 conn(0x7f9d84077150 msgr2=0x7f9d84077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 -- 172.21.15.116:0/480032231 shutdown_connections 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 -- 172.21.15.116:0/480032231 wait complete. 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 Processor -- start 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.059+0000 7f9d88e69700 1 -- start start 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d88e69700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d84198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d8259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d84198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d8259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d84198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44194/0 (socket says 172.21.15.116:44194) 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d8259c700 1 -- 172.21.15.116:0/1698077620 learned_addr learned my addr 172.21.15.116:0/1698077620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:07.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d8406b350 0x7f9d84198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 0x7f9d8419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8419d470 con 0x7f9d8406a9a0 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8419d5e0 con 0x7f9d8406b350 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8419d7a0 con 0x7f9d84107510 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.060+0000 7f9d81d9b700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d8406b350 0x7f9d84198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d82d9d700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 0x7f9d8419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d8259c700 1 -- 172.21.15.116:0/1698077620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 msgr2=0x7f9d8419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d8259c700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 0x7f9d8419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d8259c700 1 -- 172.21.15.116:0/1698077620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d8406b350 msgr2=0x7f9d84198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d8259c700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d8406b350 0x7f9d84198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d8259c700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d740095f0 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d82d9d700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 0x7f9d8419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d8259c700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d84198590 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000c00 tx=0x7f9d74005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400fa70 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8419da20 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d74023070 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.061+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d740173f0 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.062+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d84101300 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.062+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d8410a7b0 con 0x7f9d8406a9a0 2024-03-24T09:21:07.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.063+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f9d74025030 con 0x7f9d8406a9a0 2024-03-24T09:21:07.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.064+0000 7f9d737fe700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9d6c0a2260 0x7f9d6c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:07.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.064+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9d740d3510 con 0x7f9d8406a9a0 2024-03-24T09:21:07.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.064+0000 7f9d81d9b700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9d6c0a2260 0x7f9d6c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:07.191 INFO:teuthology.orchestra.run.smithi116.stdout:107374182471 2024-03-24T09:21:07.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.065+0000 7f9d81d9b700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9d6c0a2260 0x7f9d6c0a4720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d84108a90 tx=0x7f9d78009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.069+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9d7409c1c0 con 0x7f9d8406a9a0 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.281+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f9d84103910 con 0x7f9d8406a9a0 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.281+0000 7f9d737fe700 1 -- 172.21.15.116:0/1698077620 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9d7409f880 con 0x7f9d8406a9a0 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9d6c0a2260 msgr2=0x7f9d6c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9d6c0a2260 0x7f9d6c0a4720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d84108a90 tx=0x7f9d78009260 comp rx=0 tx=0).stop 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 msgr2=0x7f9d84198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d84198590 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9d74000c00 tx=0x7f9d74005e80 comp rx=0 tx=0).stop 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 shutdown_connections 2024-03-24T09:21:07.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d84107510 0x7f9d8419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9d6c0a2260 0x7f9d6c0a4720 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d8406a9a0 0x7f9d84198590 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 --2- 172.21.15.116:0/1698077620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d8406b350 0x7f9d84198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.284+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 >> 172.21.15.116:0/1698077620 conn(0x7f9d84077150 msgr2=0x7f9d841008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.285+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 shutdown_connections 2024-03-24T09:21:07.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.285+0000 7f9d88e69700 1 -- 172.21.15.116:0/1698077620 wait complete. 2024-03-24T09:21:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:07 smithi143 ceph-mon[35965]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T09:21:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:21:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:21:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:07 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:07 smithi117 ceph-mon[35788]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T09:21:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:21:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:21:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:07 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:07.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:07 smithi116 ceph-mon[32207]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T09:21:07.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:21:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:21:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:07 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' 2024-03-24T09:21:07.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182465 got 107374182471 for osd.3 2024-03-24T09:21:07.835 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:08.239 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332141 got 219043332146 for osd.7 2024-03-24T09:21:08.239 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:09.192 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:09 smithi116 ceph-mon[32207]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:09.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 -- 172.21.15.116:0/2154091347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 msgr2=0x7f3180101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:09.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 --2- 172.21.15.116:0/2154091347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180101560 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3170009990 tx=0x7f3170009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 -- 172.21.15.116:0/2154091347 shutdown_connections 2024-03-24T09:21:09.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 --2- 172.21.15.116:0/2154091347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3180101aa0 0x7f318010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 --2- 172.21.15.116:0/2154091347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3180107140 0x7f3180107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 --2- 172.21.15.116:0/2154091347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180101560 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 -- 172.21.15.116:0/2154091347 >> 172.21.15.116:0/2154091347 conn(0x7f3180078030 msgr2=0x7f3180078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 -- 172.21.15.116:0/2154091347 shutdown_connections 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 -- 172.21.15.116:0/2154091347 wait complete. 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.160+0000 7f31863c5700 1 Processor -- start 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 -- start start 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3180101aa0 0x7f3180197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3180107140 0x7f318019be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f318019c440 con 0x7f3180101100 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f318019c5b0 con 0x7f3180107140 2024-03-24T09:21:09.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f31863c5700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f318019c770 con 0x7f3180101aa0 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44202/0 (socket says 172.21.15.116:44202) 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 -- 172.21.15.116:0/2095535115 learned_addr learned my addr 172.21.15.116:0/2095535115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317f7fe700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3180101aa0 0x7f3180197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f3184962700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3180107140 0x7f318019be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 -- 172.21.15.116:0/2095535115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3180101aa0 msgr2=0x7f3180197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3180101aa0 0x7f3180197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 -- 172.21.15.116:0/2095535115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3180107140 msgr2=0x7f318019be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3180107140 0x7f318019be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.161+0000 7f317ffff700 1 -- 172.21.15.116:0/2095535115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3174009520 con 0x7f3180101100 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.162+0000 7f317ffff700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180197560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f317400ec40 tx=0x7f317400ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:09.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.162+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174013070 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.162+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3174004020 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.162+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31700095f0 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.162+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174003b10 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.162+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3180069eb0 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.163+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3180068ec0 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.164+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f3174004190 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.165+0000 7f317d7fa700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f31680a21e0 0x7f31680a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.165+0000 7f317f7fe700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f31680a21e0 0x7f31680a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.165+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f31740cfc90 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.165+0000 7f317f7fe700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f31680a21e0 0x7f31680a46a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f317000b2e0 tx=0x7f3170005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.168+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3174098740 con 0x7f3180101100 2024-03-24T09:21:09.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.387+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3180110fd0 con 0x7f3180101100 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stdout:163208757309 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.387+0000 7f317d7fa700 1 -- 172.21.15.116:0/2095535115 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f317409be00 con 0x7f3180101100 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.389+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f31680a21e0 msgr2=0x7f31680a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.389+0000 7f31863c5700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f31680a21e0 0x7f31680a46a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f317000b2e0 tx=0x7f3170005960 comp rx=0 tx=0).stop 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 msgr2=0x7f3180197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180197560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f317400ec40 tx=0x7f317400ef50 comp rx=0 tx=0).stop 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 shutdown_connections 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3180107140 0x7f318019be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f31680a21e0 0x7f31680a46a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3180101100 0x7f3180197560 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 --2- 172.21.15.116:0/2095535115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3180101aa0 0x7f3180197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 >> 172.21.15.116:0/2095535115 conn(0x7f3180078030 msgr2=0x7f31800fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 shutdown_connections 2024-03-24T09:21:09.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:04.390+0000 7f31863c5700 1 -- 172.21.15.116:0/2095535115 wait complete. 2024-03-24T09:21:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:09 smithi117 ceph-mon[35788]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:09 smithi143 ceph-mon[35965]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:09.992 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757309 for osd.5 2024-03-24T09:21:09.992 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:10.907 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920831 got 141733920834 for osd.4 2024-03-24T09:21:10.907 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:11.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/3462611116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48102610 msgr2=0x7f2c4810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/3462611116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48102610 0x7f2c4810dfe0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c40009990 tx=0x7f2c40009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/3462611116 shutdown_connections 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/3462611116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48102610 0x7f2c4810dfe0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/3462611116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c48106140 0x7f2c48106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/3462611116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48101c50 0x7f2c481020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/3462611116 >> 172.21.15.116:0/3462611116 conn(0x7f2c48076460 msgr2=0x7f2c48076870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/3462611116 shutdown_connections 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.100+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/3462611116 wait complete. 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 Processor -- start 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 -- start start 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48101c50 0x7f2c48199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48102610 0x7f2c48199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c48106140 0x7f2c481a4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4819a2a0 con 0x7f2c48101c50 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2c481a4ad0 con 0x7f2c48102610 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c4d9b2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2c481a4c10 con 0x7f2c48106140 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c46ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48101c50 0x7f2c48199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c3e7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48102610 0x7f2c48199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c46ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48101c50 0x7f2c48199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39540/0 (socket says 172.21.15.116:39540) 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c46ffd700 1 -- 172.21.15.116:0/1303960103 learned_addr learned my addr 172.21.15.116:0/1303960103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c477fe700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c48106140 0x7f2c481a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c3e7fc700 1 -- 172.21.15.116:0/1303960103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c48106140 msgr2=0x7f2c481a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c3e7fc700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c48106140 0x7f2c481a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c3e7fc700 1 -- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48101c50 msgr2=0x7f2c48199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.101+0000 7f2c3e7fc700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48101c50 0x7f2c48199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c3e7fc700 1 -- 172.21.15.116:0/1303960103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c400095f0 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c3e7fc700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48102610 0x7f2c48199a50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000ea20 tx=0x7f2c3000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3000cbd0 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c481a4e70 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c3000ee60 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c30018680 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.102+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c481a52e0 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.103+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c48109730 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.104+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f2c30010430 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.104+0000 7f2c4c9b0700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2c340a21e0 0x7f2c340a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.104+0000 7f2c46ffd700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2c340a21e0 0x7f2c340a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.104+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f2c30014070 con 0x7f2c48102610 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.105+0000 7f2c46ffd700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2c340a21e0 0x7f2c340a46a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009c10 tx=0x7f2c3800b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:11.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.110+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2c30053280 con 0x7f2c48102610 2024-03-24T09:21:11.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.311+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f2c48111070 con 0x7f2c48102610 2024-03-24T09:21:11.312 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:11 smithi116 ceph-mon[32207]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-24T09:21:11.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.311+0000 7f2c4c9b0700 1 -- 172.21.15.116:0/1303960103 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2c30096e30 con 0x7f2c48102610 2024-03-24T09:21:11.312 INFO:teuthology.orchestra.run.smithi116.stdout:253403070512 2024-03-24T09:21:11.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2c340a21e0 msgr2=0x7f2c340a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2c340a21e0 0x7f2c340a46a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009c10 tx=0x7f2c3800b040 comp rx=0 tx=0).stop 2024-03-24T09:21:11.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48102610 msgr2=0x7f2c48199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48102610 0x7f2c48199a50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c3000ea20 tx=0x7f2c3000ed30 comp rx=0 tx=0).stop 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 shutdown_connections 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c48106140 0x7f2c481a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2c340a21e0 0x7f2c340a46a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c48101c50 0x7f2c48199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 --2- 172.21.15.116:0/1303960103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c48102610 0x7f2c48199a50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 >> 172.21.15.116:0/1303960103 conn(0x7f2c48076460 msgr2=0x7f2c480feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 shutdown_connections 2024-03-24T09:21:11.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.314+0000 7f2c4d9b2700 1 -- 172.21.15.116:0/1303960103 wait complete. 2024-03-24T09:21:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:11 smithi117 ceph-mon[35788]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 -- 172.21.15.116:0/1377240902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff601071c0 msgr2=0x7eff6006a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 --2- 172.21.15.116:0/1377240902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff601071c0 0x7eff6006a740 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7eff500098f0 tx=0x7eff50009c00 comp rx=0 tx=0).stop 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 -- 172.21.15.116:0/1377240902 shutdown_connections 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 --2- 172.21.15.116:0/1377240902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 0x7eff601050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 --2- 172.21.15.116:0/1377240902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff601071c0 0x7eff6006a740 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 --2- 172.21.15.116:0/1377240902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff6006ac80 0x7eff6006b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.327+0000 7eff67925700 1 -- 172.21.15.116:0/1377240902 >> 172.21.15.116:0/1377240902 conn(0x7eff60077150 msgr2=0x7eff60077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 -- 172.21.15.116:0/1377240902 shutdown_connections 2024-03-24T09:21:11.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 -- 172.21.15.116:0/1377240902 wait complete. 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 Processor -- start 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 -- start start 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff6006ac80 0x7eff601a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 0x7eff601a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff601071c0 0x7eff6019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7eff6019f020 con 0x7eff60104c30 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7eff6019f160 con 0x7eff601071c0 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.328+0000 7eff67925700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7eff6019f320 con 0x7eff6006ac80 2024-03-24T09:21:11.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff64ec0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 0x7eff601a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff6006ac80 0x7eff601a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff64ec0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 0x7eff601a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39560/0 (socket says 172.21.15.116:39560) 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff64ec0700 1 -- 172.21.15.116:0/443779491 learned_addr learned my addr 172.21.15.116:0/443779491 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff65ec2700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff601071c0 0x7eff6019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 -- 172.21.15.116:0/443779491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff601071c0 msgr2=0x7eff6019e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff601071c0 0x7eff6019e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 -- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 msgr2=0x7eff601a4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 0x7eff601a4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 -- 172.21.15.116:0/443779491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff50009520 con 0x7eff6006ac80 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff656c1700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff6006ac80 0x7eff601a4800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff50009fd0 tx=0x7eff5000fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5001c070 con 0x7eff6006ac80 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7eff5000ba10 con 0x7eff6006ac80 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6019f5a0 con 0x7eff6006ac80 2024-03-24T09:21:11.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff500038e0 con 0x7eff6006ac80 2024-03-24T09:21:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.329+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff601ab8a0 con 0x7eff6006ac80 2024-03-24T09:21:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.330+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff60068ec0 con 0x7eff6006ac80 2024-03-24T09:21:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.331+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7eff50020030 con 0x7eff6006ac80 2024-03-24T09:21:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.332+0000 7eff567fc700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7eff4c0a2260 0x7eff4c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.332+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7eff500cee90 con 0x7eff6006ac80 2024-03-24T09:21:11.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.332+0000 7eff64ec0700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7eff4c0a2260 0x7eff4c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:11.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.333+0000 7eff64ec0700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7eff4c0a2260 0x7eff4c0a4720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff6006aa30 tx=0x7eff5c009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:11.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.341+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7eff50097a30 con 0x7eff6006ac80 2024-03-24T09:21:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:11 smithi143 ceph-mon[35965]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s 2024-03-24T09:21:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.537+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7eff601186e0 con 0x7eff6006ac80 2024-03-24T09:21:11.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.538+0000 7eff567fc700 1 -- 172.21.15.116:0/443779491 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7eff5009b0f0 con 0x7eff6006ac80 2024-03-24T09:21:11.538 INFO:teuthology.orchestra.run.smithi116.stdout:343597383712 2024-03-24T09:21:11.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7eff4c0a2260 msgr2=0x7eff4c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7eff4c0a2260 0x7eff4c0a4720 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff6006aa30 tx=0x7eff5c009310 comp rx=0 tx=0).stop 2024-03-24T09:21:11.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff6006ac80 msgr2=0x7eff601a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:11.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff6006ac80 0x7eff601a4800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7eff50009fd0 tx=0x7eff5000fa00 comp rx=0 tx=0).stop 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 shutdown_connections 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff601071c0 0x7eff6019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7eff6006ac80 0x7eff601a4800 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7eff4c0a2260 0x7eff4c0a4720 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 --2- 172.21.15.116:0/443779491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff60104c30 0x7eff601a4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 >> 172.21.15.116:0/443779491 conn(0x7eff60077150 msgr2=0x7eff60100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 shutdown_connections 2024-03-24T09:21:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:11.540+0000 7eff67925700 1 -- 172.21.15.116:0/443779491 wait complete. 2024-03-24T09:21:12.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:12 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1303960103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T09:21:12.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:12 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/443779491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T09:21:12.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:12 smithi117 ceph-mon[35788]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T09:21:12.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:12 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1303960103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T09:21:12.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:12 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/443779491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T09:21:12.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:12 smithi116 ceph-mon[32207]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T09:21:12.353 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070506 got 253403070512 for osd.8 2024-03-24T09:21:12.353 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:12 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1303960103' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T09:21:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:12 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/443779491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T09:21:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:12 smithi143 ceph-mon[35965]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-24T09:21:13.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 -- 172.21.15.116:0/3780753328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 msgr2=0x7f423c10dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:13.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 --2- 172.21.15.116:0/3780753328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c10dfb0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f423400b0a0 tx=0x7f423400b3b0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 -- 172.21.15.116:0/3780753328 shutdown_connections 2024-03-24T09:21:13.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 --2- 172.21.15.116:0/3780753328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c10dfb0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 --2- 172.21.15.116:0/3780753328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101390 0x7f423c101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 --2- 172.21.15.116:0/3780753328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101d40 0x7f423c105d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 -- 172.21.15.116:0/3780753328 >> 172.21.15.116:0/3780753328 conn(0x7f423c077150 msgr2=0x7f423c077560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 -- 172.21.15.116:0/3780753328 shutdown_connections 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 -- 172.21.15.116:0/3780753328 wait complete. 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 Processor -- start 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.139+0000 7f4241f07700 1 -- start start 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f4241f07700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 0x7f423c102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f4241f07700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 0x7f423c1032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f4241f07700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c103820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f4241f07700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f423c1a6b90 con 0x7f423c1063e0 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f4241f07700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f423c1a6d00 con 0x7f423c101d40 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f4241f07700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f423c1a6ec0 con 0x7f423c101390 2024-03-24T09:21:13.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 0x7f423c102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c103820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 0x7f423c1032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 0x7f423c102da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:43014/0 (socket says 172.21.15.116:43014) 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c103820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39584/0 (socket says 172.21.15.116:39584) 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 -- 172.21.15.116:0/2504082399 learned_addr learned my addr 172.21.15.116:0/2504082399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 0x7f423c1032e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:53680/0 (socket says 172.21.15.116:53680) 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 -- 172.21.15.116:0/2504082399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 msgr2=0x7f423c1032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 0x7f423c1032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 -- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 msgr2=0x7f423c103820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c103820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423b7fe700 1 -- 172.21.15.116:0/2504082399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4234009d00 con 0x7f423c101390 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423bfff700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.140+0000 7f423affd700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 0x7f423c1032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:13.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.141+0000 7f423b7fe700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 0x7f423c102da0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f422c00e9a0 tx=0x7f422c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.141+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c009790 con 0x7f423c101390 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.141+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f422c010430 con 0x7f423c101390 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.141+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423c1a71a0 con 0x7f423c101390 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.141+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f422c018680 con 0x7f423c101390 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.141+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423c1a76a0 con 0x7f423c101390 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.142+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423c10a710 con 0x7f423c101390 2024-03-24T09:21:13.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.144+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f422c00ca10 con 0x7f423c101390 2024-03-24T09:21:13.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.144+0000 7f4238ff9700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f42240a21e0 0x7f42240a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:13.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.145+0000 7f423affd700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f42240a21e0 0x7f42240a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:13.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.145+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f422c014070 con 0x7f423c101390 2024-03-24T09:21:13.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.145+0000 7f423affd700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f42240a21e0 0x7f42240a46a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f423c107a40 tx=0x7f4230009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:13.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.149+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f422c053280 con 0x7f423c101390 2024-03-24T09:21:13.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.354+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f423c1042d0 con 0x7f423c101390 2024-03-24T09:21:13.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.354+0000 7f4238ff9700 1 -- 172.21.15.116:0/2504082399 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f422c096e70 con 0x7f423c101390 2024-03-24T09:21:13.355 INFO:teuthology.orchestra.run.smithi116.stdout:77309411406 2024-03-24T09:21:13.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f42240a21e0 msgr2=0x7f42240a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f42240a21e0 0x7f42240a46a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f423c107a40 tx=0x7f4230009260 comp rx=0 tx=0).stop 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 msgr2=0x7f423c102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 0x7f423c102da0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f422c00e9a0 tx=0x7f422c00ecb0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 shutdown_connections 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f42240a21e0 0x7f42240a46a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f423c1063e0 0x7f423c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f423c101390 0x7f423c102da0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 --2- 172.21.15.116:0/2504082399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f423c101d40 0x7f423c1032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 >> 172.21.15.116:0/2504082399 conn(0x7f423c077150 msgr2=0x7f423c06aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 shutdown_connections 2024-03-24T09:21:13.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:13.357+0000 7f4241f07700 1 -- 172.21.15.116:0/2504082399 wait complete. 2024-03-24T09:21:13.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2504082399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T09:21:13.785 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2504082399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T09:21:13.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2504082399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T09:21:13.920 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383707 got 343597383712 for osd.11 2024-03-24T09:21:13.920 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:14.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:14 smithi116 ceph-mon[32207]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:14.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:14 smithi143 ceph-mon[35965]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:14.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:14 smithi117 ceph-mon[35788]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 -- 172.21.15.116:0/2217034360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900101390 msgr2=0x7f3900101770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 --2- 172.21.15.116:0/2217034360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900101390 0x7f3900101770 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009990 tx=0x7f38f0009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 -- 172.21.15.116:0/2217034360 shutdown_connections 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 --2- 172.21.15.116:0/2217034360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900106410 0x7f390010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 --2- 172.21.15.116:0/2217034360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900101390 0x7f3900101770 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 --2- 172.21.15.116:0/2217034360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101d40 0x7f3900105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.860+0000 7f3907da9700 1 -- 172.21.15.116:0/2217034360 >> 172.21.15.116:0/2217034360 conn(0x7f3900077110 msgr2=0x7f3900077520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 -- 172.21.15.116:0/2217034360 shutdown_connections 2024-03-24T09:21:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 -- 172.21.15.116:0/2217034360 wait complete. 2024-03-24T09:21:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 Processor -- start 2024-03-24T09:21:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 -- start start 2024-03-24T09:21:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 0x7f39001994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900101d40 0x7f39001999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900106410 0x7f39001a4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f390019a280 con 0x7f3900106410 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39001a4a60 con 0x7f3900101d40 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3907da9700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f39001a4ba0 con 0x7f3900101390 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3905b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 0x7f39001994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.861+0000 7f3905b45700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 0x7f39001994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:43034/0 (socket says 172.21.15.116:43034) 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905344700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900101d40 0x7f39001999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 -- 172.21.15.116:0/749914317 learned_addr learned my addr 172.21.15.116:0/749914317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3906346700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900106410 0x7f39001a4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 -- 172.21.15.116:0/749914317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900101d40 msgr2=0x7f39001999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900101d40 0x7f39001999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 -- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900106410 msgr2=0x7f39001a4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900106410 0x7f39001a4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 -- 172.21.15.116:0/749914317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f00095f0 con 0x7f3900101390 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3906346700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900106410 0x7f39001a4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905344700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900101d40 0x7f39001999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3905b45700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 0x7f39001994a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000c00 tx=0x7f38f0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f001d070 con 0x7f3900101390 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39001a4df0 con 0x7f3900101390 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38f0022470 con 0x7f3900101390 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0017650 con 0x7f3900101390 2024-03-24T09:21:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.862+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39001a5200 con 0x7f3900101390 2024-03-24T09:21:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.863+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3900109630 con 0x7f3900101390 2024-03-24T09:21:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.865+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f38f00225e0 con 0x7f3900101390 2024-03-24T09:21:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.865+0000 7f38f6ffd700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f38ec0a2260 0x7f38ec0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.865+0000 7f3905344700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f38ec0a2260 0x7f38ec0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.865+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f38f00cf7c0 con 0x7f3900101390 2024-03-24T09:21:14.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.866+0000 7f3905344700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f38ec0a2260 0x7f38ec0a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3900100dc0 tx=0x7f38fc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:14.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:14.873+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f38f0098300 con 0x7f3900101390 2024-03-24T09:21:15.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.073+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f3900111000 con 0x7f3900101390 2024-03-24T09:21:15.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.073+0000 7f38f6ffd700 1 -- 172.21.15.116:0/749914317 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f38f009b9c0 con 0x7f3900101390 2024-03-24T09:21:15.074 INFO:teuthology.orchestra.run.smithi116.stdout:309237645349 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f38ec0a2260 msgr2=0x7f38ec0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f38ec0a2260 0x7f38ec0a4720 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f3900100dc0 tx=0x7f38fc009260 comp rx=0 tx=0).stop 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 msgr2=0x7f39001994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 0x7f39001994a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f38f0000c00 tx=0x7f38f0004730 comp rx=0 tx=0).stop 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 shutdown_connections 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f38ec0a2260 0x7f38ec0a4720 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3900106410 0x7f39001a4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3900101390 0x7f39001994a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 --2- 172.21.15.116:0/749914317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3900101d40 0x7f39001999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 >> 172.21.15.116:0/749914317 conn(0x7f3900077110 msgr2=0x7f39000ff500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 shutdown_connections 2024-03-24T09:21:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:15.075+0000 7f3907da9700 1 -- 172.21.15.116:0/749914317 wait complete. 2024-03-24T09:21:15.297 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411400 got 77309411406 for osd.2 2024-03-24T09:21:15.297 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:15.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:15 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/749914317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T09:21:15.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:15 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/749914317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T09:21:15.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:15 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/749914317' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T09:21:15.980 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645342 got 309237645349 for osd.10 2024-03-24T09:21:15.980 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:16.732 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:16 smithi116 ceph-mon[32207]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-24T09:21:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:16 smithi143 ceph-mon[35965]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-24T09:21:16.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:16 smithi117 ceph-mon[35788]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.130+0000 7fbb841d2700 1 -- 172.21.15.116:0/749999788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c102e90 msgr2=0x7fbb7c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.130+0000 7fbb841d2700 1 --2- 172.21.15.116:0/749999788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c102e90 0x7fbb7c106e60 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c009990 tx=0x7fbb6c009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.130+0000 7fbb841d2700 1 -- 172.21.15.116:0/749999788 shutdown_connections 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.130+0000 7fbb841d2700 1 --2- 172.21.15.116:0/749999788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb7c107590 0x7fbb7c10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.130+0000 7fbb841d2700 1 --2- 172.21.15.116:0/749999788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 0x7fbb7c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.130+0000 7fbb841d2700 1 --2- 172.21.15.116:0/749999788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c102e90 0x7fbb7c106e60 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- 172.21.15.116:0/749999788 >> 172.21.15.116:0/749999788 conn(0x7fbb7c079320 msgr2=0x7fbb7c079730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:17.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- 172.21.15.116:0/749999788 shutdown_connections 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- 172.21.15.116:0/749999788 wait complete. 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 Processor -- start 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- start start 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 0x7fbb7c198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb7c102e90 0x7fbb7c198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 0x7fbb7c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c19d4f0 con 0x7fbb7c107590 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c19d660 con 0x7fbb7c102e90 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb841d2700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbb7c19d820 con 0x7fbb7c1024e0 2024-03-24T09:21:17.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb8276f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 0x7fbb7c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb8276f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 0x7fbb7c19cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44242/0 (socket says 172.21.15.116:44242) 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb81f6e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 0x7fbb7c198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.131+0000 7fbb8276f700 1 -- 172.21.15.116:0/1984950679 learned_addr learned my addr 172.21.15.116:0/1984950679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8176d700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb7c102e90 0x7fbb7c198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8276f700 1 -- 172.21.15.116:0/1984950679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 msgr2=0x7fbb7c198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8276f700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 0x7fbb7c198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8276f700 1 -- 172.21.15.116:0/1984950679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb7c102e90 msgr2=0x7fbb7c198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8276f700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb7c102e90 0x7fbb7c198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8276f700 1 -- 172.21.15.116:0/1984950679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c0095f0 con 0x7fbb7c107590 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb81f6e700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 0x7fbb7c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb8276f700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 0x7fbb7c19cee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400ef30 tx=0x7fbb7400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb74018070 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb7400f040 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7c19daa0 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7400cd20 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.132+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb7c1a5360 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.133+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb7c10a780 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.135+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fbb74003a00 con 0x7fbb7c107590 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.135+0000 7fbb72ffd700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbb680a2260 0x7fbb680a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.136+0000 7fbb81f6e700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbb680a2260 0x7fbb680a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.136+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbb740ce510 con 0x7fbb7c107590 2024-03-24T09:21:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.136+0000 7fbb81f6e700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbb680a2260 0x7fbb680a4720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb78011490 tx=0x7fbb78009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.139+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbb740970b0 con 0x7fbb7c107590 2024-03-24T09:21:17.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.336+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbb7c1999d0 con 0x7fbb7c107590 2024-03-24T09:21:17.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.336+0000 7fbb72ffd700 1 -- 172.21.15.116:0/1984950679 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb7409a770 con 0x7fbb7c107590 2024-03-24T09:21:17.337 INFO:teuthology.orchestra.run.smithi116.stdout:34359738456 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbb680a2260 msgr2=0x7fbb680a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbb680a2260 0x7fbb680a4720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb78011490 tx=0x7fbb78009260 comp rx=0 tx=0).stop 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 msgr2=0x7fbb7c19cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 0x7fbb7c19cee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400ef30 tx=0x7fbb7400c3a0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 shutdown_connections 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbb680a2260 0x7fbb680a4720 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb7c107590 0x7fbb7c19cee0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb7c1024e0 0x7fbb7c198610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 --2- 172.21.15.116:0/1984950679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb7c102e90 0x7fbb7c198b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:17.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 >> 172.21.15.116:0/1984950679 conn(0x7fbb7c079320 msgr2=0x7fbb7c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:17.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 shutdown_connections 2024-03-24T09:21:17.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:17.338+0000 7fbb841d2700 1 -- 172.21.15.116:0/1984950679 wait complete. 2024-03-24T09:21:17.701 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:17 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1984950679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T09:21:17.717 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:17 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1984950679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T09:21:17.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:17 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1984950679' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T09:21:18.066 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738450 got 34359738456 for osd.0 2024-03-24T09:21:18.066 DEBUG:teuthology.parallel:result is None 2024-03-24T09:21:18.067 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T09:21:18.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T09:21:18.314 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:18.704 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:18 smithi116 ceph-mon[32207]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:18.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:18 smithi143 ceph-mon[35965]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:18.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:18 smithi117 ceph-mon[35788]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-24T09:21:20.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 -- 172.21.15.116:0/2606854480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc101aa0 msgr2=0x7fc8dc10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:20.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 --2- 172.21.15.116:0/2606854480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc10df80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400b0a0 tx=0x7fc8d400b3b0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 -- 172.21.15.116:0/2606854480 shutdown_connections 2024-03-24T09:21:20.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 --2- 172.21.15.116:0/2606854480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc10df80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 --2- 172.21.15.116:0/2606854480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc107140 0x7fc8dc107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 --2- 172.21.15.116:0/2606854480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101100 0x7fc8dc101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 -- 172.21.15.116:0/2606854480 >> 172.21.15.116:0/2606854480 conn(0x7fc8dc078030 msgr2=0x7fc8dc078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 -- 172.21.15.116:0/2606854480 shutdown_connections 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.988+0000 7fc8e257e700 1 -- 172.21.15.116:0/2606854480 wait complete. 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 Processor -- start 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 -- start start 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc101100 0x7fc8dc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 0x7fc8dc19cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc19d590 con 0x7fc8dc107140 2024-03-24T09:21:20.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc19d700 con 0x7fc8dc101aa0 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.989+0000 7fc8e257e700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc19d8c0 con 0x7fc8dc101100 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 0x7fc8dc19cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8dbfff700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc101100 0x7fc8dc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8db7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 0x7fc8dc19cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44260/0 (socket says 172.21.15.116:44260) 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 -- 172.21.15.116:0/963218390 learned_addr learned my addr 172.21.15.116:0/963218390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 -- 172.21.15.116:0/963218390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc101100 msgr2=0x7fc8dc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc101100 0x7fc8dc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 -- 172.21.15.116:0/963218390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101aa0 msgr2=0x7fc8dc198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:20.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 -- 172.21.15.116:0/963218390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d0009520 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8db7fe700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e0b1b700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 0x7fc8dc19cf80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400b070 tx=0x7fc8d400bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400e040 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc8d4005040 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d4009d00 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d401b680 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.990+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc1a5230 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.991+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8dc1027f0 con 0x7fc8dc107140 2024-03-24T09:21:20.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.992+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fc8d40051b0 con 0x7fc8dc107140 2024-03-24T09:21:20.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.993+0000 7fc8d97fa700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc8c40a21e0 0x7fc8c40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:20.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.993+0000 7fc8dbfff700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc8c40a21e0 0x7fc8c40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:20.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.993+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc8d40cf410 con 0x7fc8dc107140 2024-03-24T09:21:20.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.993+0000 7fc8dbfff700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc8c40a21e0 0x7fc8c40a46a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009be0 tx=0x7fc8cc009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:20.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:19.997+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc8d4097fb0 con 0x7fc8dc107140 2024-03-24T09:21:20.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.193+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc8dc19de20 con 0x7fc8c40a21e0 2024-03-24T09:21:20.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.203+0000 7fc8d97fa700 1 -- 172.21.15.116:0/963218390 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340714 (secure 0 0 0) 0x7fc8dc19de20 con 0x7fc8c40a21e0 2024-03-24T09:21:20.209 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc8c40a21e0 msgr2=0x7fc8c40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc8c40a21e0 0x7fc8c40a46a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009be0 tx=0x7fc8cc009260 comp rx=0 tx=0).stop 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 msgr2=0x7fc8dc19cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 0x7fc8dc19cf80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc8d400b070 tx=0x7fc8d400bdd0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 shutdown_connections 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fc8c40a21e0 0x7fc8c40a46a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc107140 0x7fc8dc19cf80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8dc101100 0x7fc8dc198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 --2- 172.21.15.116:0/963218390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8dc101aa0 0x7fc8dc198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:20.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 >> 172.21.15.116:0/963218390 conn(0x7fc8dc078030 msgr2=0x7fc8dc100400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:20.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 shutdown_connections 2024-03-24T09:21:20.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:20.206+0000 7fc8e257e700 1 -- 172.21.15.116:0/963218390 wait complete. 2024-03-24T09:21:20.215 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:21:20.941 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-03-24T09:21:19.946300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661162,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":902,"num_read_kb":575,"num_write":425,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378176,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563648,"statfs":{"total":1151940427776,"available":1151553175552,"internally_reserved":0,"allocated":12591104,"data_stored":7969292,"data_compressed":359865,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003599"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300382+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:20:35.300382+0000","last_peered":"2024-03-24T09:20:35.300382+0000","last_clean":"2024-03-24T09:20:35.300382+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:20:35.300382+0000","last_undegraded":"2024-03-24T09:20:35.300382+0000","last_fullsized":"2024-03-24T09:20:35.300382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026713+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:20:35.026713+0000","last_peered":"2024-03-24T09:20:35.026713+0000","last_clean":"2024-03-24T09:20:35.026713+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:20:35.026713+0000","last_undegraded":"2024-03-24T09:20:35.026713+0000","last_fullsized":"2024-03-24T09:20:35.026713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026550+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:20:35.026550+0000","last_peered":"2024-03-24T09:20:35.026550+0000","last_clean":"2024-03-24T09:20:35.026550+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:20:35.026550+0000","last_undegraded":"2024-03-24T09:20:35.026550+0000","last_fullsized":"2024-03-24T09:20:35.026550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026757+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:20:35.026757+0000","last_peered":"2024-03-24T09:20:35.026757+0000","last_clean":"2024-03-24T09:20:35.026757+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:20:35.026757+0000","last_undegraded":"2024-03-24T09:20:35.026757+0000","last_fullsized":"2024-03-24T09:20:35.026757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025270+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:20:35.025270+0000","last_peered":"2024-03-24T09:20:35.025270+0000","last_clean":"2024-03-24T09:20:35.025270+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:20:35.025270+0000","last_undegraded":"2024-03-24T09:20:35.025270+0000","last_fullsized":"2024-03-24T09:20:35.025270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029608+0000","last_change":"2024-03-24T09:20:35.029608+0000","last_active":"2024-03-24T09:20:35.029608+0000","last_peered":"2024-03-24T09:20:35.029608+0000","last_clean":"2024-03-24T09:20:35.029608+0000","last_became_active":"2024-03-24T09:20:35.029268+0000","last_became_peered":"2024-03-24T09:20:35.029268+0000","last_unstale":"2024-03-24T09:20:35.029608+0000","last_undegraded":"2024-03-24T09:20:35.029608+0000","last_fullsized":"2024-03-24T09:20:35.029608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:47.087125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028091+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:20:35.028091+0000","last_peered":"2024-03-24T09:20:35.028091+0000","last_clean":"2024-03-24T09:20:35.028091+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:20:35.028091+0000","last_undegraded":"2024-03-24T09:20:35.028091+0000","last_fullsized":"2024-03-24T09:20:35.028091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300114+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:20:35.300114+0000","last_peered":"2024-03-24T09:20:35.300114+0000","last_clean":"2024-03-24T09:20:35.300114+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:20:35.300114+0000","last_undegraded":"2024-03-24T09:20:35.300114+0000","last_fullsized":"2024-03-24T09:20:35.300114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025345+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:20:35.025345+0000","last_peered":"2024-03-24T09:20:35.025345+0000","last_clean":"2024-03-24T09:20:35.025345+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:20:35.025345+0000","last_undegraded":"2024-03-24T09:20:35.025345+0000","last_fullsized":"2024-03-24T09:20:35.025345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026663+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:20:35.026663+0000","last_peered":"2024-03-24T09:20:35.026663+0000","last_clean":"2024-03-24T09:20:35.026663+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:20:35.026663+0000","last_undegraded":"2024-03-24T09:20:35.026663+0000","last_fullsized":"2024-03-24T09:20:35.026663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030438+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:20:35.030438+0000","last_peered":"2024-03-24T09:20:35.030438+0000","last_clean":"2024-03-24T09:20:35.030438+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:20:35.030438+0000","last_undegraded":"2024-03-24T09:20:35.030438+0000","last_fullsized":"2024-03-24T09:20:35.030438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026667+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:20:35.026667+0000","last_peered":"2024-03-24T09:20:35.026667+0000","last_clean":"2024-03-24T09:20:35.026667+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:20:35.026667+0000","last_undegraded":"2024-03-24T09:20:35.026667+0000","last_fullsized":"2024-03-24T09:20:35.026667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300347+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:20:35.300347+0000","last_peered":"2024-03-24T09:20:35.300347+0000","last_clean":"2024-03-24T09:20:35.300347+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:20:35.300347+0000","last_undegraded":"2024-03-24T09:20:35.300347+0000","last_fullsized":"2024-03-24T09:20:35.300347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026382+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:20:35.026382+0000","last_peered":"2024-03-24T09:20:35.026382+0000","last_clean":"2024-03-24T09:20:35.026382+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:20:35.026382+0000","last_undegraded":"2024-03-24T09:20:35.026382+0000","last_fullsized":"2024-03-24T09:20:35.026382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026741+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:20:35.026741+0000","last_peered":"2024-03-24T09:20:35.026741+0000","last_clean":"2024-03-24T09:20:35.026741+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:20:35.026741+0000","last_undegraded":"2024-03-24T09:20:35.026741+0000","last_fullsized":"2024-03-24T09:20:35.026741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024361+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:20:35.024361+0000","last_peered":"2024-03-24T09:20:35.024361+0000","last_clean":"2024-03-24T09:20:35.024361+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:20:35.024361+0000","last_undegraded":"2024-03-24T09:20:35.024361+0000","last_fullsized":"2024-03-24T09:20:35.024361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300199+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:20:35.300199+0000","last_peered":"2024-03-24T09:20:35.300199+0000","last_clean":"2024-03-24T09:20:35.300199+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:20:35.300199+0000","last_undegraded":"2024-03-24T09:20:35.300199+0000","last_fullsized":"2024-03-24T09:20:35.300199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023848+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:20:35.023848+0000","last_peered":"2024-03-24T09:20:35.023848+0000","last_clean":"2024-03-24T09:20:35.023848+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:20:35.023848+0000","last_undegraded":"2024-03-24T09:20:35.023848+0000","last_fullsized":"2024-03-24T09:20:35.023848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024297+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:20:35.024297+0000","last_peered":"2024-03-24T09:20:35.024297+0000","last_clean":"2024-03-24T09:20:35.024297+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:20:35.024297+0000","last_undegraded":"2024-03-24T09:20:35.024297+0000","last_fullsized":"2024-03-24T09:20:35.024297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023771+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:20:35.023771+0000","last_peered":"2024-03-24T09:20:35.023771+0000","last_clean":"2024-03-24T09:20:35.023771+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:20:35.023771+0000","last_undegraded":"2024-03-24T09:20:35.023771+0000","last_fullsized":"2024-03-24T09:20:35.023771+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:14.026602+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:21:14.026602+0000","last_peered":"2024-03-24T09:21:14.026602+0000","last_clean":"2024-03-24T09:21:14.026602+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:21:14.026602+0000","last_undegraded":"2024-03-24T09:21:14.026602+0000","last_fullsized":"2024-03-24T09:21:14.026602+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301400+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:20:35.301400+0000","last_peered":"2024-03-24T09:20:35.301400+0000","last_clean":"2024-03-24T09:20:35.301400+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:20:35.301400+0000","last_undegraded":"2024-03-24T09:20:35.301400+0000","last_fullsized":"2024-03-24T09:20:35.301400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.302025+0000","last_change":"2024-03-24T09:20:35.302025+0000","last_active":"2024-03-24T09:20:35.302025+0000","last_peered":"2024-03-24T09:20:35.302025+0000","last_clean":"2024-03-24T09:20:35.302025+0000","last_became_active":"2024-03-24T09:20:35.301575+0000","last_became_peered":"2024-03-24T09:20:35.301575+0000","last_unstale":"2024-03-24T09:20:35.302025+0000","last_undegraded":"2024-03-24T09:20:35.302025+0000","last_fullsized":"2024-03-24T09:20:35.302025+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:20.305175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301569+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:20:35.301569+0000","last_peered":"2024-03-24T09:20:35.301569+0000","last_clean":"2024-03-24T09:20:35.301569+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:20:35.301569+0000","last_undegraded":"2024-03-24T09:20:35.301569+0000","last_fullsized":"2024-03-24T09:20:35.301569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026514+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:20:35.026514+0000","last_peered":"2024-03-24T09:20:35.026514+0000","last_clean":"2024-03-24T09:20:35.026514+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:20:35.026514+0000","last_undegraded":"2024-03-24T09:20:35.026514+0000","last_fullsized":"2024-03-24T09:20:35.026514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025747+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:20:35.025747+0000","last_peered":"2024-03-24T09:20:35.025747+0000","last_clean":"2024-03-24T09:20:35.025747+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:20:35.025747+0000","last_undegraded":"2024-03-24T09:20:35.025747+0000","last_fullsized":"2024-03-24T09:20:35.025747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025746+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:20:35.025746+0000","last_peered":"2024-03-24T09:20:35.025746+0000","last_clean":"2024-03-24T09:20:35.025746+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:20:35.025746+0000","last_undegraded":"2024-03-24T09:20:35.025746+0000","last_fullsized":"2024-03-24T09:20:35.025746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024995+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:20:35.024995+0000","last_peered":"2024-03-24T09:20:35.024995+0000","last_clean":"2024-03-24T09:20:35.024995+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:20:35.024995+0000","last_undegraded":"2024-03-24T09:20:35.024995+0000","last_fullsized":"2024-03-24T09:20:35.024995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025428+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:20:35.025428+0000","last_peered":"2024-03-24T09:20:35.025428+0000","last_clean":"2024-03-24T09:20:35.025428+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:20:35.025428+0000","last_undegraded":"2024-03-24T09:20:35.025428+0000","last_fullsized":"2024-03-24T09:20:35.025428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025318+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:20:35.025318+0000","last_peered":"2024-03-24T09:20:35.025318+0000","last_clean":"2024-03-24T09:20:35.025318+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:20:35.025318+0000","last_undegraded":"2024-03-24T09:20:35.025318+0000","last_fullsized":"2024-03-24T09:20:35.025318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.032271+0000","last_change":"2024-03-24T09:20:35.032271+0000","last_active":"2024-03-24T09:20:35.032271+0000","last_peered":"2024-03-24T09:20:35.032271+0000","last_clean":"2024-03-24T09:20:35.032271+0000","last_became_active":"2024-03-24T09:20:35.031913+0000","last_became_peered":"2024-03-24T09:20:35.031913+0000","last_unstale":"2024-03-24T09:20:35.032271+0000","last_undegraded":"2024-03-24T09:20:35.032271+0000","last_fullsized":"2024-03-24T09:20:35.032271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:40.322197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024135+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:20:35.024135+0000","last_peered":"2024-03-24T09:20:35.024135+0000","last_clean":"2024-03-24T09:20:35.024135+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:20:35.024135+0000","last_undegraded":"2024-03-24T09:20:35.024135+0000","last_fullsized":"2024-03-24T09:20:35.024135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:17.972407+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:21:17.972407+0000","last_peered":"2024-03-24T09:21:17.972407+0000","last_clean":"2024-03-24T09:21:17.972407+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:21:17.972407+0000","last_undegraded":"2024-03-24T09:21:17.972407+0000","last_fullsized":"2024-03-24T09:21:17.972407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.031176+0000","last_change":"2024-03-24T09:20:35.031176+0000","last_active":"2024-03-24T09:20:35.031176+0000","last_peered":"2024-03-24T09:20:35.031176+0000","last_clean":"2024-03-24T09:20:35.031176+0000","last_became_active":"2024-03-24T09:20:35.030888+0000","last_became_peered":"2024-03-24T09:20:35.030888+0000","last_unstale":"2024-03-24T09:20:35.031176+0000","last_undegraded":"2024-03-24T09:20:35.031176+0000","last_fullsized":"2024-03-24T09:20:35.031176+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:43.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024380+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:20:35.024380+0000","last_peered":"2024-03-24T09:20:35.024380+0000","last_clean":"2024-03-24T09:20:35.024380+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:20:35.024380+0000","last_undegraded":"2024-03-24T09:20:35.024380+0000","last_fullsized":"2024-03-24T09:20:35.024380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027076+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:20:35.027076+0000","last_peered":"2024-03-24T09:20:35.027076+0000","last_clean":"2024-03-24T09:20:35.027076+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:20:35.027076+0000","last_undegraded":"2024-03-24T09:20:35.027076+0000","last_fullsized":"2024-03-24T09:20:35.027076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301944+0000","last_change":"2024-03-24T09:20:35.301944+0000","last_active":"2024-03-24T09:20:35.301944+0000","last_peered":"2024-03-24T09:20:35.301944+0000","last_clean":"2024-03-24T09:20:35.301944+0000","last_became_active":"2024-03-24T09:20:35.301588+0000","last_became_peered":"2024-03-24T09:20:35.301588+0000","last_unstale":"2024-03-24T09:20:35.301944+0000","last_undegraded":"2024-03-24T09:20:35.301944+0000","last_fullsized":"2024-03-24T09:20:35.301944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:33.185748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300174+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:20:35.300174+0000","last_peered":"2024-03-24T09:20:35.300174+0000","last_clean":"2024-03-24T09:20:35.300174+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:20:35.300174+0000","last_undegraded":"2024-03-24T09:20:35.300174+0000","last_fullsized":"2024-03-24T09:20:35.300174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300349+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:20:35.300349+0000","last_peered":"2024-03-24T09:20:35.300349+0000","last_clean":"2024-03-24T09:20:35.300349+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:20:35.300349+0000","last_undegraded":"2024-03-24T09:20:35.300349+0000","last_fullsized":"2024-03-24T09:20:35.300349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300220+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:20:35.300220+0000","last_peered":"2024-03-24T09:20:35.300220+0000","last_clean":"2024-03-24T09:20:35.300220+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:20:35.300220+0000","last_undegraded":"2024-03-24T09:20:35.300220+0000","last_fullsized":"2024-03-24T09:20:35.300220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026600+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:20:35.026600+0000","last_peered":"2024-03-24T09:20:35.026600+0000","last_clean":"2024-03-24T09:20:35.026600+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:20:35.026600+0000","last_undegraded":"2024-03-24T09:20:35.026600+0000","last_fullsized":"2024-03-24T09:20:35.026600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027423+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:20:35.027423+0000","last_peered":"2024-03-24T09:20:35.027423+0000","last_clean":"2024-03-24T09:20:35.027423+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:20:35.027423+0000","last_undegraded":"2024-03-24T09:20:35.027423+0000","last_fullsized":"2024-03-24T09:20:35.027423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023388+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:20:35.023388+0000","last_peered":"2024-03-24T09:20:35.023388+0000","last_clean":"2024-03-24T09:20:35.023388+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:20:35.023388+0000","last_undegraded":"2024-03-24T09:20:35.023388+0000","last_fullsized":"2024-03-24T09:20:35.023388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300227+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:20:35.300227+0000","last_peered":"2024-03-24T09:20:35.300227+0000","last_clean":"2024-03-24T09:20:35.300227+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:20:35.300227+0000","last_undegraded":"2024-03-24T09:20:35.300227+0000","last_fullsized":"2024-03-24T09:20:35.300227+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:16.009432+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:21:16.009432+0000","last_peered":"2024-03-24T09:21:16.009432+0000","last_clean":"2024-03-24T09:21:16.009432+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:21:16.009432+0000","last_undegraded":"2024-03-24T09:21:16.009432+0000","last_fullsized":"2024-03-24T09:21:16.009432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025309+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:20:35.025309+0000","last_peered":"2024-03-24T09:20:35.025309+0000","last_clean":"2024-03-24T09:20:35.025309+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:20:35.025309+0000","last_undegraded":"2024-03-24T09:20:35.025309+0000","last_fullsized":"2024-03-24T09:20:35.025309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023264+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:20:35.023264+0000","last_peered":"2024-03-24T09:20:35.023264+0000","last_clean":"2024-03-24T09:20:35.023264+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:20:35.023264+0000","last_undegraded":"2024-03-24T09:20:35.023264+0000","last_fullsized":"2024-03-24T09:20:35.023264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030509+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:20:35.030509+0000","last_peered":"2024-03-24T09:20:35.030509+0000","last_clean":"2024-03-24T09:20:35.030509+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:20:35.030509+0000","last_undegraded":"2024-03-24T09:20:35.030509+0000","last_fullsized":"2024-03-24T09:20:35.030509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301994+0000","last_change":"2024-03-24T09:20:35.301994+0000","last_active":"2024-03-24T09:20:35.301994+0000","last_peered":"2024-03-24T09:20:35.301994+0000","last_clean":"2024-03-24T09:20:35.301994+0000","last_became_active":"2024-03-24T09:20:35.301570+0000","last_became_peered":"2024-03-24T09:20:35.301570+0000","last_unstale":"2024-03-24T09:20:35.301994+0000","last_undegraded":"2024-03-24T09:20:35.301994+0000","last_fullsized":"2024-03-24T09:20:35.301994+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:34.896090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026182+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:20:35.026182+0000","last_peered":"2024-03-24T09:20:35.026182+0000","last_clean":"2024-03-24T09:20:35.026182+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:20:35.026182+0000","last_undegraded":"2024-03-24T09:20:35.026182+0000","last_fullsized":"2024-03-24T09:20:35.026182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028319+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:20:35.028319+0000","last_peered":"2024-03-24T09:20:35.028319+0000","last_clean":"2024-03-24T09:20:35.028319+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:20:35.028319+0000","last_undegraded":"2024-03-24T09:20:35.028319+0000","last_fullsized":"2024-03-24T09:20:35.028319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029821+0000","last_change":"2024-03-24T09:20:35.027718+0000","last_active":"2024-03-24T09:20:35.029821+0000","last_peered":"2024-03-24T09:20:35.029821+0000","last_clean":"2024-03-24T09:20:35.029821+0000","last_became_active":"2024-03-24T09:20:35.027347+0000","last_became_peered":"2024-03-24T09:20:35.027347+0000","last_unstale":"2024-03-24T09:20:35.029821+0000","last_undegraded":"2024-03-24T09:20:35.029821+0000","last_fullsized":"2024-03-24T09:20:35.029821+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:20.097882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.302104+0000","last_change":"2024-03-24T09:20:35.301625+0000","last_active":"2024-03-24T09:20:35.302104+0000","last_peered":"2024-03-24T09:20:35.302104+0000","last_clean":"2024-03-24T09:20:35.302104+0000","last_became_active":"2024-03-24T09:20:35.301035+0000","last_became_peered":"2024-03-24T09:20:35.301035+0000","last_unstale":"2024-03-24T09:20:35.302104+0000","last_undegraded":"2024-03-24T09:20:35.302104+0000","last_fullsized":"2024-03-24T09:20:35.302104+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:09.448556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":218,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:16.970870+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:21:16.970870+0000","last_peered":"2024-03-24T09:21:16.970870+0000","last_clean":"2024-03-24T09:21:16.970870+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:21:16.970870+0000","last_undegraded":"2024-03-24T09:21:16.970870+0000","last_fullsized":"2024-03-24T09:21:16.970870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":156,"num_read_kb":140,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025520+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:20:35.025520+0000","last_peered":"2024-03-24T09:20:35.025520+0000","last_clean":"2024-03-24T09:20:35.025520+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:20:35.025520+0000","last_undegraded":"2024-03-24T09:20:35.025520+0000","last_fullsized":"2024-03-24T09:20:35.025520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028663+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:20:35.028663+0000","last_peered":"2024-03-24T09:20:35.028663+0000","last_clean":"2024-03-24T09:20:35.028663+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:20:35.028663+0000","last_undegraded":"2024-03-24T09:20:35.028663+0000","last_fullsized":"2024-03-24T09:20:35.028663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024008+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:20:35.024008+0000","last_peered":"2024-03-24T09:20:35.024008+0000","last_clean":"2024-03-24T09:20:35.024008+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:20:35.024008+0000","last_undegraded":"2024-03-24T09:20:35.024008+0000","last_fullsized":"2024-03-24T09:20:35.024008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300604+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:20:35.300604+0000","last_peered":"2024-03-24T09:20:35.300604+0000","last_clean":"2024-03-24T09:20:35.300604+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:20:35.300604+0000","last_undegraded":"2024-03-24T09:20:35.300604+0000","last_fullsized":"2024-03-24T09:20:35.300604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300183+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:20:35.300183+0000","last_peered":"2024-03-24T09:20:35.300183+0000","last_clean":"2024-03-24T09:20:35.300183+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:20:35.300183+0000","last_undegraded":"2024-03-24T09:20:35.300183+0000","last_fullsized":"2024-03-24T09:20:35.300183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028296+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:20:35.028296+0000","last_peered":"2024-03-24T09:20:35.028296+0000","last_clean":"2024-03-24T09:20:35.028296+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:20:35.028296+0000","last_undegraded":"2024-03-24T09:20:35.028296+0000","last_fullsized":"2024-03-24T09:20:35.028296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029915+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:20:35.029915+0000","last_peered":"2024-03-24T09:20:35.029915+0000","last_clean":"2024-03-24T09:20:35.029915+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:20:35.029915+0000","last_undegraded":"2024-03-24T09:20:35.029915+0000","last_fullsized":"2024-03-24T09:20:35.029915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023855+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:20:35.023855+0000","last_peered":"2024-03-24T09:20:35.023855+0000","last_clean":"2024-03-24T09:20:35.023855+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:20:35.023855+0000","last_undegraded":"2024-03-24T09:20:35.023855+0000","last_fullsized":"2024-03-24T09:20:35.023855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:18.012185+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:21:18.012185+0000","last_peered":"2024-03-24T09:21:18.012185+0000","last_clean":"2024-03-24T09:21:18.012185+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:21:18.012185+0000","last_undegraded":"2024-03-24T09:21:18.012185+0000","last_fullsized":"2024-03-24T09:21:18.012185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.031305+0000","last_change":"2024-03-24T09:20:35.031305+0000","last_active":"2024-03-24T09:20:35.031305+0000","last_peered":"2024-03-24T09:20:35.031305+0000","last_clean":"2024-03-24T09:20:35.031305+0000","last_became_active":"2024-03-24T09:20:35.030952+0000","last_became_peered":"2024-03-24T09:20:35.030952+0000","last_unstale":"2024-03-24T09:20:35.031305+0000","last_undegraded":"2024-03-24T09:20:35.031305+0000","last_fullsized":"2024-03-24T09:20:35.031305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:40.987185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300960+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:20:35.300960+0000","last_peered":"2024-03-24T09:20:35.300960+0000","last_clean":"2024-03-24T09:20:35.300960+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:20:35.300960+0000","last_undegraded":"2024-03-24T09:20:35.300960+0000","last_fullsized":"2024-03-24T09:20:35.300960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025069+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:20:35.025069+0000","last_peered":"2024-03-24T09:20:35.025069+0000","last_clean":"2024-03-24T09:20:35.025069+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:20:35.025069+0000","last_undegraded":"2024-03-24T09:20:35.025069+0000","last_fullsized":"2024-03-24T09:20:35.025069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027110+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:20:35.027110+0000","last_peered":"2024-03-24T09:20:35.027110+0000","last_clean":"2024-03-24T09:20:35.027110+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:20:35.027110+0000","last_undegraded":"2024-03-24T09:20:35.027110+0000","last_fullsized":"2024-03-24T09:20:35.027110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027005+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:20:35.027005+0000","last_peered":"2024-03-24T09:20:35.027005+0000","last_clean":"2024-03-24T09:20:35.027005+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:20:35.027005+0000","last_undegraded":"2024-03-24T09:20:35.027005+0000","last_fullsized":"2024-03-24T09:20:35.027005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025594+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:20:35.025594+0000","last_peered":"2024-03-24T09:20:35.025594+0000","last_clean":"2024-03-24T09:20:35.025594+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:20:35.025594+0000","last_undegraded":"2024-03-24T09:20:35.025594+0000","last_fullsized":"2024-03-24T09:20:35.025594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300314+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:20:35.300314+0000","last_peered":"2024-03-24T09:20:35.300314+0000","last_clean":"2024-03-24T09:20:35.300314+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:20:35.300314+0000","last_undegraded":"2024-03-24T09:20:35.300314+0000","last_fullsized":"2024-03-24T09:20:35.300314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025817+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:20:35.025817+0000","last_peered":"2024-03-24T09:20:35.025817+0000","last_clean":"2024-03-24T09:20:35.025817+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:20:35.025817+0000","last_undegraded":"2024-03-24T09:20:35.025817+0000","last_fullsized":"2024-03-24T09:20:35.025817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023728+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:20:35.023728+0000","last_peered":"2024-03-24T09:20:35.023728+0000","last_clean":"2024-03-24T09:20:35.023728+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:20:35.023728+0000","last_undegraded":"2024-03-24T09:20:35.023728+0000","last_fullsized":"2024-03-24T09:20:35.023728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025704+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:20:35.025704+0000","last_peered":"2024-03-24T09:20:35.025704+0000","last_clean":"2024-03-24T09:20:35.025704+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:20:35.025704+0000","last_undegraded":"2024-03-24T09:20:35.025704+0000","last_fullsized":"2024-03-24T09:20:35.025704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300598+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:20:35.300598+0000","last_peered":"2024-03-24T09:20:35.300598+0000","last_clean":"2024-03-24T09:20:35.300598+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:20:35.300598+0000","last_undegraded":"2024-03-24T09:20:35.300598+0000","last_fullsized":"2024-03-24T09:20:35.300598+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:17.030030+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:21:17.030030+0000","last_peered":"2024-03-24T09:21:17.030030+0000","last_clean":"2024-03-24T09:21:17.030030+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:21:17.030030+0000","last_undegraded":"2024-03-24T09:21:17.030030+0000","last_fullsized":"2024-03-24T09:21:17.030030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025601+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:20:35.025601+0000","last_peered":"2024-03-24T09:20:35.025601+0000","last_clean":"2024-03-24T09:20:35.025601+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:20:35.025601+0000","last_undegraded":"2024-03-24T09:20:35.025601+0000","last_fullsized":"2024-03-24T09:20:35.025601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300873+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:20:35.300873+0000","last_peered":"2024-03-24T09:20:35.300873+0000","last_clean":"2024-03-24T09:20:35.300873+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:20:35.300873+0000","last_undegraded":"2024-03-24T09:20:35.300873+0000","last_fullsized":"2024-03-24T09:20:35.300873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028030+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:20:35.028030+0000","last_peered":"2024-03-24T09:20:35.028030+0000","last_clean":"2024-03-24T09:20:35.028030+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:20:35.028030+0000","last_undegraded":"2024-03-24T09:20:35.028030+0000","last_fullsized":"2024-03-24T09:20:35.028030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:17.010541+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:21:17.010541+0000","last_peered":"2024-03-24T09:21:17.010541+0000","last_clean":"2024-03-24T09:21:17.010541+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:21:17.010541+0000","last_undegraded":"2024-03-24T09:21:17.010541+0000","last_fullsized":"2024-03-24T09:21:17.010541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030605+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:20:35.030605+0000","last_peered":"2024-03-24T09:20:35.030605+0000","last_clean":"2024-03-24T09:20:35.030605+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:20:35.030605+0000","last_undegraded":"2024-03-24T09:20:35.030605+0000","last_fullsized":"2024-03-24T09:20:35.030605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301245+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:20:35.301245+0000","last_peered":"2024-03-24T09:20:35.301245+0000","last_clean":"2024-03-24T09:20:35.301245+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:20:35.301245+0000","last_undegraded":"2024-03-24T09:20:35.301245+0000","last_fullsized":"2024-03-24T09:20:35.301245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026059+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:20:35.026059+0000","last_peered":"2024-03-24T09:20:35.026059+0000","last_clean":"2024-03-24T09:20:35.026059+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:20:35.026059+0000","last_undegraded":"2024-03-24T09:20:35.026059+0000","last_fullsized":"2024-03-24T09:20:35.026059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028757+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:20:35.028757+0000","last_peered":"2024-03-24T09:20:35.028757+0000","last_clean":"2024-03-24T09:20:35.028757+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:20:35.028757+0000","last_undegraded":"2024-03-24T09:20:35.028757+0000","last_fullsized":"2024-03-24T09:20:35.028757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027181+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:20:35.027181+0000","last_peered":"2024-03-24T09:20:35.027181+0000","last_clean":"2024-03-24T09:20:35.027181+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:20:35.027181+0000","last_undegraded":"2024-03-24T09:20:35.027181+0000","last_fullsized":"2024-03-24T09:20:35.027181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300209+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:20:35.300209+0000","last_peered":"2024-03-24T09:20:35.300209+0000","last_clean":"2024-03-24T09:20:35.300209+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:20:35.300209+0000","last_undegraded":"2024-03-24T09:20:35.300209+0000","last_fullsized":"2024-03-24T09:20:35.300209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028886+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:20:35.028886+0000","last_peered":"2024-03-24T09:20:35.028886+0000","last_clean":"2024-03-24T09:20:35.028886+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:20:35.028886+0000","last_undegraded":"2024-03-24T09:20:35.028886+0000","last_fullsized":"2024-03-24T09:20:35.028886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300131+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:20:35.300131+0000","last_peered":"2024-03-24T09:20:35.300131+0000","last_clean":"2024-03-24T09:20:35.300131+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:20:35.300131+0000","last_undegraded":"2024-03-24T09:20:35.300131+0000","last_fullsized":"2024-03-24T09:20:35.300131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025678+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:20:35.025678+0000","last_peered":"2024-03-24T09:20:35.025678+0000","last_clean":"2024-03-24T09:20:35.025678+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:20:35.025678+0000","last_undegraded":"2024-03-24T09:20:35.025678+0000","last_fullsized":"2024-03-24T09:20:35.025678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300691+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:20:35.300691+0000","last_peered":"2024-03-24T09:20:35.300691+0000","last_clean":"2024-03-24T09:20:35.300691+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:20:35.300691+0000","last_undegraded":"2024-03-24T09:20:35.300691+0000","last_fullsized":"2024-03-24T09:20:35.300691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300434+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:20:35.300434+0000","last_peered":"2024-03-24T09:20:35.300434+0000","last_clean":"2024-03-24T09:20:35.300434+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:20:35.300434+0000","last_undegraded":"2024-03-24T09:20:35.300434+0000","last_fullsized":"2024-03-24T09:20:35.300434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023847+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:20:35.023847+0000","last_peered":"2024-03-24T09:20:35.023847+0000","last_clean":"2024-03-24T09:20:35.023847+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:20:35.023847+0000","last_undegraded":"2024-03-24T09:20:35.023847+0000","last_fullsized":"2024-03-24T09:20:35.023847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301502+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:20:35.301502+0000","last_peered":"2024-03-24T09:20:35.301502+0000","last_clean":"2024-03-24T09:20:35.301502+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:20:35.301502+0000","last_undegraded":"2024-03-24T09:20:35.301502+0000","last_fullsized":"2024-03-24T09:20:35.301502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023569+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:20:35.023569+0000","last_peered":"2024-03-24T09:20:35.023569+0000","last_clean":"2024-03-24T09:20:35.023569+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:20:35.023569+0000","last_undegraded":"2024-03-24T09:20:35.023569+0000","last_fullsized":"2024-03-24T09:20:35.023569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026098+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:20:35.026098+0000","last_peered":"2024-03-24T09:20:35.026098+0000","last_clean":"2024-03-24T09:20:35.026098+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:20:35.026098+0000","last_undegraded":"2024-03-24T09:20:35.026098+0000","last_fullsized":"2024-03-24T09:20:35.026098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026645+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:20:35.026645+0000","last_peered":"2024-03-24T09:20:35.026645+0000","last_clean":"2024-03-24T09:20:35.026645+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:20:35.026645+0000","last_undegraded":"2024-03-24T09:20:35.026645+0000","last_fullsized":"2024-03-24T09:20:35.026645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026005+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:20:35.026005+0000","last_peered":"2024-03-24T09:20:35.026005+0000","last_clean":"2024-03-24T09:20:35.026005+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:20:35.026005+0000","last_undegraded":"2024-03-24T09:20:35.026005+0000","last_fullsized":"2024-03-24T09:20:35.026005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024428+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:20:35.024428+0000","last_peered":"2024-03-24T09:20:35.024428+0000","last_clean":"2024-03-24T09:20:35.024428+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:20:35.024428+0000","last_undegraded":"2024-03-24T09:20:35.024428+0000","last_fullsized":"2024-03-24T09:20:35.024428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024965+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:20:35.024965+0000","last_peered":"2024-03-24T09:20:35.024965+0000","last_clean":"2024-03-24T09:20:35.024965+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:20:35.024965+0000","last_undegraded":"2024-03-24T09:20:35.024965+0000","last_fullsized":"2024-03-24T09:20:35.024965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023972+0000","last_change":"2024-03-24T09:19:42.743462+0000","last_active":"2024-03-24T09:20:35.023972+0000","last_peered":"2024-03-24T09:20:35.023972+0000","last_clean":"2024-03-24T09:20:35.023972+0000","last_became_active":"2024-03-24T09:19:42.743078+0000","last_became_peered":"2024-03-24T09:19:42.743078+0000","last_unstale":"2024-03-24T09:20:35.023972+0000","last_undegraded":"2024-03-24T09:20:35.023972+0000","last_fullsized":"2024-03-24T09:20:35.023972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:49.958507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300051+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:20:35.300051+0000","last_peered":"2024-03-24T09:20:35.300051+0000","last_clean":"2024-03-24T09:20:35.300051+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:20:35.300051+0000","last_undegraded":"2024-03-24T09:20:35.300051+0000","last_fullsized":"2024-03-24T09:20:35.300051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.299959+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:20:35.299959+0000","last_peered":"2024-03-24T09:20:35.299959+0000","last_clean":"2024-03-24T09:20:35.299959+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:20:35.299959+0000","last_undegraded":"2024-03-24T09:20:35.299959+0000","last_fullsized":"2024-03-24T09:20:35.299959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026556+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:20:35.026556+0000","last_peered":"2024-03-24T09:20:35.026556+0000","last_clean":"2024-03-24T09:20:35.026556+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:20:35.026556+0000","last_undegraded":"2024-03-24T09:20:35.026556+0000","last_fullsized":"2024-03-24T09:20:35.026556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023812+0000","last_change":"2024-03-24T09:19:38.100940+0000","last_active":"2024-03-24T09:20:35.023812+0000","last_peered":"2024-03-24T09:20:35.023812+0000","last_clean":"2024-03-24T09:20:35.023812+0000","last_became_active":"2024-03-24T09:19:38.100613+0000","last_became_peered":"2024-03-24T09:19:38.100613+0000","last_unstale":"2024-03-24T09:20:35.023812+0000","last_undegraded":"2024-03-24T09:20:35.023812+0000","last_fullsized":"2024-03-24T09:20:35.023812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:53.483332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300739+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:20:35.300739+0000","last_peered":"2024-03-24T09:20:35.300739+0000","last_clean":"2024-03-24T09:20:35.300739+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:20:35.300739+0000","last_undegraded":"2024-03-24T09:20:35.300739+0000","last_fullsized":"2024-03-24T09:20:35.300739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025812+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:20:35.025812+0000","last_peered":"2024-03-24T09:20:35.025812+0000","last_clean":"2024-03-24T09:20:35.025812+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:20:35.025812+0000","last_undegraded":"2024-03-24T09:20:35.025812+0000","last_fullsized":"2024-03-24T09:20:35.025812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025373+0000","last_change":"2024-03-24T09:19:38.105336+0000","last_active":"2024-03-24T09:20:35.025373+0000","last_peered":"2024-03-24T09:20:35.025373+0000","last_clean":"2024-03-24T09:20:35.025373+0000","last_became_active":"2024-03-24T09:19:38.103695+0000","last_became_peered":"2024-03-24T09:19:38.103695+0000","last_unstale":"2024-03-24T09:20:35.025373+0000","last_undegraded":"2024-03-24T09:20:35.025373+0000","last_fullsized":"2024-03-24T09:20:35.025373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:11:41.784391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029356+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:20:35.029356+0000","last_peered":"2024-03-24T09:20:35.029356+0000","last_clean":"2024-03-24T09:20:35.029356+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:20:35.029356+0000","last_undegraded":"2024-03-24T09:20:35.029356+0000","last_fullsized":"2024-03-24T09:20:35.029356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024741+0000","last_change":"2024-03-24T09:19:42.744163+0000","last_active":"2024-03-24T09:20:35.024741+0000","last_peered":"2024-03-24T09:20:35.024741+0000","last_clean":"2024-03-24T09:20:35.024741+0000","last_became_active":"2024-03-24T09:19:42.743754+0000","last_became_peered":"2024-03-24T09:19:42.743754+0000","last_unstale":"2024-03-24T09:20:35.024741+0000","last_undegraded":"2024-03-24T09:20:35.024741+0000","last_fullsized":"2024-03-24T09:20:35.024741+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:07.082490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:18.012142+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:21:18.012142+0000","last_peered":"2024-03-24T09:21:18.012142+0000","last_clean":"2024-03-24T09:21:18.012142+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:21:18.012142+0000","last_undegraded":"2024-03-24T09:21:18.012142+0000","last_fullsized":"2024-03-24T09:21:18.012142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029915+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:20:35.029915+0000","last_peered":"2024-03-24T09:20:35.029915+0000","last_clean":"2024-03-24T09:20:35.029915+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:20:35.029915+0000","last_undegraded":"2024-03-24T09:20:35.029915+0000","last_fullsized":"2024-03-24T09:20:35.029915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026374+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:20:35.026374+0000","last_peered":"2024-03-24T09:20:35.026374+0000","last_clean":"2024-03-24T09:20:35.026374+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:20:35.026374+0000","last_undegraded":"2024-03-24T09:20:35.026374+0000","last_fullsized":"2024-03-24T09:20:35.026374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028460+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:20:35.028460+0000","last_peered":"2024-03-24T09:20:35.028460+0000","last_clean":"2024-03-24T09:20:35.028460+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:20:35.028460+0000","last_undegraded":"2024-03-24T09:20:35.028460+0000","last_fullsized":"2024-03-24T09:20:35.028460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.031333+0000","last_change":"2024-03-24T09:20:35.031333+0000","last_active":"2024-03-24T09:20:35.031333+0000","last_peered":"2024-03-24T09:20:35.031333+0000","last_clean":"2024-03-24T09:20:35.031333+0000","last_became_active":"2024-03-24T09:20:35.030995+0000","last_became_peered":"2024-03-24T09:20:35.030995+0000","last_unstale":"2024-03-24T09:20:35.031333+0000","last_undegraded":"2024-03-24T09:20:35.031333+0000","last_fullsized":"2024-03-24T09:20:35.031333+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:05.746740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026790+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:20:35.026790+0000","last_peered":"2024-03-24T09:20:35.026790+0000","last_clean":"2024-03-24T09:20:35.026790+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:20:35.026790+0000","last_undegraded":"2024-03-24T09:20:35.026790+0000","last_fullsized":"2024-03-24T09:20:35.026790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300874+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:20:35.300874+0000","last_peered":"2024-03-24T09:20:35.300874+0000","last_clean":"2024-03-24T09:20:35.300874+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:20:35.300874+0000","last_undegraded":"2024-03-24T09:20:35.300874+0000","last_fullsized":"2024-03-24T09:20:35.300874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030633+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:20:35.030633+0000","last_peered":"2024-03-24T09:20:35.030633+0000","last_clean":"2024-03-24T09:20:35.030633+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:20:35.030633+0000","last_undegraded":"2024-03-24T09:20:35.030633+0000","last_fullsized":"2024-03-24T09:20:35.030633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300002+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:20:35.300002+0000","last_peered":"2024-03-24T09:20:35.300002+0000","last_clean":"2024-03-24T09:20:35.300002+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:20:35.300002+0000","last_undegraded":"2024-03-24T09:20:35.300002+0000","last_fullsized":"2024-03-24T09:20:35.300002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023873+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:20:35.023873+0000","last_peered":"2024-03-24T09:20:35.023873+0000","last_clean":"2024-03-24T09:20:35.023873+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:20:35.023873+0000","last_undegraded":"2024-03-24T09:20:35.023873+0000","last_fullsized":"2024-03-24T09:20:35.023873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023785+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:20:35.023785+0000","last_peered":"2024-03-24T09:20:35.023785+0000","last_clean":"2024-03-24T09:20:35.023785+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:20:35.023785+0000","last_undegraded":"2024-03-24T09:20:35.023785+0000","last_fullsized":"2024-03-24T09:20:35.023785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026663+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:20:35.026663+0000","last_peered":"2024-03-24T09:20:35.026663+0000","last_clean":"2024-03-24T09:20:35.026663+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:20:35.026663+0000","last_undegraded":"2024-03-24T09:20:35.026663+0000","last_fullsized":"2024-03-24T09:20:35.026663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300709+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:20:35.300709+0000","last_peered":"2024-03-24T09:20:35.300709+0000","last_clean":"2024-03-24T09:20:35.300709+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:20:35.300709+0000","last_undegraded":"2024-03-24T09:20:35.300709+0000","last_fullsized":"2024-03-24T09:20:35.300709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300370+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:20:35.300370+0000","last_peered":"2024-03-24T09:20:35.300370+0000","last_clean":"2024-03-24T09:20:35.300370+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:20:35.300370+0000","last_undegraded":"2024-03-24T09:20:35.300370+0000","last_fullsized":"2024-03-24T09:20:35.300370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024488+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:20:35.024488+0000","last_peered":"2024-03-24T09:20:35.024488+0000","last_clean":"2024-03-24T09:20:35.024488+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:20:35.024488+0000","last_undegraded":"2024-03-24T09:20:35.024488+0000","last_fullsized":"2024-03-24T09:20:35.024488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301418+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:20:35.301418+0000","last_peered":"2024-03-24T09:20:35.301418+0000","last_clean":"2024-03-24T09:20:35.301418+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:20:35.301418+0000","last_undegraded":"2024-03-24T09:20:35.301418+0000","last_fullsized":"2024-03-24T09:20:35.301418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027522+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:20:35.027522+0000","last_peered":"2024-03-24T09:20:35.027522+0000","last_clean":"2024-03-24T09:20:35.027522+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:20:35.027522+0000","last_undegraded":"2024-03-24T09:20:35.027522+0000","last_fullsized":"2024-03-24T09:20:35.027522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.299996+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:20:35.299996+0000","last_peered":"2024-03-24T09:20:35.299996+0000","last_clean":"2024-03-24T09:20:35.299996+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:20:35.299996+0000","last_undegraded":"2024-03-24T09:20:35.299996+0000","last_fullsized":"2024-03-24T09:20:35.299996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025002+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:20:35.025002+0000","last_peered":"2024-03-24T09:20:35.025002+0000","last_clean":"2024-03-24T09:20:35.025002+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:20:35.025002+0000","last_undegraded":"2024-03-24T09:20:35.025002+0000","last_fullsized":"2024-03-24T09:20:35.025002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300252+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:20:35.300252+0000","last_peered":"2024-03-24T09:20:35.300252+0000","last_clean":"2024-03-24T09:20:35.300252+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:20:35.300252+0000","last_undegraded":"2024-03-24T09:20:35.300252+0000","last_fullsized":"2024-03-24T09:20:35.300252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300177+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:20:35.300177+0000","last_peered":"2024-03-24T09:20:35.300177+0000","last_clean":"2024-03-24T09:20:35.300177+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:20:35.300177+0000","last_undegraded":"2024-03-24T09:20:35.300177+0000","last_fullsized":"2024-03-24T09:20:35.300177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280427,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70560,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":766,"num_read_kb":456,"num_write":242,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":211680,"data_compressed":4809,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501433,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1151996,"data_compressed":57942,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":501741,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1077248,"data_stored":1098629,"data_compressed":56339,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":561253,"data_compressed":22274,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":561864,"data_compressed":22274,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":501332,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":1093820,"data_compressed":56339,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65108,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65719,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65483,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:21:20.947 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T09:21:21.190 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:21 smithi143 ceph-mon[35965]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:21.311 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:21 smithi116 ceph-mon[32207]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:21 smithi117 ceph-mon[35788]: pgmap v344: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:22.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:22 smithi143 ceph-mon[35965]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:21:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:22 smithi117 ceph-mon[35788]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:21:22.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:22 smithi116 ceph-mon[32207]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 -- 172.21.15.116:0/51701262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102c10 msgr2=0x7ffae410f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 --2- 172.21.15.116:0/51701262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102c10 0x7ffae410f050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00b0d0 tx=0x7ffadc00b3e0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 -- 172.21.15.116:0/51701262 shutdown_connections 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 --2- 172.21.15.116:0/51701262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102c10 0x7ffae410f050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 --2- 172.21.15.116:0/51701262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffae4108230 0x7ffae4108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 --2- 172.21.15.116:0/51701262 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102250 0x7ffae41026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 -- 172.21.15.116:0/51701262 >> 172.21.15.116:0/51701262 conn(0x7ffae4079330 msgr2=0x7ffae4079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 -- 172.21.15.116:0/51701262 shutdown_connections 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 -- 172.21.15.116:0/51701262 wait complete. 2024-03-24T09:21:23.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.025+0000 7ffae9836700 1 Processor -- start 2024-03-24T09:21:23.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 -- start start 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 0x7ffae4198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102c10 0x7ffae4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffae4108230 0x7ffae419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffae419d610 con 0x7ffae4102250 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffae419d780 con 0x7ffae4102c10 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae9836700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffae419d940 con 0x7ffae4108230 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 0x7ffae4198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 0x7ffae4198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44276/0 (socket says 172.21.15.116:44276) 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae2ffd700 1 -- 172.21.15.116:0/4098538345 learned_addr learned my addr 172.21.15.116:0/4098538345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae27fc700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102c10 0x7ffae4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:23.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae37fe700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffae4108230 0x7ffae419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:23.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:23 smithi116 ceph-mon[32207]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae27fc700 1 -- 172.21.15.116:0/4098538345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffae4108230 msgr2=0x7ffae419d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae27fc700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffae4108230 0x7ffae419d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae27fc700 1 -- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 msgr2=0x7ffae4198730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae27fc700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 0x7ffae4198730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae27fc700 1 -- 172.21.15.116:0/4098538345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffadc009d00 con 0x7ffae4102c10 2024-03-24T09:21:23.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.026+0000 7ffae2ffd700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 0x7ffae4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.027+0000 7ffae27fc700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102c10 0x7ffae4198c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffad800e9f0 tx=0x7ffad800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.027+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad800c9c0 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.027+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffad8010430 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.027+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae419dc20 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.027+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8018680 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.027+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae4074600 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.028+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae410a710 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.029+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7ffad80105a0 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.029+0000 7ffae8834700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffacc0a21e0 0x7ffacc0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.029+0000 7ffae2ffd700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffacc0a21e0 0x7ffacc0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.029+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ffad8014070 con 0x7ffae4102c10 2024-03-24T09:21:23.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.030+0000 7ffae2ffd700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffacc0a21e0 0x7ffacc0a46a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffad4011550 tx=0x7ffad4009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:23.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.034+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffad8053280 con 0x7ffae4102c10 2024-03-24T09:21:23.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.223+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffae4199a90 con 0x7ffacc0a21e0 2024-03-24T09:21:23.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.232+0000 7ffae8834700 1 -- 172.21.15.116:0/4098538345 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340714 (secure 0 0 0) 0x7ffae4199a90 con 0x7ffacc0a21e0 2024-03-24T09:21:23.234 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffacc0a21e0 msgr2=0x7ffacc0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffacc0a21e0 0x7ffacc0a46a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ffad4011550 tx=0x7ffad4009260 comp rx=0 tx=0).stop 2024-03-24T09:21:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102c10 msgr2=0x7ffae4198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102c10 0x7ffae4198c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffad800e9f0 tx=0x7ffad800ed00 comp rx=0 tx=0).stop 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 shutdown_connections 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffae4108230 0x7ffae419d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ffacc0a21e0 0x7ffacc0a46a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffae4102250 0x7ffae4198730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.235+0000 7ffae9836700 1 --2- 172.21.15.116:0/4098538345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae4102c10 0x7ffae4198c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.236+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 >> 172.21.15.116:0/4098538345 conn(0x7ffae4079330 msgr2=0x7ffae4100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.236+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 shutdown_connections 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:23.236+0000 7ffae9836700 1 -- 172.21.15.116:0/4098538345 wait complete. 2024-03-24T09:21:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:21:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:23 smithi143 ceph-mon[35965]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:23 smithi117 ceph-mon[35788]: pgmap v345: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:24.168 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-03-24T09:21:21.946990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661162,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":907,"num_read_kb":580,"num_write":425,"num_write_kb":3081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":350,"ondisk_log_size":350,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378176,"kb_used_data":12296,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563648,"statfs":{"total":1151940427776,"available":1151553175552,"internally_reserved":0,"allocated":12591104,"data_stored":7969292,"data_compressed":359865,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003800"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300382+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:20:35.300382+0000","last_peered":"2024-03-24T09:20:35.300382+0000","last_clean":"2024-03-24T09:20:35.300382+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:20:35.300382+0000","last_undegraded":"2024-03-24T09:20:35.300382+0000","last_fullsized":"2024-03-24T09:20:35.300382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026713+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:20:35.026713+0000","last_peered":"2024-03-24T09:20:35.026713+0000","last_clean":"2024-03-24T09:20:35.026713+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:20:35.026713+0000","last_undegraded":"2024-03-24T09:20:35.026713+0000","last_fullsized":"2024-03-24T09:20:35.026713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026550+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:20:35.026550+0000","last_peered":"2024-03-24T09:20:35.026550+0000","last_clean":"2024-03-24T09:20:35.026550+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:20:35.026550+0000","last_undegraded":"2024-03-24T09:20:35.026550+0000","last_fullsized":"2024-03-24T09:20:35.026550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026757+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:20:35.026757+0000","last_peered":"2024-03-24T09:20:35.026757+0000","last_clean":"2024-03-24T09:20:35.026757+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:20:35.026757+0000","last_undegraded":"2024-03-24T09:20:35.026757+0000","last_fullsized":"2024-03-24T09:20:35.026757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025270+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:20:35.025270+0000","last_peered":"2024-03-24T09:20:35.025270+0000","last_clean":"2024-03-24T09:20:35.025270+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:20:35.025270+0000","last_undegraded":"2024-03-24T09:20:35.025270+0000","last_fullsized":"2024-03-24T09:20:35.025270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029608+0000","last_change":"2024-03-24T09:20:35.029608+0000","last_active":"2024-03-24T09:20:35.029608+0000","last_peered":"2024-03-24T09:20:35.029608+0000","last_clean":"2024-03-24T09:20:35.029608+0000","last_became_active":"2024-03-24T09:20:35.029268+0000","last_became_peered":"2024-03-24T09:20:35.029268+0000","last_unstale":"2024-03-24T09:20:35.029608+0000","last_undegraded":"2024-03-24T09:20:35.029608+0000","last_fullsized":"2024-03-24T09:20:35.029608+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:47.087125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028091+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:20:35.028091+0000","last_peered":"2024-03-24T09:20:35.028091+0000","last_clean":"2024-03-24T09:20:35.028091+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:20:35.028091+0000","last_undegraded":"2024-03-24T09:20:35.028091+0000","last_fullsized":"2024-03-24T09:20:35.028091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300114+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:20:35.300114+0000","last_peered":"2024-03-24T09:20:35.300114+0000","last_clean":"2024-03-24T09:20:35.300114+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:20:35.300114+0000","last_undegraded":"2024-03-24T09:20:35.300114+0000","last_fullsized":"2024-03-24T09:20:35.300114+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025345+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:20:35.025345+0000","last_peered":"2024-03-24T09:20:35.025345+0000","last_clean":"2024-03-24T09:20:35.025345+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:20:35.025345+0000","last_undegraded":"2024-03-24T09:20:35.025345+0000","last_fullsized":"2024-03-24T09:20:35.025345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026663+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:20:35.026663+0000","last_peered":"2024-03-24T09:20:35.026663+0000","last_clean":"2024-03-24T09:20:35.026663+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:20:35.026663+0000","last_undegraded":"2024-03-24T09:20:35.026663+0000","last_fullsized":"2024-03-24T09:20:35.026663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030438+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:20:35.030438+0000","last_peered":"2024-03-24T09:20:35.030438+0000","last_clean":"2024-03-24T09:20:35.030438+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:20:35.030438+0000","last_undegraded":"2024-03-24T09:20:35.030438+0000","last_fullsized":"2024-03-24T09:20:35.030438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026667+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:20:35.026667+0000","last_peered":"2024-03-24T09:20:35.026667+0000","last_clean":"2024-03-24T09:20:35.026667+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:20:35.026667+0000","last_undegraded":"2024-03-24T09:20:35.026667+0000","last_fullsized":"2024-03-24T09:20:35.026667+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300347+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:20:35.300347+0000","last_peered":"2024-03-24T09:20:35.300347+0000","last_clean":"2024-03-24T09:20:35.300347+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:20:35.300347+0000","last_undegraded":"2024-03-24T09:20:35.300347+0000","last_fullsized":"2024-03-24T09:20:35.300347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026382+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:20:35.026382+0000","last_peered":"2024-03-24T09:20:35.026382+0000","last_clean":"2024-03-24T09:20:35.026382+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:20:35.026382+0000","last_undegraded":"2024-03-24T09:20:35.026382+0000","last_fullsized":"2024-03-24T09:20:35.026382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026741+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:20:35.026741+0000","last_peered":"2024-03-24T09:20:35.026741+0000","last_clean":"2024-03-24T09:20:35.026741+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:20:35.026741+0000","last_undegraded":"2024-03-24T09:20:35.026741+0000","last_fullsized":"2024-03-24T09:20:35.026741+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024361+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:20:35.024361+0000","last_peered":"2024-03-24T09:20:35.024361+0000","last_clean":"2024-03-24T09:20:35.024361+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:20:35.024361+0000","last_undegraded":"2024-03-24T09:20:35.024361+0000","last_fullsized":"2024-03-24T09:20:35.024361+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300199+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:20:35.300199+0000","last_peered":"2024-03-24T09:20:35.300199+0000","last_clean":"2024-03-24T09:20:35.300199+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:20:35.300199+0000","last_undegraded":"2024-03-24T09:20:35.300199+0000","last_fullsized":"2024-03-24T09:20:35.300199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023848+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:20:35.023848+0000","last_peered":"2024-03-24T09:20:35.023848+0000","last_clean":"2024-03-24T09:20:35.023848+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:20:35.023848+0000","last_undegraded":"2024-03-24T09:20:35.023848+0000","last_fullsized":"2024-03-24T09:20:35.023848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024297+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:20:35.024297+0000","last_peered":"2024-03-24T09:20:35.024297+0000","last_clean":"2024-03-24T09:20:35.024297+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:20:35.024297+0000","last_undegraded":"2024-03-24T09:20:35.024297+0000","last_fullsized":"2024-03-24T09:20:35.024297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023771+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:20:35.023771+0000","last_peered":"2024-03-24T09:20:35.023771+0000","last_clean":"2024-03-24T09:20:35.023771+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:20:35.023771+0000","last_undegraded":"2024-03-24T09:20:35.023771+0000","last_fullsized":"2024-03-24T09:20:35.023771+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:19.031666+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:21:19.031666+0000","last_peered":"2024-03-24T09:21:19.031666+0000","last_clean":"2024-03-24T09:21:19.031666+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:21:19.031666+0000","last_undegraded":"2024-03-24T09:21:19.031666+0000","last_fullsized":"2024-03-24T09:21:19.031666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301400+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:20:35.301400+0000","last_peered":"2024-03-24T09:20:35.301400+0000","last_clean":"2024-03-24T09:20:35.301400+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:20:35.301400+0000","last_undegraded":"2024-03-24T09:20:35.301400+0000","last_fullsized":"2024-03-24T09:20:35.301400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.302025+0000","last_change":"2024-03-24T09:20:35.302025+0000","last_active":"2024-03-24T09:20:35.302025+0000","last_peered":"2024-03-24T09:20:35.302025+0000","last_clean":"2024-03-24T09:20:35.302025+0000","last_became_active":"2024-03-24T09:20:35.301575+0000","last_became_peered":"2024-03-24T09:20:35.301575+0000","last_unstale":"2024-03-24T09:20:35.302025+0000","last_undegraded":"2024-03-24T09:20:35.302025+0000","last_fullsized":"2024-03-24T09:20:35.302025+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:20.305175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301569+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:20:35.301569+0000","last_peered":"2024-03-24T09:20:35.301569+0000","last_clean":"2024-03-24T09:20:35.301569+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:20:35.301569+0000","last_undegraded":"2024-03-24T09:20:35.301569+0000","last_fullsized":"2024-03-24T09:20:35.301569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026514+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:20:35.026514+0000","last_peered":"2024-03-24T09:20:35.026514+0000","last_clean":"2024-03-24T09:20:35.026514+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:20:35.026514+0000","last_undegraded":"2024-03-24T09:20:35.026514+0000","last_fullsized":"2024-03-24T09:20:35.026514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025747+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:20:35.025747+0000","last_peered":"2024-03-24T09:20:35.025747+0000","last_clean":"2024-03-24T09:20:35.025747+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:20:35.025747+0000","last_undegraded":"2024-03-24T09:20:35.025747+0000","last_fullsized":"2024-03-24T09:20:35.025747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025746+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:20:35.025746+0000","last_peered":"2024-03-24T09:20:35.025746+0000","last_clean":"2024-03-24T09:20:35.025746+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:20:35.025746+0000","last_undegraded":"2024-03-24T09:20:35.025746+0000","last_fullsized":"2024-03-24T09:20:35.025746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024995+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:20:35.024995+0000","last_peered":"2024-03-24T09:20:35.024995+0000","last_clean":"2024-03-24T09:20:35.024995+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:20:35.024995+0000","last_undegraded":"2024-03-24T09:20:35.024995+0000","last_fullsized":"2024-03-24T09:20:35.024995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025428+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:20:35.025428+0000","last_peered":"2024-03-24T09:20:35.025428+0000","last_clean":"2024-03-24T09:20:35.025428+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:20:35.025428+0000","last_undegraded":"2024-03-24T09:20:35.025428+0000","last_fullsized":"2024-03-24T09:20:35.025428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025318+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:20:35.025318+0000","last_peered":"2024-03-24T09:20:35.025318+0000","last_clean":"2024-03-24T09:20:35.025318+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:20:35.025318+0000","last_undegraded":"2024-03-24T09:20:35.025318+0000","last_fullsized":"2024-03-24T09:20:35.025318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.032271+0000","last_change":"2024-03-24T09:20:35.032271+0000","last_active":"2024-03-24T09:20:35.032271+0000","last_peered":"2024-03-24T09:20:35.032271+0000","last_clean":"2024-03-24T09:20:35.032271+0000","last_became_active":"2024-03-24T09:20:35.031913+0000","last_became_peered":"2024-03-24T09:20:35.031913+0000","last_unstale":"2024-03-24T09:20:35.032271+0000","last_undegraded":"2024-03-24T09:20:35.032271+0000","last_fullsized":"2024-03-24T09:20:35.032271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:40.322197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024135+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:20:35.024135+0000","last_peered":"2024-03-24T09:20:35.024135+0000","last_clean":"2024-03-24T09:20:35.024135+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:20:35.024135+0000","last_undegraded":"2024-03-24T09:20:35.024135+0000","last_fullsized":"2024-03-24T09:20:35.024135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:17.972407+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:21:17.972407+0000","last_peered":"2024-03-24T09:21:17.972407+0000","last_clean":"2024-03-24T09:21:17.972407+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:21:17.972407+0000","last_undegraded":"2024-03-24T09:21:17.972407+0000","last_fullsized":"2024-03-24T09:21:17.972407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.031176+0000","last_change":"2024-03-24T09:20:35.031176+0000","last_active":"2024-03-24T09:20:35.031176+0000","last_peered":"2024-03-24T09:20:35.031176+0000","last_clean":"2024-03-24T09:20:35.031176+0000","last_became_active":"2024-03-24T09:20:35.030888+0000","last_became_peered":"2024-03-24T09:20:35.030888+0000","last_unstale":"2024-03-24T09:20:35.031176+0000","last_undegraded":"2024-03-24T09:20:35.031176+0000","last_fullsized":"2024-03-24T09:20:35.031176+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:43.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024380+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:20:35.024380+0000","last_peered":"2024-03-24T09:20:35.024380+0000","last_clean":"2024-03-24T09:20:35.024380+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:20:35.024380+0000","last_undegraded":"2024-03-24T09:20:35.024380+0000","last_fullsized":"2024-03-24T09:20:35.024380+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027076+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:20:35.027076+0000","last_peered":"2024-03-24T09:20:35.027076+0000","last_clean":"2024-03-24T09:20:35.027076+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:20:35.027076+0000","last_undegraded":"2024-03-24T09:20:35.027076+0000","last_fullsized":"2024-03-24T09:20:35.027076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301944+0000","last_change":"2024-03-24T09:20:35.301944+0000","last_active":"2024-03-24T09:20:35.301944+0000","last_peered":"2024-03-24T09:20:35.301944+0000","last_clean":"2024-03-24T09:20:35.301944+0000","last_became_active":"2024-03-24T09:20:35.301588+0000","last_became_peered":"2024-03-24T09:20:35.301588+0000","last_unstale":"2024-03-24T09:20:35.301944+0000","last_undegraded":"2024-03-24T09:20:35.301944+0000","last_fullsized":"2024-03-24T09:20:35.301944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:33.185748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300174+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:20:35.300174+0000","last_peered":"2024-03-24T09:20:35.300174+0000","last_clean":"2024-03-24T09:20:35.300174+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:20:35.300174+0000","last_undegraded":"2024-03-24T09:20:35.300174+0000","last_fullsized":"2024-03-24T09:20:35.300174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300349+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:20:35.300349+0000","last_peered":"2024-03-24T09:20:35.300349+0000","last_clean":"2024-03-24T09:20:35.300349+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:20:35.300349+0000","last_undegraded":"2024-03-24T09:20:35.300349+0000","last_fullsized":"2024-03-24T09:20:35.300349+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300220+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:20:35.300220+0000","last_peered":"2024-03-24T09:20:35.300220+0000","last_clean":"2024-03-24T09:20:35.300220+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:20:35.300220+0000","last_undegraded":"2024-03-24T09:20:35.300220+0000","last_fullsized":"2024-03-24T09:20:35.300220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026600+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:20:35.026600+0000","last_peered":"2024-03-24T09:20:35.026600+0000","last_clean":"2024-03-24T09:20:35.026600+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:20:35.026600+0000","last_undegraded":"2024-03-24T09:20:35.026600+0000","last_fullsized":"2024-03-24T09:20:35.026600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027423+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:20:35.027423+0000","last_peered":"2024-03-24T09:20:35.027423+0000","last_clean":"2024-03-24T09:20:35.027423+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:20:35.027423+0000","last_undegraded":"2024-03-24T09:20:35.027423+0000","last_fullsized":"2024-03-24T09:20:35.027423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023388+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:20:35.023388+0000","last_peered":"2024-03-24T09:20:35.023388+0000","last_clean":"2024-03-24T09:20:35.023388+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:20:35.023388+0000","last_undegraded":"2024-03-24T09:20:35.023388+0000","last_fullsized":"2024-03-24T09:20:35.023388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300227+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:20:35.300227+0000","last_peered":"2024-03-24T09:20:35.300227+0000","last_clean":"2024-03-24T09:20:35.300227+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:20:35.300227+0000","last_undegraded":"2024-03-24T09:20:35.300227+0000","last_fullsized":"2024-03-24T09:20:35.300227+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:16.009432+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:21:16.009432+0000","last_peered":"2024-03-24T09:21:16.009432+0000","last_clean":"2024-03-24T09:21:16.009432+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:21:16.009432+0000","last_undegraded":"2024-03-24T09:21:16.009432+0000","last_fullsized":"2024-03-24T09:21:16.009432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025309+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:20:35.025309+0000","last_peered":"2024-03-24T09:20:35.025309+0000","last_clean":"2024-03-24T09:20:35.025309+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:20:35.025309+0000","last_undegraded":"2024-03-24T09:20:35.025309+0000","last_fullsized":"2024-03-24T09:20:35.025309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023264+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:20:35.023264+0000","last_peered":"2024-03-24T09:20:35.023264+0000","last_clean":"2024-03-24T09:20:35.023264+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:20:35.023264+0000","last_undegraded":"2024-03-24T09:20:35.023264+0000","last_fullsized":"2024-03-24T09:20:35.023264+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030509+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:20:35.030509+0000","last_peered":"2024-03-24T09:20:35.030509+0000","last_clean":"2024-03-24T09:20:35.030509+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:20:35.030509+0000","last_undegraded":"2024-03-24T09:20:35.030509+0000","last_fullsized":"2024-03-24T09:20:35.030509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301994+0000","last_change":"2024-03-24T09:20:35.301994+0000","last_active":"2024-03-24T09:20:35.301994+0000","last_peered":"2024-03-24T09:20:35.301994+0000","last_clean":"2024-03-24T09:20:35.301994+0000","last_became_active":"2024-03-24T09:20:35.301570+0000","last_became_peered":"2024-03-24T09:20:35.301570+0000","last_unstale":"2024-03-24T09:20:35.301994+0000","last_undegraded":"2024-03-24T09:20:35.301994+0000","last_fullsized":"2024-03-24T09:20:35.301994+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:34.896090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026182+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:20:35.026182+0000","last_peered":"2024-03-24T09:20:35.026182+0000","last_clean":"2024-03-24T09:20:35.026182+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:20:35.026182+0000","last_undegraded":"2024-03-24T09:20:35.026182+0000","last_fullsized":"2024-03-24T09:20:35.026182+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028319+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:20:35.028319+0000","last_peered":"2024-03-24T09:20:35.028319+0000","last_clean":"2024-03-24T09:20:35.028319+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:20:35.028319+0000","last_undegraded":"2024-03-24T09:20:35.028319+0000","last_fullsized":"2024-03-24T09:20:35.028319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029821+0000","last_change":"2024-03-24T09:20:35.027718+0000","last_active":"2024-03-24T09:20:35.029821+0000","last_peered":"2024-03-24T09:20:35.029821+0000","last_clean":"2024-03-24T09:20:35.029821+0000","last_became_active":"2024-03-24T09:20:35.027347+0000","last_became_peered":"2024-03-24T09:20:35.027347+0000","last_unstale":"2024-03-24T09:20:35.029821+0000","last_undegraded":"2024-03-24T09:20:35.029821+0000","last_fullsized":"2024-03-24T09:20:35.029821+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:20.097882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.302104+0000","last_change":"2024-03-24T09:20:35.301625+0000","last_active":"2024-03-24T09:20:35.302104+0000","last_peered":"2024-03-24T09:20:35.302104+0000","last_clean":"2024-03-24T09:20:35.302104+0000","last_became_active":"2024-03-24T09:20:35.301035+0000","last_became_peered":"2024-03-24T09:20:35.301035+0000","last_unstale":"2024-03-24T09:20:35.302104+0000","last_undegraded":"2024-03-24T09:20:35.302104+0000","last_fullsized":"2024-03-24T09:20:35.302104+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:09.448556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":218,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:16.970870+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:21:16.970870+0000","last_peered":"2024-03-24T09:21:16.970870+0000","last_clean":"2024-03-24T09:21:16.970870+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:21:16.970870+0000","last_undegraded":"2024-03-24T09:21:16.970870+0000","last_fullsized":"2024-03-24T09:21:16.970870+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":156,"num_read_kb":140,"num_write":37,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025520+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:20:35.025520+0000","last_peered":"2024-03-24T09:20:35.025520+0000","last_clean":"2024-03-24T09:20:35.025520+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:20:35.025520+0000","last_undegraded":"2024-03-24T09:20:35.025520+0000","last_fullsized":"2024-03-24T09:20:35.025520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028663+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:20:35.028663+0000","last_peered":"2024-03-24T09:20:35.028663+0000","last_clean":"2024-03-24T09:20:35.028663+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:20:35.028663+0000","last_undegraded":"2024-03-24T09:20:35.028663+0000","last_fullsized":"2024-03-24T09:20:35.028663+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024008+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:20:35.024008+0000","last_peered":"2024-03-24T09:20:35.024008+0000","last_clean":"2024-03-24T09:20:35.024008+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:20:35.024008+0000","last_undegraded":"2024-03-24T09:20:35.024008+0000","last_fullsized":"2024-03-24T09:20:35.024008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300604+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:20:35.300604+0000","last_peered":"2024-03-24T09:20:35.300604+0000","last_clean":"2024-03-24T09:20:35.300604+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:20:35.300604+0000","last_undegraded":"2024-03-24T09:20:35.300604+0000","last_fullsized":"2024-03-24T09:20:35.300604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300183+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:20:35.300183+0000","last_peered":"2024-03-24T09:20:35.300183+0000","last_clean":"2024-03-24T09:20:35.300183+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:20:35.300183+0000","last_undegraded":"2024-03-24T09:20:35.300183+0000","last_fullsized":"2024-03-24T09:20:35.300183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028296+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:20:35.028296+0000","last_peered":"2024-03-24T09:20:35.028296+0000","last_clean":"2024-03-24T09:20:35.028296+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:20:35.028296+0000","last_undegraded":"2024-03-24T09:20:35.028296+0000","last_fullsized":"2024-03-24T09:20:35.028296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029915+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:20:35.029915+0000","last_peered":"2024-03-24T09:20:35.029915+0000","last_clean":"2024-03-24T09:20:35.029915+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:20:35.029915+0000","last_undegraded":"2024-03-24T09:20:35.029915+0000","last_fullsized":"2024-03-24T09:20:35.029915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023855+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:20:35.023855+0000","last_peered":"2024-03-24T09:20:35.023855+0000","last_clean":"2024-03-24T09:20:35.023855+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:20:35.023855+0000","last_undegraded":"2024-03-24T09:20:35.023855+0000","last_fullsized":"2024-03-24T09:20:35.023855+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":108,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:18.012185+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:21:18.012185+0000","last_peered":"2024-03-24T09:21:18.012185+0000","last_clean":"2024-03-24T09:21:18.012185+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:21:18.012185+0000","last_undegraded":"2024-03-24T09:21:18.012185+0000","last_fullsized":"2024-03-24T09:21:18.012185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.031305+0000","last_change":"2024-03-24T09:20:35.031305+0000","last_active":"2024-03-24T09:20:35.031305+0000","last_peered":"2024-03-24T09:20:35.031305+0000","last_clean":"2024-03-24T09:20:35.031305+0000","last_became_active":"2024-03-24T09:20:35.030952+0000","last_became_peered":"2024-03-24T09:20:35.030952+0000","last_unstale":"2024-03-24T09:20:35.031305+0000","last_undegraded":"2024-03-24T09:20:35.031305+0000","last_fullsized":"2024-03-24T09:20:35.031305+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:40.987185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300960+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:20:35.300960+0000","last_peered":"2024-03-24T09:20:35.300960+0000","last_clean":"2024-03-24T09:20:35.300960+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:20:35.300960+0000","last_undegraded":"2024-03-24T09:20:35.300960+0000","last_fullsized":"2024-03-24T09:20:35.300960+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":45,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025069+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:20:35.025069+0000","last_peered":"2024-03-24T09:20:35.025069+0000","last_clean":"2024-03-24T09:20:35.025069+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:20:35.025069+0000","last_undegraded":"2024-03-24T09:20:35.025069+0000","last_fullsized":"2024-03-24T09:20:35.025069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027110+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:20:35.027110+0000","last_peered":"2024-03-24T09:20:35.027110+0000","last_clean":"2024-03-24T09:20:35.027110+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:20:35.027110+0000","last_undegraded":"2024-03-24T09:20:35.027110+0000","last_fullsized":"2024-03-24T09:20:35.027110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027005+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:20:35.027005+0000","last_peered":"2024-03-24T09:20:35.027005+0000","last_clean":"2024-03-24T09:20:35.027005+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:20:35.027005+0000","last_undegraded":"2024-03-24T09:20:35.027005+0000","last_fullsized":"2024-03-24T09:20:35.027005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025594+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:20:35.025594+0000","last_peered":"2024-03-24T09:20:35.025594+0000","last_clean":"2024-03-24T09:20:35.025594+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:20:35.025594+0000","last_undegraded":"2024-03-24T09:20:35.025594+0000","last_fullsized":"2024-03-24T09:20:35.025594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300314+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:20:35.300314+0000","last_peered":"2024-03-24T09:20:35.300314+0000","last_clean":"2024-03-24T09:20:35.300314+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:20:35.300314+0000","last_undegraded":"2024-03-24T09:20:35.300314+0000","last_fullsized":"2024-03-24T09:20:35.300314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025817+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:20:35.025817+0000","last_peered":"2024-03-24T09:20:35.025817+0000","last_clean":"2024-03-24T09:20:35.025817+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:20:35.025817+0000","last_undegraded":"2024-03-24T09:20:35.025817+0000","last_fullsized":"2024-03-24T09:20:35.025817+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023728+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:20:35.023728+0000","last_peered":"2024-03-24T09:20:35.023728+0000","last_clean":"2024-03-24T09:20:35.023728+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:20:35.023728+0000","last_undegraded":"2024-03-24T09:20:35.023728+0000","last_fullsized":"2024-03-24T09:20:35.023728+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025704+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:20:35.025704+0000","last_peered":"2024-03-24T09:20:35.025704+0000","last_clean":"2024-03-24T09:20:35.025704+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:20:35.025704+0000","last_undegraded":"2024-03-24T09:20:35.025704+0000","last_fullsized":"2024-03-24T09:20:35.025704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300598+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:20:35.300598+0000","last_peered":"2024-03-24T09:20:35.300598+0000","last_clean":"2024-03-24T09:20:35.300598+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:20:35.300598+0000","last_undegraded":"2024-03-24T09:20:35.300598+0000","last_fullsized":"2024-03-24T09:20:35.300598+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:17.030030+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:21:17.030030+0000","last_peered":"2024-03-24T09:21:17.030030+0000","last_clean":"2024-03-24T09:21:17.030030+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:21:17.030030+0000","last_undegraded":"2024-03-24T09:21:17.030030+0000","last_fullsized":"2024-03-24T09:21:17.030030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025601+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:20:35.025601+0000","last_peered":"2024-03-24T09:20:35.025601+0000","last_clean":"2024-03-24T09:20:35.025601+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:20:35.025601+0000","last_undegraded":"2024-03-24T09:20:35.025601+0000","last_fullsized":"2024-03-24T09:20:35.025601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300873+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:20:35.300873+0000","last_peered":"2024-03-24T09:20:35.300873+0000","last_clean":"2024-03-24T09:20:35.300873+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:20:35.300873+0000","last_undegraded":"2024-03-24T09:20:35.300873+0000","last_fullsized":"2024-03-24T09:20:35.300873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028030+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:20:35.028030+0000","last_peered":"2024-03-24T09:20:35.028030+0000","last_clean":"2024-03-24T09:20:35.028030+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:20:35.028030+0000","last_undegraded":"2024-03-24T09:20:35.028030+0000","last_fullsized":"2024-03-24T09:20:35.028030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:17.010541+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:21:17.010541+0000","last_peered":"2024-03-24T09:21:17.010541+0000","last_clean":"2024-03-24T09:21:17.010541+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:21:17.010541+0000","last_undegraded":"2024-03-24T09:21:17.010541+0000","last_fullsized":"2024-03-24T09:21:17.010541+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030605+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:20:35.030605+0000","last_peered":"2024-03-24T09:20:35.030605+0000","last_clean":"2024-03-24T09:20:35.030605+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:20:35.030605+0000","last_undegraded":"2024-03-24T09:20:35.030605+0000","last_fullsized":"2024-03-24T09:20:35.030605+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301245+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:20:35.301245+0000","last_peered":"2024-03-24T09:20:35.301245+0000","last_clean":"2024-03-24T09:20:35.301245+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:20:35.301245+0000","last_undegraded":"2024-03-24T09:20:35.301245+0000","last_fullsized":"2024-03-24T09:20:35.301245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026059+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:20:35.026059+0000","last_peered":"2024-03-24T09:20:35.026059+0000","last_clean":"2024-03-24T09:20:35.026059+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:20:35.026059+0000","last_undegraded":"2024-03-24T09:20:35.026059+0000","last_fullsized":"2024-03-24T09:20:35.026059+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":362,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028757+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:20:35.028757+0000","last_peered":"2024-03-24T09:20:35.028757+0000","last_clean":"2024-03-24T09:20:35.028757+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:20:35.028757+0000","last_undegraded":"2024-03-24T09:20:35.028757+0000","last_fullsized":"2024-03-24T09:20:35.028757+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027181+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:20:35.027181+0000","last_peered":"2024-03-24T09:20:35.027181+0000","last_clean":"2024-03-24T09:20:35.027181+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:20:35.027181+0000","last_undegraded":"2024-03-24T09:20:35.027181+0000","last_fullsized":"2024-03-24T09:20:35.027181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300209+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:20:35.300209+0000","last_peered":"2024-03-24T09:20:35.300209+0000","last_clean":"2024-03-24T09:20:35.300209+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:20:35.300209+0000","last_undegraded":"2024-03-24T09:20:35.300209+0000","last_fullsized":"2024-03-24T09:20:35.300209+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028886+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:20:35.028886+0000","last_peered":"2024-03-24T09:20:35.028886+0000","last_clean":"2024-03-24T09:20:35.028886+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:20:35.028886+0000","last_undegraded":"2024-03-24T09:20:35.028886+0000","last_fullsized":"2024-03-24T09:20:35.028886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300131+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:20:35.300131+0000","last_peered":"2024-03-24T09:20:35.300131+0000","last_clean":"2024-03-24T09:20:35.300131+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:20:35.300131+0000","last_undegraded":"2024-03-24T09:20:35.300131+0000","last_fullsized":"2024-03-24T09:20:35.300131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025678+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:20:35.025678+0000","last_peered":"2024-03-24T09:20:35.025678+0000","last_clean":"2024-03-24T09:20:35.025678+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:20:35.025678+0000","last_undegraded":"2024-03-24T09:20:35.025678+0000","last_fullsized":"2024-03-24T09:20:35.025678+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300691+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:20:35.300691+0000","last_peered":"2024-03-24T09:20:35.300691+0000","last_clean":"2024-03-24T09:20:35.300691+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:20:35.300691+0000","last_undegraded":"2024-03-24T09:20:35.300691+0000","last_fullsized":"2024-03-24T09:20:35.300691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300434+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:20:35.300434+0000","last_peered":"2024-03-24T09:20:35.300434+0000","last_clean":"2024-03-24T09:20:35.300434+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:20:35.300434+0000","last_undegraded":"2024-03-24T09:20:35.300434+0000","last_fullsized":"2024-03-24T09:20:35.300434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'12","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023847+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:20:35.023847+0000","last_peered":"2024-03-24T09:20:35.023847+0000","last_clean":"2024-03-24T09:20:35.023847+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:20:35.023847+0000","last_undegraded":"2024-03-24T09:20:35.023847+0000","last_fullsized":"2024-03-24T09:20:35.023847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301502+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:20:35.301502+0000","last_peered":"2024-03-24T09:20:35.301502+0000","last_clean":"2024-03-24T09:20:35.301502+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:20:35.301502+0000","last_undegraded":"2024-03-24T09:20:35.301502+0000","last_fullsized":"2024-03-24T09:20:35.301502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023569+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:20:35.023569+0000","last_peered":"2024-03-24T09:20:35.023569+0000","last_clean":"2024-03-24T09:20:35.023569+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:20:35.023569+0000","last_undegraded":"2024-03-24T09:20:35.023569+0000","last_fullsized":"2024-03-24T09:20:35.023569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026098+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:20:35.026098+0000","last_peered":"2024-03-24T09:20:35.026098+0000","last_clean":"2024-03-24T09:20:35.026098+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:20:35.026098+0000","last_undegraded":"2024-03-24T09:20:35.026098+0000","last_fullsized":"2024-03-24T09:20:35.026098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026645+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:20:35.026645+0000","last_peered":"2024-03-24T09:20:35.026645+0000","last_clean":"2024-03-24T09:20:35.026645+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:20:35.026645+0000","last_undegraded":"2024-03-24T09:20:35.026645+0000","last_fullsized":"2024-03-24T09:20:35.026645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026005+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:20:35.026005+0000","last_peered":"2024-03-24T09:20:35.026005+0000","last_clean":"2024-03-24T09:20:35.026005+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:20:35.026005+0000","last_undegraded":"2024-03-24T09:20:35.026005+0000","last_fullsized":"2024-03-24T09:20:35.026005+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024428+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:20:35.024428+0000","last_peered":"2024-03-24T09:20:35.024428+0000","last_clean":"2024-03-24T09:20:35.024428+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:20:35.024428+0000","last_undegraded":"2024-03-24T09:20:35.024428+0000","last_fullsized":"2024-03-24T09:20:35.024428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024965+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:20:35.024965+0000","last_peered":"2024-03-24T09:20:35.024965+0000","last_clean":"2024-03-24T09:20:35.024965+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:20:35.024965+0000","last_undegraded":"2024-03-24T09:20:35.024965+0000","last_fullsized":"2024-03-24T09:20:35.024965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023972+0000","last_change":"2024-03-24T09:19:42.743462+0000","last_active":"2024-03-24T09:20:35.023972+0000","last_peered":"2024-03-24T09:20:35.023972+0000","last_clean":"2024-03-24T09:20:35.023972+0000","last_became_active":"2024-03-24T09:19:42.743078+0000","last_became_peered":"2024-03-24T09:19:42.743078+0000","last_unstale":"2024-03-24T09:20:35.023972+0000","last_undegraded":"2024-03-24T09:20:35.023972+0000","last_fullsized":"2024-03-24T09:20:35.023972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:26:49.958507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300051+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:20:35.300051+0000","last_peered":"2024-03-24T09:20:35.300051+0000","last_clean":"2024-03-24T09:20:35.300051+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:20:35.300051+0000","last_undegraded":"2024-03-24T09:20:35.300051+0000","last_fullsized":"2024-03-24T09:20:35.300051+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.299959+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:20:35.299959+0000","last_peered":"2024-03-24T09:20:35.299959+0000","last_clean":"2024-03-24T09:20:35.299959+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:20:35.299959+0000","last_undegraded":"2024-03-24T09:20:35.299959+0000","last_fullsized":"2024-03-24T09:20:35.299959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026556+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:20:35.026556+0000","last_peered":"2024-03-24T09:20:35.026556+0000","last_clean":"2024-03-24T09:20:35.026556+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:20:35.026556+0000","last_undegraded":"2024-03-24T09:20:35.026556+0000","last_fullsized":"2024-03-24T09:20:35.026556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023812+0000","last_change":"2024-03-24T09:19:38.100940+0000","last_active":"2024-03-24T09:20:35.023812+0000","last_peered":"2024-03-24T09:20:35.023812+0000","last_clean":"2024-03-24T09:20:35.023812+0000","last_became_active":"2024-03-24T09:19:38.100613+0000","last_became_peered":"2024-03-24T09:19:38.100613+0000","last_unstale":"2024-03-24T09:20:35.023812+0000","last_undegraded":"2024-03-24T09:20:35.023812+0000","last_fullsized":"2024-03-24T09:20:35.023812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:12:53.483332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300739+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:20:35.300739+0000","last_peered":"2024-03-24T09:20:35.300739+0000","last_clean":"2024-03-24T09:20:35.300739+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:20:35.300739+0000","last_undegraded":"2024-03-24T09:20:35.300739+0000","last_fullsized":"2024-03-24T09:20:35.300739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025812+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:20:35.025812+0000","last_peered":"2024-03-24T09:20:35.025812+0000","last_clean":"2024-03-24T09:20:35.025812+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:20:35.025812+0000","last_undegraded":"2024-03-24T09:20:35.025812+0000","last_fullsized":"2024-03-24T09:20:35.025812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025373+0000","last_change":"2024-03-24T09:19:38.105336+0000","last_active":"2024-03-24T09:20:35.025373+0000","last_peered":"2024-03-24T09:20:35.025373+0000","last_clean":"2024-03-24T09:20:35.025373+0000","last_became_active":"2024-03-24T09:19:38.103695+0000","last_became_peered":"2024-03-24T09:19:38.103695+0000","last_unstale":"2024-03-24T09:20:35.025373+0000","last_undegraded":"2024-03-24T09:20:35.025373+0000","last_fullsized":"2024-03-24T09:20:35.025373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:11:41.784391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029356+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:20:35.029356+0000","last_peered":"2024-03-24T09:20:35.029356+0000","last_clean":"2024-03-24T09:20:35.029356+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:20:35.029356+0000","last_undegraded":"2024-03-24T09:20:35.029356+0000","last_fullsized":"2024-03-24T09:20:35.029356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024741+0000","last_change":"2024-03-24T09:19:42.744163+0000","last_active":"2024-03-24T09:20:35.024741+0000","last_peered":"2024-03-24T09:20:35.024741+0000","last_clean":"2024-03-24T09:20:35.024741+0000","last_became_active":"2024-03-24T09:19:42.743754+0000","last_became_peered":"2024-03-24T09:19:42.743754+0000","last_unstale":"2024-03-24T09:20:35.024741+0000","last_undegraded":"2024-03-24T09:20:35.024741+0000","last_fullsized":"2024-03-24T09:20:35.024741+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:07.082490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":107,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:21:18.012142+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:21:18.012142+0000","last_peered":"2024-03-24T09:21:18.012142+0000","last_clean":"2024-03-24T09:21:18.012142+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:21:18.012142+0000","last_undegraded":"2024-03-24T09:21:18.012142+0000","last_fullsized":"2024-03-24T09:21:18.012142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.029915+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:20:35.029915+0000","last_peered":"2024-03-24T09:20:35.029915+0000","last_clean":"2024-03-24T09:20:35.029915+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:20:35.029915+0000","last_undegraded":"2024-03-24T09:20:35.029915+0000","last_fullsized":"2024-03-24T09:20:35.029915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026374+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:20:35.026374+0000","last_peered":"2024-03-24T09:20:35.026374+0000","last_clean":"2024-03-24T09:20:35.026374+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:20:35.026374+0000","last_undegraded":"2024-03-24T09:20:35.026374+0000","last_fullsized":"2024-03-24T09:20:35.026374+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.028460+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:20:35.028460+0000","last_peered":"2024-03-24T09:20:35.028460+0000","last_clean":"2024-03-24T09:20:35.028460+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:20:35.028460+0000","last_undegraded":"2024-03-24T09:20:35.028460+0000","last_fullsized":"2024-03-24T09:20:35.028460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.031333+0000","last_change":"2024-03-24T09:20:35.031333+0000","last_active":"2024-03-24T09:20:35.031333+0000","last_peered":"2024-03-24T09:20:35.031333+0000","last_clean":"2024-03-24T09:20:35.031333+0000","last_became_active":"2024-03-24T09:20:35.030995+0000","last_became_peered":"2024-03-24T09:20:35.030995+0000","last_unstale":"2024-03-24T09:20:35.031333+0000","last_undegraded":"2024-03-24T09:20:35.031333+0000","last_fullsized":"2024-03-24T09:20:35.031333+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:05.746740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026790+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:20:35.026790+0000","last_peered":"2024-03-24T09:20:35.026790+0000","last_clean":"2024-03-24T09:20:35.026790+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:20:35.026790+0000","last_undegraded":"2024-03-24T09:20:35.026790+0000","last_fullsized":"2024-03-24T09:20:35.026790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300874+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:20:35.300874+0000","last_peered":"2024-03-24T09:20:35.300874+0000","last_clean":"2024-03-24T09:20:35.300874+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:20:35.300874+0000","last_undegraded":"2024-03-24T09:20:35.300874+0000","last_fullsized":"2024-03-24T09:20:35.300874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.030633+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:20:35.030633+0000","last_peered":"2024-03-24T09:20:35.030633+0000","last_clean":"2024-03-24T09:20:35.030633+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:20:35.030633+0000","last_undegraded":"2024-03-24T09:20:35.030633+0000","last_fullsized":"2024-03-24T09:20:35.030633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300002+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:20:35.300002+0000","last_peered":"2024-03-24T09:20:35.300002+0000","last_clean":"2024-03-24T09:20:35.300002+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:20:35.300002+0000","last_undegraded":"2024-03-24T09:20:35.300002+0000","last_fullsized":"2024-03-24T09:20:35.300002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023873+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:20:35.023873+0000","last_peered":"2024-03-24T09:20:35.023873+0000","last_clean":"2024-03-24T09:20:35.023873+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:20:35.023873+0000","last_undegraded":"2024-03-24T09:20:35.023873+0000","last_fullsized":"2024-03-24T09:20:35.023873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.023785+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:20:35.023785+0000","last_peered":"2024-03-24T09:20:35.023785+0000","last_clean":"2024-03-24T09:20:35.023785+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:20:35.023785+0000","last_undegraded":"2024-03-24T09:20:35.023785+0000","last_fullsized":"2024-03-24T09:20:35.023785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.026663+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:20:35.026663+0000","last_peered":"2024-03-24T09:20:35.026663+0000","last_clean":"2024-03-24T09:20:35.026663+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:20:35.026663+0000","last_undegraded":"2024-03-24T09:20:35.026663+0000","last_fullsized":"2024-03-24T09:20:35.026663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300709+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:20:35.300709+0000","last_peered":"2024-03-24T09:20:35.300709+0000","last_clean":"2024-03-24T09:20:35.300709+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:20:35.300709+0000","last_undegraded":"2024-03-24T09:20:35.300709+0000","last_fullsized":"2024-03-24T09:20:35.300709+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300370+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:20:35.300370+0000","last_peered":"2024-03-24T09:20:35.300370+0000","last_clean":"2024-03-24T09:20:35.300370+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:20:35.300370+0000","last_undegraded":"2024-03-24T09:20:35.300370+0000","last_fullsized":"2024-03-24T09:20:35.300370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.024488+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:20:35.024488+0000","last_peered":"2024-03-24T09:20:35.024488+0000","last_clean":"2024-03-24T09:20:35.024488+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:20:35.024488+0000","last_undegraded":"2024-03-24T09:20:35.024488+0000","last_fullsized":"2024-03-24T09:20:35.024488+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.301418+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:20:35.301418+0000","last_peered":"2024-03-24T09:20:35.301418+0000","last_clean":"2024-03-24T09:20:35.301418+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:20:35.301418+0000","last_undegraded":"2024-03-24T09:20:35.301418+0000","last_fullsized":"2024-03-24T09:20:35.301418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.027522+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:20:35.027522+0000","last_peered":"2024-03-24T09:20:35.027522+0000","last_clean":"2024-03-24T09:20:35.027522+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:20:35.027522+0000","last_undegraded":"2024-03-24T09:20:35.027522+0000","last_fullsized":"2024-03-24T09:20:35.027522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.299996+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:20:35.299996+0000","last_peered":"2024-03-24T09:20:35.299996+0000","last_clean":"2024-03-24T09:20:35.299996+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:20:35.299996+0000","last_undegraded":"2024-03-24T09:20:35.299996+0000","last_fullsized":"2024-03-24T09:20:35.299996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.025002+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:20:35.025002+0000","last_peered":"2024-03-24T09:20:35.025002+0000","last_clean":"2024-03-24T09:20:35.025002+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:20:35.025002+0000","last_undegraded":"2024-03-24T09:20:35.025002+0000","last_fullsized":"2024-03-24T09:20:35.025002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300252+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:20:35.300252+0000","last_peered":"2024-03-24T09:20:35.300252+0000","last_clean":"2024-03-24T09:20:35.300252+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:20:35.300252+0000","last_undegraded":"2024-03-24T09:20:35.300252+0000","last_fullsized":"2024-03-24T09:20:35.300252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-24T09:20:35.300177+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:20:35.300177+0000","last_peered":"2024-03-24T09:20:35.300177+0000","last_clean":"2024-03-24T09:20:35.300177+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:20:35.300177+0000","last_undegraded":"2024-03-24T09:20:35.300177+0000","last_fullsized":"2024-03-24T09:20:35.300177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":280427,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70560,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":771,"num_read_kb":461,"num_write":242,"num_write_kb":141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":211680,"data_compressed":4809,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":211,"ondisk_log_size":211,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":501433,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645350,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1150976,"data_stored":1151996,"data_compressed":57942,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":505459,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":501741,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1077248,"data_stored":1098629,"data_compressed":56339,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":496867,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":561253,"data_compressed":22274,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":498323,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":561864,"data_compressed":22274,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":496575,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":501332,"data_compressed":20671,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1085440,"data_stored":1093820,"data_compressed":56339,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34029,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":65108,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65719,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":65483,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:21:24.173 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T09:21:24.173 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T09:21:24.174 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T09:21:24.174 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T09:21:24.415 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:25 smithi116 ceph-mon[32207]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:21:25.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:25 smithi116 ceph-mon[32207]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:25 smithi143 ceph-mon[35965]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:21:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:25 smithi143 ceph-mon[35965]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:25 smithi117 ceph-mon[35788]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:21:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:25 smithi117 ceph-mon[35788]: pgmap v346: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 -- 172.21.15.116:0/1586482911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881013e0 msgr2=0x7f5488101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 --2- 172.21.15.116:0/1586482911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881013e0 0x7f5488101860 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5470009990 tx=0x7f5470009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 -- 172.21.15.116:0/1586482911 shutdown_connections 2024-03-24T09:21:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 --2- 172.21.15.116:0/1586482911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5488101da0 0x7f548810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 --2- 172.21.15.116:0/1586482911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54881072f0 0x7f54881076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 --2- 172.21.15.116:0/1586482911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881013e0 0x7f5488101860 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 -- 172.21.15.116:0/1586482911 >> 172.21.15.116:0/1586482911 conn(0x7f5488077f60 msgr2=0x7f5488078370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 -- 172.21.15.116:0/1586482911 shutdown_connections 2024-03-24T09:21:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.170+0000 7f548e155700 1 -- 172.21.15.116:0/1586482911 wait complete. 2024-03-24T09:21:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 Processor -- start 2024-03-24T09:21:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 -- start start 2024-03-24T09:21:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54881013e0 0x7f5488198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5488101da0 0x7f5488198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 0x7f548819cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f548819d590 con 0x7f54881072f0 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f548819d700 con 0x7f5488101da0 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f548e155700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f548819d8c0 con 0x7f54881013e0 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f5487fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 0x7f548819cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.171+0000 7f547effd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5488101da0 0x7f5488198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f5487fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 0x7f548819cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44292/0 (socket says 172.21.15.116:44292) 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f5487fff700 1 -- 172.21.15.116:0/3938124172 learned_addr learned my addr 172.21.15.116:0/3938124172 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f54877fe700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54881013e0 0x7f5488198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f547effd700 1 -- 172.21.15.116:0/3938124172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54881013e0 msgr2=0x7f5488198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f547effd700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54881013e0 0x7f5488198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f547effd700 1 -- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 msgr2=0x7f548819cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f547effd700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 0x7f548819cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f547effd700 1 -- 172.21.15.116:0/3938124172 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54700095f0 con 0x7f5488101da0 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f54877fe700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54881013e0 0x7f5488198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f5487fff700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 0x7f548819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f547effd700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5488101da0 0x7f5488198c40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f54700059d0 tx=0x7f5470004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547001d070 con 0x7f5488101da0 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548819db40 con 0x7f5488101da0 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5470022470 con 0x7f5488101da0 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5470017650 con 0x7f5488101da0 2024-03-24T09:21:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.172+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54881a5400 con 0x7f5488101da0 2024-03-24T09:21:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.173+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5488109720 con 0x7f5488101da0 2024-03-24T09:21:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.173+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f54700225e0 con 0x7f5488101da0 2024-03-24T09:21:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.175+0000 7f54857fa700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f54740a21e0 0x7f54740a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.175+0000 7f54877fe700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f54740a21e0 0x7f54740a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.175+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f54700cf220 con 0x7f5488101da0 2024-03-24T09:21:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.175+0000 7f54877fe700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f54740a21e0 0x7f54740a46a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5478009fd0 tx=0x7f5478009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.184+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f547005e280 con 0x7f5488101da0 2024-03-24T09:21:26.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.419+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5488068ec0 con 0x7f5488101da0 2024-03-24T09:21:26.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.419+0000 7f54857fa700 1 -- 172.21.15.116:0/3938124172 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5470097dc0 con 0x7f5488101da0 2024-03-24T09:21:26.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:26.420 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T09:21:26.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f54740a21e0 msgr2=0x7f54740a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:26.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f54740a21e0 0x7f54740a46a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5478009fd0 tx=0x7f5478009260 comp rx=0 tx=0).stop 2024-03-24T09:21:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5488101da0 msgr2=0x7f5488198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5488101da0 0x7f5488198c40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f54700059d0 tx=0x7f5470004730 comp rx=0 tx=0).stop 2024-03-24T09:21:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 shutdown_connections 2024-03-24T09:21:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f54740a21e0 0x7f54740a46a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54881072f0 0x7f548819cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54881013e0 0x7f5488198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 --2- 172.21.15.116:0/3938124172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5488101da0 0x7f5488198c40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.421+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 >> 172.21.15.116:0/3938124172 conn(0x7f5488077f60 msgr2=0x7f5488100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.422+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 shutdown_connections 2024-03-24T09:21:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:26.422+0000 7f548e155700 1 -- 172.21.15.116:0/3938124172 wait complete. 2024-03-24T09:21:27.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:27 smithi116 ceph-mon[32207]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T09:21:27.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:27 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3938124172' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T09:21:27.206 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T09:21:27.206 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:21:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:27 smithi143 ceph-mon[35965]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T09:21:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:27 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3938124172' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- 172.21.15.116:0/530494629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 msgr2=0x7febb0102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- 172.21.15.116:0/530494629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb0102450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7febac009900 tx=0x7febac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- 172.21.15.116:0/530494629 shutdown_connections 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- 172.21.15.116:0/530494629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0105340 0x7febb01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- 172.21.15.116:0/530494629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febb0102990 0x7febb0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- 172.21.15.116:0/530494629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb0102450 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- 172.21.15.116:0/530494629 >> 172.21.15.116:0/530494629 conn(0x7febb00fbc40 msgr2=0x7febb00fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- 172.21.15.116:0/530494629 shutdown_connections 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- 172.21.15.116:0/530494629 wait complete. 2024-03-24T09:21:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 Processor -- start 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- start start 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0102990 0x7febb019e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febb0105340 0x7febb01a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febb01a57e0 con 0x7febb0100020 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7febb01a5950 con 0x7febb0105340 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb822a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7febb01a5b10 con 0x7febb0102990 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.313+0000 7febb5fc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb019dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33002/0 (socket says 172.21.15.116:33002) 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 -- 172.21.15.116:0/1051848471 learned_addr learned my addr 172.21.15.116:0/1051848471 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb67c7640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febb0105340 0x7febb01a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb57c5640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0102990 0x7febb019e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 -- 172.21.15.116:0/1051848471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0102990 msgr2=0x7febb019e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0102990 0x7febb019e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 -- 172.21.15.116:0/1051848471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febb0105340 msgr2=0x7febb01a5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febb0105340 0x7febb01a5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 -- 172.21.15.116:0/1051848471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febac009560 con 0x7febb0100020 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb57c5640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0102990 0x7febb019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb5fc6640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb019dd00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7febac006010 tx=0x7febac004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac004a50 con 0x7febb0100020 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7febac02de90 con 0x7febb0100020 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac005410 con 0x7febb0100020 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febb01a5d90 con 0x7febb0100020 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.314+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febb01a6190 con 0x7febb0100020 2024-03-24T09:21:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.315+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febb00fd730 con 0x7febb0100020 2024-03-24T09:21:27.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:27 smithi117 ceph-mon[35788]: pgmap v347: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-03-24T09:21:27.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:27 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3938124172' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T09:21:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.316+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7febac005570 con 0x7febb0100020 2024-03-24T09:21:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.317+0000 7feba6ffd640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feb9c0a21d0 0x7feb9c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.317+0000 7febb57c5640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feb9c0a21d0 0x7feb9c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.317+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7febac0f0190 con 0x7febb0100020 2024-03-24T09:21:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.317+0000 7febb57c5640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feb9c0a21d0 0x7feb9c0a4690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feba0009c20 tx=0x7feba0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:27.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.321+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7febac0b8d20 con 0x7febb0100020 2024-03-24T09:21:27.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.494+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febb0113870 con 0x7febb0100020 2024-03-24T09:21:27.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.495+0000 7feba6ffd640 1 -- 172.21.15.116:0/1051848471 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24669 (secure 0 0 0) 0x7febac0bac50 con 0x7febb0100020 2024-03-24T09:21:27.496 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:27.496 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":91,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:20:35.020321+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:21:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feb9c0a21d0 msgr2=0x7feb9c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feb9c0a21d0 0x7feb9c0a4690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7feba0009c20 tx=0x7feba0005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 msgr2=0x7febb019dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb019dd00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7febac006010 tx=0x7febac004370 comp rx=0 tx=0).stop 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 shutdown_connections 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febb0105340 0x7febb01a5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7feb9c0a21d0 0x7feb9c0a4690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7febb0102990 0x7febb019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 --2- 172.21.15.116:0/1051848471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febb0100020 0x7febb019dd00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.497+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 >> 172.21.15.116:0/1051848471 conn(0x7febb00fbc40 msgr2=0x7febb0103320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.498+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 shutdown_connections 2024-03-24T09:21:27.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.498+0000 7febb822a640 1 -- 172.21.15.116:0/1051848471 wait complete. 2024-03-24T09:21:27.510 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:21:27.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:21:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 -- 172.21.15.116:0/348825369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4101800 msgr2=0x7f4dd4101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/348825369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4101800 0x7f4dd4101c80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009900 tx=0x7f4dc802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:27.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 -- 172.21.15.116:0/348825369 shutdown_connections 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/348825369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dd41021c0 0x7f4dd4108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/348825369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4101800 0x7f4dd4101c80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/348825369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dd4100610 0x7f4dd4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 -- 172.21.15.116:0/348825369 >> 172.21.15.116:0/348825369 conn(0x7f4dd40fbb80 msgr2=0x7f4dd40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 -- 172.21.15.116:0/348825369 shutdown_connections 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.615+0000 7f4ddb13e640 1 -- 172.21.15.116:0/348825369 wait complete. 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 Processor -- start 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 -- start start 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 0x7f4dd41976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dd4101800 0x7f4dd4197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dd41021c0 0x7f4dd419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd419f1a0 con 0x7f4dd4100610 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd419f310 con 0x7f4dd4101800 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4ddb13e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd419f4d0 con 0x7f4dd41021c0 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 0x7f4dd41976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 0x7f4dd41976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33014/0 (socket says 172.21.15.116:33014) 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 -- 172.21.15.116:0/1764878755 learned_addr learned my addr 172.21.15.116:0/1764878755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd3fff640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dd4101800 0x7f4dd4197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd96db640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dd41021c0 0x7f4dd419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 -- 172.21.15.116:0/1764878755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dd41021c0 msgr2=0x7f4dd419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dd41021c0 0x7f4dd419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 -- 172.21.15.116:0/1764878755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dd4101800 msgr2=0x7f4dd4197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dd4101800 0x7f4dd4197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 -- 172.21.15.116:0/1764878755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc8009560 con 0x7f4dd4100610 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.616+0000 7f4dd8eda640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 0x7f4dd41976b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400e9c0 tx=0x7f4dc400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.617+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4009700 con 0x7f4dd4100610 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.617+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4dc40044d0 con 0x7f4dd4100610 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.617+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd419f7b0 con 0x7f4dd4100610 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.617+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4010430 con 0x7f4dd4100610 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.617+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd419fc90 con 0x7f4dd4100610 2024-03-24T09:21:27.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.618+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd410bed0 con 0x7f4dd4100610 2024-03-24T09:21:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.619+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f4dc4003f30 con 0x7f4dd4100610 2024-03-24T09:21:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.619+0000 7f4dd1ffb640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4dbc0a2150 0x7f4dbc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.620+0000 7f4dd3fff640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4dbc0a2150 0x7f4dbc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.620+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4dc4014070 con 0x7f4dd4100610 2024-03-24T09:21:27.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.620+0000 7f4dd3fff640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4dbc0a2150 0x7f4dbc0a4610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009230 tx=0x7f4dc8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:27.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.624+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4dc4097080 con 0x7f4dd4100610 2024-03-24T09:21:27.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.798+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4dd410c080 con 0x7f4dd4100610 2024-03-24T09:21:27.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.799+0000 7f4dd1ffb640 1 -- 172.21.15.116:0/1764878755 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f4dc409a740 con 0x7f4dd4100610 2024-03-24T09:21:27.799 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.800+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4dbc0a2150 msgr2=0x7f4dbc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.800+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4dbc0a2150 0x7f4dbc0a4610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009230 tx=0x7f4dc8005fd0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.800+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 msgr2=0x7f4dd41976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.800+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 0x7f4dd41976b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400e9c0 tx=0x7f4dc400ee80 comp rx=0 tx=0).stop 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 shutdown_connections 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4dbc0a2150 0x7f4dbc0a4610 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dd41021c0 0x7f4dd419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dd4101800 0x7f4dd4197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 --2- 172.21.15.116:0/1764878755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dd4100610 0x7f4dd41976b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 >> 172.21.15.116:0/1764878755 conn(0x7f4dd40fbb80 msgr2=0x7f4dd4068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 shutdown_connections 2024-03-24T09:21:27.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.801+0000 7f4ddb13e640 1 -- 172.21.15.116:0/1764878755 wait complete. 2024-03-24T09:21:27.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:21:27.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- 172.21.15.116:0/229072744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 msgr2=0x7fdfd4100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/229072744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 0x7fdfd4100a50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009900 tx=0x7fdfc802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:27.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- 172.21.15.116:0/229072744 shutdown_connections 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/229072744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfd41021e0 0x7fdfd4108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/229072744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd4101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/229072744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 0x7fdfd4100a50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- 172.21.15.116:0/229072744 >> 172.21.15.116:0/229072744 conn(0x7fdfd40fbbc0 msgr2=0x7fdfd40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- 172.21.15.116:0/229072744 shutdown_connections 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- 172.21.15.116:0/229072744 wait complete. 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 Processor -- start 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- start start 2024-03-24T09:21:27.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 0x7fdfd406a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfd41021e0 0x7fdfd419ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd419f230 con 0x7fdfd4101820 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd419f3a0 con 0x7fdfd4100630 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.916+0000 7fdfdabb5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd419f560 con 0x7fdfd41021e0 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd419c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33040/0 (socket says 172.21.15.116:33040) 2024-03-24T09:21:27.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd8951640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 0x7fdfd406a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 -- 172.21.15.116:0/3727542998 learned_addr learned my addr 172.21.15.116:0/3727542998 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd9152640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfd41021e0 0x7fdfd419ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 -- 172.21.15.116:0/3727542998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfd41021e0 msgr2=0x7fdfd419ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfd41021e0 0x7fdfd419ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 -- 172.21.15.116:0/3727542998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 msgr2=0x7fdfd406a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 0x7fdfd406a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 -- 172.21.15.116:0/3727542998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc8009560 con 0x7fdfd4101820 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd3fff640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd419c430 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000e9f0 tx=0x7fdfc000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0009700 con 0x7fdfd4101820 2024-03-24T09:21:27.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd419f7e0 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfc000cd80 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0018840 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.917+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd419fd40 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.918+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd41079a0 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.919+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fdfc00189a0 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.920+0000 7fdfd1ffb640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdfc40a2150 0x7fdfc40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.920+0000 7fdfd8951640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdfc40a2150 0x7fdfc40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.920+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fdfc0014070 con 0x7fdfd4101820 2024-03-24T09:21:27.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.920+0000 7fdfd8951640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdfc40a2150 0x7fdfc40a4610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdfc802c240 tx=0x7fdfc802c770 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:27.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:27.924+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdfc0053280 con 0x7fdfd4101820 2024-03-24T09:21:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:28 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1051848471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:28.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:28 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1764878755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:28.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.102+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdfd410c140 con 0x7fdfd4101820 2024-03-24T09:21:28.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.102+0000 7fdfd1ffb640 1 -- 172.21.15.116:0/3727542998 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fdfc0097080 con 0x7fdfd4101820 2024-03-24T09:21:28.103 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:28.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdfc40a2150 msgr2=0x7fdfc40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdfc40a2150 0x7fdfc40a4610 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdfc802c240 tx=0x7fdfc802c770 comp rx=0 tx=0).stop 2024-03-24T09:21:28.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 msgr2=0x7fdfd419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd419c430 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000e9f0 tx=0x7fdfc000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 shutdown_connections 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdfc40a2150 0x7fdfc40a4610 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfd41021e0 0x7fdfd419ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfd4101820 0x7fdfd419c430 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 --2- 172.21.15.116:0/3727542998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd4100630 0x7fdfd406a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 >> 172.21.15.116:0/3727542998 conn(0x7fdfd40fbbc0 msgr2=0x7fdfd4104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 shutdown_connections 2024-03-24T09:21:28.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.104+0000 7fdfdabb5640 1 -- 172.21.15.116:0/3727542998 wait complete. 2024-03-24T09:21:28.116 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:21:28.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 -- 172.21.15.116:0/1505733473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 msgr2=0x7f4b30101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 --2- 172.21.15.116:0/1505733473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30101c80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009860 tx=0x7f4b2002bb70 comp rx=0 tx=0).stop 2024-03-24T09:21:28.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 -- 172.21.15.116:0/1505733473 shutdown_connections 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 --2- 172.21.15.116:0/1505733473 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b301021c0 0x7f4b30108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 --2- 172.21.15.116:0/1505733473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30101c80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 --2- 172.21.15.116:0/1505733473 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b30100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 -- 172.21.15.116:0/1505733473 >> 172.21.15.116:0/1505733473 conn(0x7f4b300fbb80 msgr2=0x7f4b300fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 -- 172.21.15.116:0/1505733473 shutdown_connections 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 -- 172.21.15.116:0/1505733473 wait complete. 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.224+0000 7f4b374fb640 1 Processor -- start 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 -- start start 2024-03-24T09:21:28.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b301976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b301021c0 0x7f4b3019ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3019f1b0 con 0x7f4b30100610 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3019f320 con 0x7f4b30101800 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b374fb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3019f4e0 con 0x7f4b301021c0 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b35297640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b301976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b35297640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b301976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33050/0 (socket says 172.21.15.116:33050) 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b35297640 1 -- 172.21.15.116:0/441444186 learned_addr learned my addr 172.21.15.116:0/441444186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b35a98640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b301021c0 0x7f4b3019ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 -- 172.21.15.116:0/441444186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b301021c0 msgr2=0x7f4b3019ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b301021c0 0x7f4b3019ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 -- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 msgr2=0x7f4b301976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b301976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 -- 172.21.15.116:0/441444186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b2c009560 con 0x7f4b30101800 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b35297640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b301976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b34a96640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30197c00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b2002c0b0 tx=0x7f4b20032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2002c510 con 0x7f4b30101800 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b2002ce90 con 0x7f4b30101800 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.225+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b20003e20 con 0x7f4b30101800 2024-03-24T09:21:28.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.226+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b20009490 con 0x7f4b30101800 2024-03-24T09:21:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.226+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3019fad0 con 0x7f4b30101800 2024-03-24T09:21:28.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.227+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b30068500 con 0x7f4b30101800 2024-03-24T09:21:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.227+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f4b2003b040 con 0x7f4b30101800 2024-03-24T09:21:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.228+0000 7f4b267fc640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4b1c0a21d0 0x7f4b1c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.228+0000 7f4b35297640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4b1c0a21d0 0x7f4b1c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.228+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4b200ef9b0 con 0x7f4b30101800 2024-03-24T09:21:28.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.228+0000 7f4b35297640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4b1c0a21d0 0x7f4b1c0a4690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c0058c0 tx=0x7f4b2c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:28.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.232+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b200b8680 con 0x7f4b30101800 2024-03-24T09:21:28.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:28 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1051848471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:28.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:28 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1764878755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:28.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:28 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1051848471' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:28.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:28 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1764878755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:28.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.413+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4b3019fe10 con 0x7f4b30101800 2024-03-24T09:21:28.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.414+0000 7f4b267fc640 1 -- 172.21.15.116:0/441444186 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f4b200bbd40 con 0x7f4b30101800 2024-03-24T09:21:28.415 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:21:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4b1c0a21d0 msgr2=0x7f4b1c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4b1c0a21d0 0x7f4b1c0a4690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c0058c0 tx=0x7f4b2c009260 comp rx=0 tx=0).stop 2024-03-24T09:21:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 msgr2=0x7f4b30197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30197c00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4b2002c0b0 tx=0x7f4b20032430 comp rx=0 tx=0).stop 2024-03-24T09:21:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 shutdown_connections 2024-03-24T09:21:28.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4b1c0a21d0 0x7f4b1c0a4690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b301021c0 0x7f4b3019ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b30101800 0x7f4b30197c00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 --2- 172.21.15.116:0/441444186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b30100610 0x7f4b301976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 >> 172.21.15.116:0/441444186 conn(0x7f4b300fbb80 msgr2=0x7f4b30068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 shutdown_connections 2024-03-24T09:21:28.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.416+0000 7f4b374fb640 1 -- 172.21.15.116:0/441444186 wait complete. 2024-03-24T09:21:28.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:21:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 -- 172.21.15.116:0/2126988324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff570104ac0 msgr2=0x7ff570106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 --2- 172.21.15.116:0/2126988324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff570104ac0 0x7ff570106eb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff564009900 tx=0x7ff56402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 -- 172.21.15.116:0/2126988324 shutdown_connections 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 --2- 172.21.15.116:0/2126988324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5701073f0 0x7ff570109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 --2- 172.21.15.116:0/2126988324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff570104ac0 0x7ff570106eb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 --2- 172.21.15.116:0/2126988324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff570102190 0x7ff570104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 -- 172.21.15.116:0/2126988324 >> 172.21.15.116:0/2126988324 conn(0x7ff5700fbba0 msgr2=0x7ff5700fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 -- 172.21.15.116:0/2126988324 shutdown_connections 2024-03-24T09:21:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 -- 172.21.15.116:0/2126988324 wait complete. 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 Processor -- start 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 -- start start 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff570102190 0x7ff570197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.536+0000 7ff576780640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff570104ac0 0x7ff570197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff576780640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 0x7ff57019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff576780640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff57019f1f0 con 0x7ff5701073f0 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff576780640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff57019f360 con 0x7ff570102190 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff576780640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff57019f520 con 0x7ff570104ac0 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff574d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 0x7ff57019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff574d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 0x7ff57019ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33070/0 (socket says 172.21.15.116:33070) 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff570102190 0x7ff570197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff574d1d640 1 -- 172.21.15.116:0/1228052427 learned_addr learned my addr 172.21.15.116:0/1228052427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56f7fe640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff570104ac0 0x7ff570197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 -- 172.21.15.116:0/1228052427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff570104ac0 msgr2=0x7ff570197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff570104ac0 0x7ff570197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 -- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 msgr2=0x7ff57019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 0x7ff57019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 -- 172.21.15.116:0/1228052427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff564009560 con 0x7ff570102190 2024-03-24T09:21:28.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff574d1d640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 0x7ff57019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56ffff640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff570102190 0x7ff570197700 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff56000e9f0 tx=0x7ff56000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56000ccc0 con 0x7ff570102190 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57019f800 con 0x7ff570102190 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff56000ce20 con 0x7ff570102190 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5600187a0 con 0x7ff570102190 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.537+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57019fd30 con 0x7ff570102190 2024-03-24T09:21:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.538+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5700fd690 con 0x7ff570102190 2024-03-24T09:21:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.540+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7ff560010430 con 0x7ff570102190 2024-03-24T09:21:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.540+0000 7ff56d7fa640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff5580a2150 0x7ff5580a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.540+0000 7ff56f7fe640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff5580a2150 0x7ff5580a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.540+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff560014070 con 0x7ff570102190 2024-03-24T09:21:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.541+0000 7ff56f7fe640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff5580a2150 0x7ff5580a4610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff56402c7d0 tx=0x7ff564005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.545+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff5600974c0 con 0x7ff570102190 2024-03-24T09:21:28.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.727+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff57010cfa0 con 0x7ff570102190 2024-03-24T09:21:28.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.728+0000 7ff56d7fa640 1 -- 172.21.15.116:0/1228052427 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7ff56009ab80 con 0x7ff570102190 2024-03-24T09:21:28.729 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:28.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff5580a2150 msgr2=0x7ff5580a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff5580a2150 0x7ff5580a4610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff56402c7d0 tx=0x7ff564005fd0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff570102190 msgr2=0x7ff570197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff570102190 0x7ff570197700 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff56000e9f0 tx=0x7ff56000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 shutdown_connections 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5701073f0 0x7ff57019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff5580a2150 0x7ff5580a4610 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff570104ac0 0x7ff570197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 --2- 172.21.15.116:0/1228052427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff570102190 0x7ff570197700 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 >> 172.21.15.116:0/1228052427 conn(0x7ff5700fbba0 msgr2=0x7ff570104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 shutdown_connections 2024-03-24T09:21:28.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.730+0000 7ff576780640 1 -- 172.21.15.116:0/1228052427 wait complete. 2024-03-24T09:21:28.742 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.846+0000 7fa40b102640 1 -- 172.21.15.116:0/891191897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100630 msgr2=0x7fa404100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.846+0000 7fa40b102640 1 --2- 172.21.15.116:0/891191897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100630 0x7fa404100a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009900 tx=0x7fa3f802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- 172.21.15.116:0/891191897 shutdown_connections 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 --2- 172.21.15.116:0/891191897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4041021e0 0x7fa404108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 --2- 172.21.15.116:0/891191897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404101820 0x7fa404101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 --2- 172.21.15.116:0/891191897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100630 0x7fa404100a50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- 172.21.15.116:0/891191897 >> 172.21.15.116:0/891191897 conn(0x7fa4040fbba0 msgr2=0x7fa4040fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- 172.21.15.116:0/891191897 shutdown_connections 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- 172.21.15.116:0/891191897 wait complete. 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 Processor -- start 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- start start 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100630 0x7fa4041045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404101820 0x7fa404102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 0x7fa404103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041038a0 con 0x7fa4041021e0 2024-03-24T09:21:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa404103a10 con 0x7fa404101820 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40b102640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041a1820 con 0x7fa404100630 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40969f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 0x7fa404103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa408e9e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100630 0x7fa4041045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40969f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 0x7fa404103140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33084/0 (socket says 172.21.15.116:33084) 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa40969f640 1 -- 172.21.15.116:0/2280961798 learned_addr learned my addr 172.21.15.116:0/2280961798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.847+0000 7fa403fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404101820 0x7fa404102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40969f640 1 -- 172.21.15.116:0/2280961798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100630 msgr2=0x7fa4041045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40969f640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100630 0x7fa4041045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40969f640 1 -- 172.21.15.116:0/2280961798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404101820 msgr2=0x7fa404102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40969f640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404101820 0x7fa404102c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40969f640 1 -- 172.21.15.116:0/2280961798 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f0009490 con 0x7fa4041021e0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa408e9e640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100630 0x7fa4041045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa403fff640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404101820 0x7fa404102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40969f640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 0x7fa404103140 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009cf0 tx=0x7fa3fc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc009b40 con 0x7fa4041021e0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc009840 con 0x7fa4041021e0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3fc003b70 con 0x7fa4041021e0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc0107e0 con 0x7fa4041021e0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.848+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f8009560 con 0x7fa4041021e0 2024-03-24T09:21:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.849+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa40410bf10 con 0x7fa4041021e0 2024-03-24T09:21:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.850+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fa3fc003ce0 con 0x7fa4041021e0 2024-03-24T09:21:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.851+0000 7fa401ffb640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa3f40a2150 0x7fa3f40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.851+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa3fc014070 con 0x7fa4041021e0 2024-03-24T09:21:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.851+0000 7fa408e9e640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa3f40a2150 0x7fa3f40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.851+0000 7fa408e9e640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa3f40a2150 0x7fa3f40a4610 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa3f802c240 tx=0x7fa3f802c770 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:28.855+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa3fc0972d0 con 0x7fa4041021e0 2024-03-24T09:21:29.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.080+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4041a1ef0 con 0x7fa4041021e0 2024-03-24T09:21:29.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.080+0000 7fa401ffb640 1 -- 172.21.15.116:0/2280961798 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7fa4041a1ef0 con 0x7fa4041021e0 2024-03-24T09:21:29.081 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:29.082 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:20:06.925494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:21:29.082 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa3f40a2150 msgr2=0x7fa3f40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa3f40a2150 0x7fa3f40a4610 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa3f802c240 tx=0x7fa3f802c770 comp rx=0 tx=0).stop 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 msgr2=0x7fa404103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 0x7fa404103140 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009cf0 tx=0x7fa3fc00c6a0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 shutdown_connections 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4041021e0 0x7fa404103140 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404101820 0x7fa404102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa3f40a2150 0x7fa3f40a4610 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 --2- 172.21.15.116:0/2280961798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100630 0x7fa4041045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 >> 172.21.15.116:0/2280961798 conn(0x7fa4040fbba0 msgr2=0x7fa4040688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 shutdown_connections 2024-03-24T09:21:29.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.083+0000 7fa40b102640 1 -- 172.21.15.116:0/2280961798 wait complete. 2024-03-24T09:21:29.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:21:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 -- 172.21.15.116:0/1314959494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7728100630 msgr2=0x7f7728100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 --2- 172.21.15.116:0/1314959494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7728100630 0x7f7728100a50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7724009900 tx=0x7f772402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 -- 172.21.15.116:0/1314959494 shutdown_connections 2024-03-24T09:21:29.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 --2- 172.21.15.116:0/1314959494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77281021e0 0x7f7728108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 --2- 172.21.15.116:0/1314959494 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7728101820 0x7f7728101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 --2- 172.21.15.116:0/1314959494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7728100630 0x7f7728100a50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 -- 172.21.15.116:0/1314959494 >> 172.21.15.116:0/1314959494 conn(0x7f77280fbba0 msgr2=0x7f77280fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 -- 172.21.15.116:0/1314959494 shutdown_connections 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.198+0000 7f772faae640 1 -- 172.21.15.116:0/1314959494 wait complete. 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 Processor -- start 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 -- start start 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7728100630 0x7f7728197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7728101820 0x7f7728197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 0x7f772819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f772819f260 con 0x7f77281021e0 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f772819f3d0 con 0x7f7728101820 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772faae640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f772819f590 con 0x7f7728100630 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 0x7f772819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 0x7f772819ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33094/0 (socket says 172.21.15.116:33094) 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 -- 172.21.15.116:0/1248191890 learned_addr learned my addr 172.21.15.116:0/1248191890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772d84a640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7728100630 0x7f7728197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772d049640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7728101820 0x7f7728197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 -- 172.21.15.116:0/1248191890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7728100630 msgr2=0x7f7728197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7728100630 0x7f7728197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 -- 172.21.15.116:0/1248191890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7728101820 msgr2=0x7f7728197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7728101820 0x7f7728197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 -- 172.21.15.116:0/1248191890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7724009560 con 0x7f77281021e0 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f772e04b640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 0x7f772819ed20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f772000eb10 tx=0x7f772000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7720004b20 con 0x7f77281021e0 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.199+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7720004c80 con 0x7f77281021e0 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.200+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772819f870 con 0x7f77281021e0 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.200+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7720010690 con 0x7f77281021e0 2024-03-24T09:21:29.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.200+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772819fd30 con 0x7f77281021e0 2024-03-24T09:21:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.201+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772810bf10 con 0x7f77281021e0 2024-03-24T09:21:29.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.201+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f7720005020 con 0x7f77281021e0 2024-03-24T09:21:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.202+0000 7f771effd640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f77140a2150 0x7f77140a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.202+0000 7f772d84a640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f77140a2150 0x7f77140a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.202+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7720014070 con 0x7f77281021e0 2024-03-24T09:21:29.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.203+0000 7f772d84a640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f77140a2150 0x7f77140a4610 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f772402c240 tx=0x7f772402c770 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:29.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.206+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7720097250 con 0x7f77281021e0 2024-03-24T09:21:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:29 smithi143 ceph-mon[35965]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:29 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3727542998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:29 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/441444186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:29 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1228052427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:29 smithi117 ceph-mon[35788]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:29 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3727542998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:29 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/441444186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:29 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1228052427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:29 smithi116 ceph-mon[32207]: pgmap v348: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:29 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3727542998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:29 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/441444186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:29 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1228052427' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.389+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77281a0170 con 0x7f77281021e0 2024-03-24T09:21:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.391+0000 7f771effd640 1 -- 172.21.15.116:0/1248191890 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24669 (secure 0 0 0) 0x7f772009a910 con 0x7f77281021e0 2024-03-24T09:21:29.392 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:29.392 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":91,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:20:35.020321+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f77140a2150 msgr2=0x7f77140a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f77140a2150 0x7f77140a4610 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f772402c240 tx=0x7f772402c770 comp rx=0 tx=0).stop 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 msgr2=0x7f772819ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 0x7f772819ed20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f772000eb10 tx=0x7f772000c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 shutdown_connections 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77281021e0 0x7f772819ed20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7728101820 0x7f7728197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f77140a2150 0x7f77140a4610 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 --2- 172.21.15.116:0/1248191890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7728100630 0x7f7728197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 >> 172.21.15.116:0/1248191890 conn(0x7f77280fbba0 msgr2=0x7f7728068960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 shutdown_connections 2024-03-24T09:21:29.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.393+0000 7f772faae640 1 -- 172.21.15.116:0/1248191890 wait complete. 2024-03-24T09:21:29.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:21:29.405 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 -- 172.21.15.116:0/3978381009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04104ac0 msgr2=0x7f9d04106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/3978381009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04104ac0 0x7f9d04106eb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009900 tx=0x7f9cf402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 -- 172.21.15.116:0/3978381009 shutdown_connections 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/3978381009 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d041073f0 0x7f9d04109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/3978381009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04104ac0 0x7f9d04106eb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/3978381009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d04102190 0x7f9d04104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.509+0000 7f9d0bf13640 1 -- 172.21.15.116:0/3978381009 >> 172.21.15.116:0/3978381009 conn(0x7f9d040fbba0 msgr2=0x7f9d040fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 -- 172.21.15.116:0/3978381009 shutdown_connections 2024-03-24T09:21:29.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 -- 172.21.15.116:0/3978381009 wait complete. 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 Processor -- start 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 -- start start 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 0x7f9d041976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d04104ac0 0x7f9d04197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d041073f0 0x7f9d0419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0419f1c0 con 0x7f9d04102190 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0419f330 con 0x7f9d041073f0 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0bf13640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0419f4f0 con 0x7f9d04104ac0 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d09caf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 0x7f9d041976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d09caf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 0x7f9d041976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33114/0 (socket says 172.21.15.116:33114) 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d09caf640 1 -- 172.21.15.116:0/1735999683 learned_addr learned my addr 172.21.15.116:0/1735999683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d094ae640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d04104ac0 0x7f9d04197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.510+0000 7f9d0a4b0640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d041073f0 0x7f9d0419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d09caf640 1 -- 172.21.15.116:0/1735999683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d04104ac0 msgr2=0x7f9d04197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d09caf640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d04104ac0 0x7f9d04197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d09caf640 1 -- 172.21.15.116:0/1735999683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d041073f0 msgr2=0x7f9d0419ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d09caf640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d041073f0 0x7f9d0419ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d09caf640 1 -- 172.21.15.116:0/1735999683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf4009560 con 0x7f9d04102190 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d09caf640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 0x7f9d041976d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000e9f0 tx=0x7f9d0000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00009700 con 0x7f9d04102190 2024-03-24T09:21:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9d0000cd80 con 0x7f9d04102190 2024-03-24T09:21:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0419f7d0 con 0x7f9d04102190 2024-03-24T09:21:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00018840 con 0x7f9d04102190 2024-03-24T09:21:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.511+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0419fcb0 con 0x7f9d04102190 2024-03-24T09:21:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.512+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d040fd690 con 0x7f9d04102190 2024-03-24T09:21:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.513+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f9d000189a0 con 0x7f9d04102190 2024-03-24T09:21:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.513+0000 7f9cfaffd640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9cf00a2150 0x7f9cf00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.513+0000 7f9d094ae640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9cf00a2150 0x7f9cf00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.514+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9d00014070 con 0x7f9d04102190 2024-03-24T09:21:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.514+0000 7f9d094ae640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9cf00a2150 0x7f9cf00a4610 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9cf402c240 tx=0x7f9cf402c770 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:29.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.518+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9d00053280 con 0x7f9d04102190 2024-03-24T09:21:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.690+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d0402d160 con 0x7f9d04102190 2024-03-24T09:21:29.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.691+0000 7f9cfaffd640 1 -- 172.21.15.116:0/1735999683 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f9d00097080 con 0x7f9d04102190 2024-03-24T09:21:29.691 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9cf00a2150 msgr2=0x7f9cf00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9cf00a2150 0x7f9cf00a4610 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9cf402c240 tx=0x7f9cf402c770 comp rx=0 tx=0).stop 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 msgr2=0x7f9d041976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 0x7f9d041976d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000e9f0 tx=0x7f9d0000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 shutdown_connections 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d041073f0 0x7f9d0419ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9cf00a2150 0x7f9cf00a4610 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9d04104ac0 0x7f9d04197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 --2- 172.21.15.116:0/1735999683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9d04102190 0x7f9d041976d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 >> 172.21.15.116:0/1735999683 conn(0x7f9d040fbba0 msgr2=0x7f9d04102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 shutdown_connections 2024-03-24T09:21:29.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.693+0000 7f9d0bf13640 1 -- 172.21.15.116:0/1735999683 wait complete. 2024-03-24T09:21:29.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:21:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 -- 172.21.15.116:0/3067359733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 msgr2=0x7f1a20108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 --2- 172.21.15.116:0/3067359733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a20108a70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009ca0 tx=0x7f1a1802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 -- 172.21.15.116:0/3067359733 shutdown_connections 2024-03-24T09:21:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 --2- 172.21.15.116:0/3067359733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a20108a70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 --2- 172.21.15.116:0/3067359733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20101820 0x7f1a20101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 --2- 172.21.15.116:0/3067359733 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20100630 0x7f1a20100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 -- 172.21.15.116:0/3067359733 >> 172.21.15.116:0/3067359733 conn(0x7f1a200fbbc0 msgr2=0x7f1a200fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 -- 172.21.15.116:0/3067359733 shutdown_connections 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 -- 172.21.15.116:0/3067359733 wait complete. 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.809+0000 7f1a27a47640 1 Processor -- start 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 -- start start 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20100630 0x7f1a201976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20101820 0x7f1a20197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a2019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2019f190 con 0x7f1a201021e0 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2019f300 con 0x7f1a20100630 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2019f4c0 con 0x7f1a20101820 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a2019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a257e3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20100630 0x7f1a201976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a2019ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33142/0 (socket says 172.21.15.116:33142) 2024-03-24T09:21:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a257e3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20100630 0x7f1a201976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:49074/0 (socket says 172.21.15.116:49074) 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 -- 172.21.15.116:0/2942843585 learned_addr learned my addr 172.21.15.116:0/2942843585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a24fe2640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20101820 0x7f1a20197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 -- 172.21.15.116:0/2942843585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20101820 msgr2=0x7f1a20197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20101820 0x7f1a20197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 -- 172.21.15.116:0/2942843585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20100630 msgr2=0x7f1a201976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20100630 0x7f1a201976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 -- 172.21.15.116:0/2942843585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a18009900 con 0x7f1a201021e0 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a25fe4640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a2019ec50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1a18037040 tx=0x7f1a18032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18034e70 con 0x7f1a201021e0 2024-03-24T09:21:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a18003e20 con 0x7f1a201021e0 2024-03-24T09:21:29.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2019f740 con 0x7f1a201021e0 2024-03-24T09:21:29.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18004710 con 0x7f1a201021e0 2024-03-24T09:21:29.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.810+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2019fc40 con 0x7f1a201021e0 2024-03-24T09:21:29.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.812+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a20107950 con 0x7f1a201021e0 2024-03-24T09:21:29.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.813+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f1a1803b040 con 0x7f1a201021e0 2024-03-24T09:21:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.813+0000 7f1a167fc640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1a0c0a21d0 0x7f1a0c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.813+0000 7f1a257e3640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1a0c0a21d0 0x7f1a0c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.814+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f1a180f05b0 con 0x7f1a201021e0 2024-03-24T09:21:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.814+0000 7f1a257e3640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1a0c0a21d0 0x7f1a0c0a4690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1a20101680 tx=0x7f1a10005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.818+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1a180b9280 con 0x7f1a201021e0 2024-03-24T09:21:29.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.992+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a2002d1d0 con 0x7f1a201021e0 2024-03-24T09:21:29.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.993+0000 7f1a167fc640 1 -- 172.21.15.116:0/2942843585 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f1a180bc940 con 0x7f1a201021e0 2024-03-24T09:21:29.993 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1a0c0a21d0 msgr2=0x7f1a0c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1a0c0a21d0 0x7f1a0c0a4690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1a20101680 tx=0x7f1a10005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 msgr2=0x7f1a2019ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a2019ec50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1a18037040 tx=0x7f1a18032980 comp rx=0 tx=0).stop 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 shutdown_connections 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a201021e0 0x7f1a2019ec50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f1a0c0a21d0 0x7f1a0c0a4690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a20101820 0x7f1a20197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 --2- 172.21.15.116:0/2942843585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a20100630 0x7f1a201976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:29.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 >> 172.21.15.116:0/2942843585 conn(0x7f1a200fbbc0 msgr2=0x7f1a20104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:29.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 shutdown_connections 2024-03-24T09:21:29.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:29.995+0000 7f1a27a47640 1 -- 172.21.15.116:0/2942843585 wait complete. 2024-03-24T09:21:30.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 -- 172.21.15.116:0/199440606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 msgr2=0x7f8d1c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 --2- 172.21.15.116:0/199440606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c101ca0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009900 tx=0x7f8d0402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 -- 172.21.15.116:0/199440606 shutdown_connections 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 --2- 172.21.15.116:0/199440606 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d1c1021e0 0x7f8d1c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 --2- 172.21.15.116:0/199440606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c101ca0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 --2- 172.21.15.116:0/199440606 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d1c100630 0x7f8d1c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 -- 172.21.15.116:0/199440606 >> 172.21.15.116:0/199440606 conn(0x7f8d1c0fbbe0 msgr2=0x7f8d1c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 -- 172.21.15.116:0/199440606 shutdown_connections 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.113+0000 7f8d20c15640 1 -- 172.21.15.116:0/199440606 wait complete. 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 Processor -- start 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 -- start start 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d1c100630 0x7f8d1c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d1c1021e0 0x7f8d1c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c19f1b0 con 0x7f8d1c101820 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c19f320 con 0x7f8d1c1021e0 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d20c15640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c19f4e0 con 0x7f8d1c100630 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.114+0000 7f8d1a59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d1c100630 0x7f8d1c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33156/0 (socket says 172.21.15.116:33156) 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d1c1021e0 0x7f8d1c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 -- 172.21.15.116:0/3960372234 learned_addr learned my addr 172.21.15.116:0/3960372234 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 -- 172.21.15.116:0/3960372234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d1c100630 msgr2=0x7f8d1c1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d1c100630 0x7f8d1c1976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 -- 172.21.15.116:0/3960372234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d1c1021e0 msgr2=0x7f8d1c19ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d1c1021e0 0x7f8d1c19ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 -- 172.21.15.116:0/3960372234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d04009560 con 0x7f8d1c101820 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d13fff640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c197c00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006010 tx=0x7f8d04004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04004a50 con 0x7f8d1c101820 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8d0402de90 con 0x7f8d1c101820 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c19f760 con 0x7f8d1c101820 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04005410 con 0x7f8d1c101820 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.115+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c19fc70 con 0x7f8d1c101820 2024-03-24T09:21:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.116+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d1c10bef0 con 0x7f8d1c101820 2024-03-24T09:21:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.117+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f8d04005570 con 0x7f8d1c101820 2024-03-24T09:21:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.118+0000 7f8d137fe640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8d080a2150 0x7f8d080a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.118+0000 7f8d1a59c640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8d080a2150 0x7f8d080a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.118+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8d040f00d0 con 0x7f8d1c101820 2024-03-24T09:21:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.119+0000 7f8d1a59c640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8d080a2150 0x7f8d080a4610 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c101680 tx=0x7f8d0c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:30.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.123+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8d040b8cf0 con 0x7f8d1c101820 2024-03-24T09:21:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.296+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8d1c1a0100 con 0x7f8d1c101820 2024-03-24T09:21:30.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.297+0000 7f8d137fe640 1 -- 172.21.15.116:0/3960372234 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f8d040bc3b0 con 0x7f8d1c101820 2024-03-24T09:21:30.297 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.298+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8d080a2150 msgr2=0x7f8d080a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.298+0000 7f8d20c15640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8d080a2150 0x7f8d080a4610 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c101680 tx=0x7f8d0c005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.298+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 msgr2=0x7f8d1c197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.298+0000 7f8d20c15640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c197c00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8d04006010 tx=0x7f8d04004370 comp rx=0 tx=0).stop 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 shutdown_connections 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d1c1021e0 0x7f8d1c19ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d1c101820 0x7f8d1c197c00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8d080a2150 0x7f8d080a4610 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 --2- 172.21.15.116:0/3960372234 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d1c100630 0x7f8d1c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 >> 172.21.15.116:0/3960372234 conn(0x7f8d1c0fbbe0 msgr2=0x7f8d1c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 shutdown_connections 2024-03-24T09:21:30.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.299+0000 7f8d20c15640 1 -- 172.21.15.116:0/3960372234 wait complete. 2024-03-24T09:21:30.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:30 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2280961798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:30.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:30 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1248191890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:30.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:30 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1735999683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:30.309 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:30 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2942843585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:30.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:21:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:30 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2280961798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:30 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1248191890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:30 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1735999683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:30.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:30 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2942843585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 -- 172.21.15.116:0/3878033532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344104ba0 msgr2=0x7f9344106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 --2- 172.21.15.116:0/3878033532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344104ba0 0x7f9344106f90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f932c009900 tx=0x7f932c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 -- 172.21.15.116:0/3878033532 shutdown_connections 2024-03-24T09:21:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 --2- 172.21.15.116:0/3878033532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93441074d0 0x7f93441098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 --2- 172.21.15.116:0/3878033532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344104ba0 0x7f9344106f90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 --2- 172.21.15.116:0/3878033532 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9344102270 0x7f9344104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 -- 172.21.15.116:0/3878033532 >> 172.21.15.116:0/3878033532 conn(0x7f93440fbbc0 msgr2=0x7f93440fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 -- 172.21.15.116:0/3878033532 shutdown_connections 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 -- 172.21.15.116:0/3878033532 wait complete. 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 Processor -- start 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 -- start start 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.415+0000 7f934a0eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9344102270 0x7f9344197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9344104ba0 0x7f9344197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 0x7f934419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f934419f230 con 0x7f93441074d0 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f934419f3a0 con 0x7f9344102270 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f934419f560 con 0x7f9344104ba0 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 0x7f934419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 0x7f934419ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33168/0 (socket says 172.21.15.116:33168) 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f93437fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9344102270 0x7f9344197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 -- 172.21.15.116:0/1876792223 learned_addr learned my addr 172.21.15.116:0/1876792223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f933affd640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9344104ba0 0x7f9344197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 -- 172.21.15.116:0/1876792223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9344104ba0 msgr2=0x7f9344197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9344104ba0 0x7f9344197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 -- 172.21.15.116:0/1876792223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9344102270 msgr2=0x7f9344197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9344102270 0x7f9344197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 -- 172.21.15.116:0/1876792223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9334009490 con 0x7f93441074d0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f9343fff640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 0x7f934419ecf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f933c00eb10 tx=0x7f933c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c004b20 con 0x7f93441074d0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f933c004c80 con 0x7f93441074d0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c009560 con 0x7f93441074d0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c010690 con 0x7f93441074d0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.416+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f934419fb40 con 0x7f93441074d0 2024-03-24T09:21:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.417+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93440fd6b0 con 0x7f93441074d0 2024-03-24T09:21:30.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.418+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f933c005020 con 0x7f93441074d0 2024-03-24T09:21:30.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.419+0000 7f93417fa640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93240a2150 0x7f93240a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.419+0000 7f93437fe640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93240a2150 0x7f93240a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.419+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f933c014070 con 0x7f93441074d0 2024-03-24T09:21:30.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.419+0000 7f93437fe640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93240a2150 0x7f93240a4610 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9334009d60 tx=0x7f9334005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:30.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.423+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f933c097250 con 0x7f93441074d0 2024-03-24T09:21:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:30 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2280961798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:30 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1248191890' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:30 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1735999683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:30 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2942843585' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:30.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.597+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f934402d1d0 con 0x7f93441074d0 2024-03-24T09:21:30.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.597+0000 7f93417fa640 1 -- 172.21.15.116:0/1876792223 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f933c09a910 con 0x7f93441074d0 2024-03-24T09:21:30.598 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:30.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93240a2150 msgr2=0x7f93240a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93240a2150 0x7f93240a4610 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9334009d60 tx=0x7f9334005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:30.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 msgr2=0x7f934419ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 0x7f934419ecf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f933c00eb10 tx=0x7f933c00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 shutdown_connections 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93441074d0 0x7f934419ecf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f93240a2150 0x7f93240a4610 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9344104ba0 0x7f9344197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 --2- 172.21.15.116:0/1876792223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9344102270 0x7f9344197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 >> 172.21.15.116:0/1876792223 conn(0x7f93440fbbc0 msgr2=0x7f9344102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 shutdown_connections 2024-03-24T09:21:30.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.599+0000 7f934a0eb640 1 -- 172.21.15.116:0/1876792223 wait complete. 2024-03-24T09:21:30.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:21:30.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 -- 172.21.15.116:0/15324927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714100630 msgr2=0x7f5714100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 --2- 172.21.15.116:0/15324927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714100630 0x7f5714100a50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f570c009ca0 tx=0x7f570c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 -- 172.21.15.116:0/15324927 shutdown_connections 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 --2- 172.21.15.116:0/15324927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57141021e0 0x7f5714108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 --2- 172.21.15.116:0/15324927 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5714101820 0x7f5714101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 --2- 172.21.15.116:0/15324927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714100630 0x7f5714100a50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 -- 172.21.15.116:0/15324927 >> 172.21.15.116:0/15324927 conn(0x7f57140fbbc0 msgr2=0x7f57140fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 -- 172.21.15.116:0/15324927 shutdown_connections 2024-03-24T09:21:30.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.712+0000 7f5718e09640 1 -- 172.21.15.116:0/15324927 wait complete. 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 Processor -- start 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 -- start start 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5714100630 0x7f5714197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 0x7f5714197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57141021e0 0x7f571419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f571419f1f0 con 0x7f5714101820 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f571419f360 con 0x7f5714100630 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5718e09640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f571419f520 con 0x7f57141021e0 2024-03-24T09:21:30.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5711d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 0x7f5714197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5711d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 0x7f5714197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33182/0 (socket says 172.21.15.116:33182) 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5711d9b640 1 -- 172.21.15.116:0/1294937934 learned_addr learned my addr 172.21.15.116:0/1294937934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5714100630 0x7f5714197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5712d9d640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57141021e0 0x7f571419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 -- 172.21.15.116:0/1294937934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57141021e0 msgr2=0x7f571419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57141021e0 0x7f571419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 -- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 msgr2=0x7f5714197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 0x7f5714197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 -- 172.21.15.116:0/1294937934 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56fc009490 con 0x7f5714100630 2024-03-24T09:21:30.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f5711d9b640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 0x7f5714197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.713+0000 7f571259c640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5714100630 0x7f5714197700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f570c02c630 tx=0x7f570c032980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.714+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c034e70 con 0x7f5714100630 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.714+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c009900 con 0x7f5714100630 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.714+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f571419fb10 con 0x7f5714100630 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.714+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f570c003e20 con 0x7f5714100630 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.714+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c004800 con 0x7f5714100630 2024-03-24T09:21:30.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.715+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f571410bef0 con 0x7f5714100630 2024-03-24T09:21:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.716+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f570c03b040 con 0x7f5714100630 2024-03-24T09:21:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.716+0000 7f570b7fe640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f57000a2150 0x7f57000a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.716+0000 7f5711d9b640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f57000a2150 0x7f57000a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.716+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f570c0f0830 con 0x7f5714100630 2024-03-24T09:21:30.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.717+0000 7f5711d9b640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f57000a2150 0x7f57000a4610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5714198c10 tx=0x7f56fc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:30.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.724+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f570c0b9500 con 0x7f5714100630 2024-03-24T09:21:30.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.937+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f571410c0b0 con 0x7f5714100630 2024-03-24T09:21:30.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.938+0000 7f570b7fe640 1 -- 172.21.15.116:0/1294937934 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f570c0bcbc0 con 0x7f5714100630 2024-03-24T09:21:30.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:30.939 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:20:06.925494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:21:30.940 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:21:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f57000a2150 msgr2=0x7f57000a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f57000a2150 0x7f57000a4610 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5714198c10 tx=0x7f56fc005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5714100630 msgr2=0x7f5714197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5714100630 0x7f5714197700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f570c02c630 tx=0x7f570c032980 comp rx=0 tx=0).stop 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 shutdown_connections 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f57000a2150 0x7f57000a4610 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57141021e0 0x7f571419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5714101820 0x7f5714197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 --2- 172.21.15.116:0/1294937934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5714100630 0x7f5714197700 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 >> 172.21.15.116:0/1294937934 conn(0x7f57140fbbc0 msgr2=0x7f5714069d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:30.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 shutdown_connections 2024-03-24T09:21:30.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:30.940+0000 7f5718e09640 1 -- 172.21.15.116:0/1294937934 wait complete. 2024-03-24T09:21:30.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:21:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.054+0000 7f26b3c34640 1 -- 172.21.15.116:0/2349274793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac101820 msgr2=0x7f26ac101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.054+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2349274793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac101820 0x7f26ac101ca0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009900 tx=0x7f26a802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- 172.21.15.116:0/2349274793 shutdown_connections 2024-03-24T09:21:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2349274793 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26ac1021e0 0x7f26ac108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2349274793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac101820 0x7f26ac101ca0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2349274793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ac100630 0x7f26ac100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- 172.21.15.116:0/2349274793 >> 172.21.15.116:0/2349274793 conn(0x7f26ac0fbba0 msgr2=0x7f26ac0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- 172.21.15.116:0/2349274793 shutdown_connections 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- 172.21.15.116:0/2349274793 wait complete. 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 Processor -- start 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- start start 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ac100630 0x7f26ac196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26ac101820 0x7f26ac196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 0x7f26ac19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f26ac19e0f0 con 0x7f26ac1021e0 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f26ac19e260 con 0x7f26ac100630 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b3c34640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f26ac19e420 con 0x7f26ac101820 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b21d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 0x7f26ac19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b21d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 0x7f26ac19dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33206/0 (socket says 172.21.15.116:33206) 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b21d1640 1 -- 172.21.15.116:0/2921494115 learned_addr learned my addr 172.21.15.116:0/2921494115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b11cf640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26ac101820 0x7f26ac196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.055+0000 7f26b19d0640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ac100630 0x7f26ac196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b21d1640 1 -- 172.21.15.116:0/2921494115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26ac101820 msgr2=0x7f26ac196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b21d1640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26ac101820 0x7f26ac196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b21d1640 1 -- 172.21.15.116:0/2921494115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ac100630 msgr2=0x7f26ac196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b21d1640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ac100630 0x7f26ac196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b21d1640 1 -- 172.21.15.116:0/2921494115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a8009560 con 0x7f26ac1021e0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b21d1640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 0x7f26ac19dbb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26a400eb10 tx=0x7f26a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4004b20 con 0x7f26ac1021e0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f26a4004c80 con 0x7f26ac1021e0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ac19e700 con 0x7f26ac1021e0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4010690 con 0x7f26ac1021e0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.056+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ac19ebe0 con 0x7f26ac1021e0 2024-03-24T09:21:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.057+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ac107970 con 0x7f26ac1021e0 2024-03-24T09:21:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.058+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f26a4005020 con 0x7f26ac1021e0 2024-03-24T09:21:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.058+0000 7f269effd640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f26980a2150 0x7f26980a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.059+0000 7f26b19d0640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f26980a2150 0x7f26980a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.059+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f26a4014070 con 0x7f26ac1021e0 2024-03-24T09:21:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.059+0000 7f26b19d0640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f26980a2150 0x7f26980a4610 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f26ac101680 tx=0x7f26a0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.063+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f26a4097200 con 0x7f26ac1021e0 2024-03-24T09:21:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.237+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26ac10bef0 con 0x7f26ac1021e0 2024-03-24T09:21:31.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.238+0000 7f269effd640 1 -- 172.21.15.116:0/2921494115 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24669 (secure 0 0 0) 0x7f26a409a8c0 con 0x7f26ac1021e0 2024-03-24T09:21:31.239 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:31.239 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":91,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:20:35.020321+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:21:31.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f26980a2150 msgr2=0x7f26980a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f26980a2150 0x7f26980a4610 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f26ac101680 tx=0x7f26a0005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 msgr2=0x7f26ac19dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 0x7f26ac19dbb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f26a400eb10 tx=0x7f26a400c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 shutdown_connections 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26ac1021e0 0x7f26ac19dbb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f26980a2150 0x7f26980a4610 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26ac101820 0x7f26ac196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 --2- 172.21.15.116:0/2921494115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26ac100630 0x7f26ac196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 >> 172.21.15.116:0/2921494115 conn(0x7f26ac0fbba0 msgr2=0x7f26ac104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 shutdown_connections 2024-03-24T09:21:31.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.240+0000 7f26b3c34640 1 -- 172.21.15.116:0/2921494115 wait complete. 2024-03-24T09:21:31.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:31 smithi116 ceph-mon[32207]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:31.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:31 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3960372234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:31.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:31 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1876792223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:31.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:31 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1294937934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:31.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:21:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:31 smithi143 ceph-mon[35965]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:31 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3960372234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:31 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1876792223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:31 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1294937934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:31 smithi117 ceph-mon[35788]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-24T09:21:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:31 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3960372234' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:31 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1876792223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:31 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1294937934' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/3269459857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68100630 msgr2=0x7f3d68100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/3269459857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68100630 0x7f3d68100a50 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009900 tx=0x7f3d5802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/3269459857 shutdown_connections 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/3269459857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d681021e0 0x7f3d68108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/3269459857 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68101820 0x7f3d68101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/3269459857 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68100630 0x7f3d68100a50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/3269459857 >> 172.21.15.116:0/3269459857 conn(0x7f3d680fbba0 msgr2=0x7f3d680fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/3269459857 shutdown_connections 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.354+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/3269459857 wait complete. 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 Processor -- start 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 -- start start 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 0x7f3d680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68101820 0x7f3d6819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d681021e0 0x7f3d6819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6819f240 con 0x7f3d68101820 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6819f3b0 con 0x7f3d681021e0 2024-03-24T09:21:31.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ef3e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6819f570 con 0x7f3d68100630 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d67fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68101820 0x7f3d6819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ccda640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 0x7f3d680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ccda640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 0x7f3d680693e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41102/0 (socket says 172.21.15.116:41102) 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ccda640 1 -- 172.21.15.116:0/1671475334 learned_addr learned my addr 172.21.15.116:0/1671475334 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6d4db640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d681021e0 0x7f3d6819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d67fff640 1 -- 172.21.15.116:0/1671475334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 msgr2=0x7f3d680693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d67fff640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 0x7f3d680693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d67fff640 1 -- 172.21.15.116:0/1671475334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d681021e0 msgr2=0x7f3d6819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d67fff640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d681021e0 0x7f3d6819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d67fff640 1 -- 172.21.15.116:0/1671475334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d58009560 con 0x7f3d68101820 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.355+0000 7f3d6ccda640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 0x7f3d680693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.356+0000 7f3d67fff640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68101820 0x7f3d6819c440 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000e9f0 tx=0x7f3d5000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.356+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5000ccc0 con 0x7f3d68101820 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.356+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6819f7f0 con 0x7f3d68101820 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.356+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d5000ce20 con 0x7f3d68101820 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.356+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d500187a0 con 0x7f3d68101820 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.356+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6819fd00 con 0x7f3d68101820 2024-03-24T09:21:31.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.357+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d681079d0 con 0x7f3d68101820 2024-03-24T09:21:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.358+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f3d50010430 con 0x7f3d68101820 2024-03-24T09:21:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.359+0000 7f3d65ffb640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3d540a2150 0x7f3d540a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.359+0000 7f3d6ccda640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3d540a2150 0x7f3d540a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.359+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3d50014070 con 0x7f3d68101820 2024-03-24T09:21:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.359+0000 7f3d6ccda640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3d540a2150 0x7f3d540a4610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d58005940 tx=0x7f3d58005890 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.362+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3d500974c0 con 0x7f3d68101820 2024-03-24T09:21:31.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.574+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d6810bef0 con 0x7f3d68101820 2024-03-24T09:21:31.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.575+0000 7f3d65ffb640 1 -- 172.21.15.116:0/1671475334 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f3d5009ab80 con 0x7f3d68101820 2024-03-24T09:21:31.576 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:31.576 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:20:06.925494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:21:31.576 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:21:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3d540a2150 msgr2=0x7f3d540a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3d540a2150 0x7f3d540a4610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d58005940 tx=0x7f3d58005890 comp rx=0 tx=0).stop 2024-03-24T09:21:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68101820 msgr2=0x7f3d6819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68101820 0x7f3d6819c440 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000e9f0 tx=0x7f3d5000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 shutdown_connections 2024-03-24T09:21:31.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d681021e0 0x7f3d6819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d68101820 0x7f3d6819c440 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3d540a2150 0x7f3d540a4610 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 --2- 172.21.15.116:0/1671475334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d68100630 0x7f3d680693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 >> 172.21.15.116:0/1671475334 conn(0x7f3d680fbba0 msgr2=0x7f3d68104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 shutdown_connections 2024-03-24T09:21:31.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.577+0000 7f3d6ef3e640 1 -- 172.21.15.116:0/1671475334 wait complete. 2024-03-24T09:21:31.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T09:19:38.431545+0000', 'modified': '2024-03-24T09:20:06.925494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24515, 'mds_1': 24484, 'mds_2': 24451, 'mds_3': 24443, 'mds_4': 24422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1877964932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1877964932}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1877964932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14649': {'gid': 14649, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6841/891659164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6840', 'nonce': 891659164}, {'type': 'v1', 'addr': '172.21.15.116:6841', 'nonce': 891659164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6833/2011428167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2011428167}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2011428167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.117:6833/3849440293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6832', 'nonce': 3849440293}, {'type': 'v1', 'addr': '172.21.15.117:6833', 'nonce': 3849440293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.117:6835/1932277434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1932277434}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1932277434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.143:6835/1036797529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1036797529}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1036797529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6837/3300095096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 3300095096}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 3300095096}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6837/309438710', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 309438710}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 309438710}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.143:6839/3139487666', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6838', 'nonce': 3139487666}, {'type': 'v1', 'addr': '172.21.15.143:6839', 'nonce': 3139487666}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6839/3950675281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3950675281}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3950675281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24515, 'qdb_cluster': [24484, 24515, 24451, 24443, 24422]} 2024-03-24T09:21:31.589 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T09:21:31.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-24T09:21:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 -- 172.21.15.116:0/3716918447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 msgr2=0x7f74e4109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 --2- 172.21.15.116:0/3716918447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 0x7f74e4109840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009ca0 tx=0x7f74dc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 -- 172.21.15.116:0/3716918447 shutdown_connections 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 --2- 172.21.15.116:0/3716918447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 0x7f74e4109840 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 --2- 172.21.15.116:0/3716918447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74e4104ac0 0x7f74e4106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 --2- 172.21.15.116:0/3716918447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e4104580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.691+0000 7f74ea44e640 1 -- 172.21.15.116:0/3716918447 >> 172.21.15.116:0/3716918447 conn(0x7f74e40fbba0 msgr2=0x7f74e40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 -- 172.21.15.116:0/3716918447 shutdown_connections 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 -- 172.21.15.116:0/3716918447 wait complete. 2024-03-24T09:21:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 Processor -- start 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 -- start start 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e41976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74e4104ac0 0x7f74e4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 0x7f74e419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f74e419f1d0 con 0x7f74e4102190 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f74e419f340 con 0x7f74e4104ac0 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74ea44e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f74e419f500 con 0x7f74e41073f0 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e41976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e41976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33238/0 (socket says 172.21.15.116:33238) 2024-03-24T09:21:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74e3fff640 1 -- 172.21.15.116:0/2072643115 learned_addr learned my addr 172.21.15.116:0/2072643115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.692+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 0x7f74e419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e37fe640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74e4104ac0 0x7f74e4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e3fff640 1 -- 172.21.15.116:0/2072643115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 msgr2=0x7f74e419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 0x7f74e419ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e3fff640 1 -- 172.21.15.116:0/2072643115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74e4104ac0 msgr2=0x7f74e4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74e4104ac0 0x7f74e4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e3fff640 1 -- 172.21.15.116:0/2072643115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc009900 con 0x7f74e4102190 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e41976e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f74d400e9f0 tx=0x7f74d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e17fa640 1 -- 172.21.15.116:0/2072643115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4009700 con 0x7f74e4102190 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e419f7e0 con 0x7f74e4102190 2024-03-24T09:21:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e17fa640 1 -- 172.21.15.116:0/2072643115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74d400cd80 con 0x7f74e4102190 2024-03-24T09:21:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74e17fa640 1 -- 172.21.15.116:0/2072643115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4018840 con 0x7f74e4102190 2024-03-24T09:21:31.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.693+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e419fd40 con 0x7f74e4102190 2024-03-24T09:21:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.695+0000 7f74e17fa640 1 -- 172.21.15.116:0/2072643115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f74d40189a0 con 0x7f74e4102190 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 Processor -- start 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 -- start start 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603560 0x557c91603980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603f50 0x557c91608a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c916090c0 0x557c9160b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557c91572180 con 0x557c91603560 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557c9156f6b0 con 0x557c916090c0 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74ed02d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557c9156ae90 con 0x557c91603f50 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e17fa640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f74cc0a2150 0x7f74cc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603560 0x557c91603980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603560 0x557c91603980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33242/0 (socket says 172.21.15.116:33242) 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e89eb640 1 -- 172.21.15.116:0/3260071101 learned_addr learned my addr 172.21.15.116:0/3260071101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e3fff640 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603f50 0x557c91608a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c916090c0 0x557c9160b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f74cc0a2150 0x7f74cc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e17fa640 1 -- 172.21.15.116:0/2072643115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74d4014070 con 0x7f74e4102190 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 -- 172.21.15.116:0/3260071101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603f50 msgr2=0x557c91608a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603f50 0x557c91608a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 -- 172.21.15.116:0/3260071101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603560 msgr2=0x557c91603980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603560 0x557c91603980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 -- 172.21.15.116:0/3260071101 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c91527d80 con 0x557c916090c0 2024-03-24T09:21:31.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f74cc0a2150 0x7f74cc0a4610 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x557c916047b0 tx=0x7f74dc0325d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.696+0000 7f74e37fe640 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c916090c0 0x557c9160b550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f74e40693d0 tx=0x7f74d8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=792ff44811df395d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ca7fc640 1 -- 172.21.15.116:0/3260071101 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8004230 con 0x557c916090c0 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ca7fc640 1 -- 172.21.15.116:0/3260071101 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74d8009e10 con 0x557c916090c0 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ca7fc640 1 -- 172.21.15.116:0/3260071101 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8004ba0 con 0x557c916090c0 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 -- 172.21.15.116:0/3260071101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c916090c0 msgr2=0x557c9160b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c916090c0 0x557c9160b550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f74e40693d0 tx=0x7f74d8011810 comp rx=0 tx=0).stop 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 -- 172.21.15.116:0/3260071101 shutdown_connections 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c916090c0 0x557c9160b550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603f50 0x557c91608a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 --2- 172.21.15.116:0/3260071101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603560 0x557c91603980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 -- 172.21.15.116:0/3260071101 >> 172.21.15.116:0/3260071101 conn(0x557c915fead0 msgr2=0x557c915feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 -- 172.21.15.116:0/3260071101 shutdown_connections 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.697+0000 7f74ed02d740 1 -- 172.21.15.116:0/3260071101 wait complete. 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 -- start start 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 Processor -- start 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c9160d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603f50 0x557c9160da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 0x557c91614ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557c9150fd50 con 0x557c916090c0 2024-03-24T09:21:31.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557c9123e310 con 0x557c91603f50 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557c914f08f0 con 0x557c91603560 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74ed02d740 10 client.0 _ll_register_callbacks cb 0x557c9127bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 0x557c91614ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c9160d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 0x557c91614ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33252/0 (socket says 172.21.15.116:33252) 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e89eb640 1 -- 172.21.15.116:0/774349233 learned_addr learned my addr 172.21.15.116:0/774349233 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c9160d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41154/0 (socket says 172.21.15.116:41154) 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e37fe640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603f50 0x557c9160da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 -- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603f50 msgr2=0x557c9160da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603f50 0x557c9160da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 -- 172.21.15.116:0/774349233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 msgr2=0x557c91614ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 0x557c91614ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e3fff640 1 -- 172.21.15.116:0/774349233 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d8009830 con 0x557c91603560 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.698+0000 7f74e89eb640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 0x557c91614ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74e3fff640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c9160d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74d40cbaf0 tx=0x7f74d406ad30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f460bda4253a1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74c37fe640 1 -- 172.21.15.116:0/774349233 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40a34b0 con 0x557c91603560 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74c37fe640 1 -- 172.21.15.116:0/774349233 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74d40cbd40 con 0x557c91603560 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557c9126a6c0 con 0x557c91603560 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74c37fe640 1 -- 172.21.15.116:0/774349233 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74d40cbf00 con 0x557c91603560 2024-03-24T09:21:31.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 10 client.24970 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c91569510 con 0x557c91603560 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74c37fe640 1 -- 172.21.15.116:0/774349233 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74d40b7080 con 0x557c91603560 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74c37fe640 1 -- 172.21.15.116:0/774349233 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f74d4099d10 con 0x557c91603560 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 10 client.24970 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 10 client.24970 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 20 client.24970 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c91619dc0 0x557c9161b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 4 client.24970 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.699+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c912bb270 con 0x557c91619dc0 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.700+0000 7f74e37fe640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c91619dc0 0x557c9161b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.700+0000 7f74c37fe640 10 client.24970 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.700+0000 7f74e37fe640 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c91619dc0 0x557c9161b230 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bdc646015125d0c9 server_cookie=237653bfcd13b45 in_seq=0 out_seq=0 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74c37fe640 1 -- 172.21.15.116:0/774349233 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c912bb270 con 0x557c91619dc0 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74c37fe640 10 client.24970 handle_command_reply: tid=0 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 client.24970 shutdown 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 msgr2=0x557c9160d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c9160d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f74d40cbaf0 tx=0x7f74d406ad30 comp rx=0 tx=0).stop 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 shutdown_connections 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c91619dc0 0x557c9161b230 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c916090c0 0x557c91614ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603f50 0x557c9160da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74c37fe640 0 client.24970 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 --2- 172.21.15.116:0/774349233 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c9160d530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 >> 172.21.15.116:0/774349233 conn(0x557c915fead0 msgr2=0x557c915fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 shutdown_connections 2024-03-24T09:21:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 1 -- 172.21.15.116:0/774349233 wait complete. 2024-03-24T09:21:31.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.701+0000 7f74ed02d740 20 client.24970 trim_cache size 0 max 16384 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 Processor -- start 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 -- start start 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c915fead0 0x557c9160cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c91614a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c9160d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557c912bb270 con 0x557c91603f50 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557c9161a1a0 con 0x557c915fead0 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74ed02d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557c916047e0 con 0x557c91603560 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c9160d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c9160d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33268/0 (socket says 172.21.15.116:33268) 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74e89eb640 1 -- 172.21.15.116:0/2392226668 learned_addr learned my addr 172.21.15.116:0/2392226668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c915fead0 0x557c9160cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.719+0000 7f74e37fe640 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c91614a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74e89eb640 1 -- 172.21.15.116:0/2392226668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 msgr2=0x557c91614a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c91614a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74e89eb640 1 -- 172.21.15.116:0/2392226668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c915fead0 msgr2=0x557c9160cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c915fead0 0x557c9160cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74e89eb640 1 -- 172.21.15.116:0/2392226668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c91505a80 con 0x557c91603f50 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c9160d140 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0098d0 tx=0x7f74dc02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe605881d527f2a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74c9ffb640 1 -- 172.21.15.116:0/2392226668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc034a60 con 0x557c91603f50 2024-03-24T09:21:31.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74c9ffb640 1 -- 172.21.15.116:0/2392226668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74dc003e20 con 0x557c91603f50 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74c9ffb640 1 -- 172.21.15.116:0/2392226668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc004600 con 0x557c91603f50 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74ed02d740 1 -- 172.21.15.116:0/2392226668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 msgr2=0x557c9160d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.720+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c9160d140 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f74dc0098d0 tx=0x7f74dc02e040 comp rx=0 tx=0).stop 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- 172.21.15.116:0/2392226668 shutdown_connections 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c9160d140 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c91603560 0x557c91614a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2392226668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c915fead0 0x557c9160cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- 172.21.15.116:0/2392226668 >> 172.21.15.116:0/2392226668 conn(0x557c91608bc0 msgr2=0x557c915feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- 172.21.15.116:0/2392226668 shutdown_connections 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- 172.21.15.116:0/2392226668 wait complete. 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- start start 2024-03-24T09:21:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 Processor -- start 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c915fead0 0x557c91610650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603560 0x557c91610b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c91613bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557c916140f0 con 0x557c91603f50 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557c91614260 con 0x557c91603560 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557c91614420 con 0x557c915fead0 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74ed02d740 10 client.0 _ll_register_callbacks cb 0x7f74d400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c91613bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74e3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c915fead0 0x557c91610650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74e89eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c91613bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33272/0 (socket says 172.21.15.116:33272) 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74e89eb640 1 -- 172.21.15.116:0/2821018070 learned_addr learned my addr 172.21.15.116:0/2821018070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.721+0000 7f74e37fe640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603560 0x557c91610b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e3fff640 1 -- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603560 msgr2=0x557c91610b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603560 0x557c91610b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e3fff640 1 -- 172.21.15.116:0/2821018070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 msgr2=0x557c91613bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c91613bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e3fff640 1 -- 172.21.15.116:0/2821018070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e410a420 con 0x557c915fead0 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e89eb640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c91613bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74e3fff640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c915fead0 0x557c91610650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74d4099070 tx=0x7f74d4010d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace926daca49c936 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:21:31.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74c37fe640 1 -- 172.21.15.116:0/2821018070 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40a34b0 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d8009e10 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74c37fe640 1 -- 172.21.15.116:0/2821018070 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f74d4099c80 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74c37fe640 1 -- 172.21.15.116:0/2821018070 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f74d4099e60 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74ed02d740 10 client.24976 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c9152b7e0 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74c37fe640 1 -- 172.21.15.116:0/2821018070 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f74d4064020 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.722+0000 7f74c37fe640 1 -- 172.21.15.116:0/2821018070 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f74d40b3a40 con 0x557c915fead0 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 10 client.24976 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 10 client.24976 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 20 client.24976 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:31.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c916090c0 0x557c916831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 4 client.24976 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "status"}) -- 0x557c912a7610 con 0x557c916090c0 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74e37fe640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c916090c0 0x557c916831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74c37fe640 10 client.24976 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74e37fe640 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c916090c0 0x557c916831e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=20d3e635ec0f55f server_cookie=28e5b065dba99b16 in_seq=0 out_seq=0 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74c37fe640 1 -- 172.21.15.116:0/2821018070 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557c912a7610 con 0x557c916090c0 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74c37fe640 10 client.24976 handle_command_reply: tid=0 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 client.24976 shutdown 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c915fead0 msgr2=0x557c91610650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c915fead0 0x557c91610650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f74d4099070 tx=0x7f74d4010d80 comp rx=0 tx=0).stop 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 shutdown_connections 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557c916090c0 0x557c916831e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557c91603f50 0x557c91613bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74c37fe640 0 client.24976 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557c91603560 0x557c91610b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 --2- 172.21.15.116:0/2821018070 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557c915fead0 0x557c91610650 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.723+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 >> 172.21.15.116:0/2821018070 conn(0x557c91608bc0 msgr2=0x557c915feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 shutdown_connections 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ed02d740 1 -- 172.21.15.116:0/2821018070 wait complete. 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ed02d740 20 client.24976 trim_cache size 0 max 16384 2024-03-24T09:21:31.734 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 4, 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24422, 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 85.813166738999996, 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 91, 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 147.70528787500001 2024-03-24T09:21:31.735 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f74cc0a2150 msgr2=0x7f74cc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f74cc0a2150 0x7f74cc0a4610 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x557c916047b0 tx=0x7f74dc0325d0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 msgr2=0x7f74e41976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e41976e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f74d400e9f0 tx=0x7f74d400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 shutdown_connections 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f74cc0a2150 0x7f74cc0a4610 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f74e41073f0 0x7f74e419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74e4104ac0 0x7f74e4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 --2- 172.21.15.116:0/2072643115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4102190 0x7f74e41976e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 >> 172.21.15.116:0/2072643115 conn(0x7f74e40fbba0 msgr2=0x7f74e4102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 shutdown_connections 2024-03-24T09:21:31.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.724+0000 7f74ea44e640 1 -- 172.21.15.116:0/2072643115 wait complete. 2024-03-24T09:21:31.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-24T09:21:31.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 -- 172.21.15.116:0/2467987038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c100630 msgr2=0x7f3a8c100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 --2- 172.21.15.116:0/2467987038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c100630 0x7f3a8c100a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009900 tx=0x7f3a7402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 -- 172.21.15.116:0/2467987038 shutdown_connections 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 --2- 172.21.15.116:0/2467987038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a8c1021e0 0x7f3a8c108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 --2- 172.21.15.116:0/2467987038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c101820 0x7f3a8c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 --2- 172.21.15.116:0/2467987038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c100630 0x7f3a8c100a50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.840+0000 7f3a92a15640 1 -- 172.21.15.116:0/2467987038 >> 172.21.15.116:0/2467987038 conn(0x7f3a8c0fbba0 msgr2=0x7f3a8c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 -- 172.21.15.116:0/2467987038 shutdown_connections 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 -- 172.21.15.116:0/2467987038 wait complete. 2024-03-24T09:21:31.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 Processor -- start 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 -- start start 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a8c100630 0x7f3a8c1966c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 0x7f3a8c196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c1021e0 0x7f3a8c19dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c19e1b0 con 0x7f3a8c101820 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c19e320 con 0x7f3a8c1021e0 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.841+0000 7f3a92a15640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c19e4e0 con 0x7f3a8c100630 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 0x7f3a8c196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 0x7f3a8c196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33298/0 (socket says 172.21.15.116:33298) 2024-03-24T09:21:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3719266573 learned_addr learned my addr 172.21.15.116:0/3719266573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c1021e0 0x7f3a8c19dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a8c100630 0x7f3a8c1966c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3719266573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a8c100630 msgr2=0x7f3a8c1966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a8c100630 0x7f3a8c1966c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3719266573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c1021e0 msgr2=0x7f3a8c19dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c1021e0 0x7f3a8c19dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3719266573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a74009560 con 0x7f3a8c101820 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 0x7f3a8c196c00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00eb00 tx=0x7f3a7c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a897fa640 1 -- 172.21.15.116:0/3719266573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c013070 con 0x7f3a8c101820 2024-03-24T09:21:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a897fa640 1 -- 172.21.15.116:0/3719266573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a7c00bc30 con 0x7f3a8c101820 2024-03-24T09:21:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8c069a10 con 0x7f3a8c101820 2024-03-24T09:21:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a897fa640 1 -- 172.21.15.116:0/3719266573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c016610 con 0x7f3a8c101820 2024-03-24T09:21:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.842+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a8c069f70 con 0x7f3a8c101820 2024-03-24T09:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a897fa640 1 -- 172.21.15.116:0/3719266573 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f3a7c017410 con 0x7f3a8c101820 2024-03-24T09:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 Processor -- start 2024-03-24T09:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 -- start start 2024-03-24T09:21:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 0x5570eaf8a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8af50 0x5570eaf8fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf92550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5570eaef9180 con 0x5570eaf900c0 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5570eaef66b0 con 0x5570eaf8a560 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a955f5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5570eaef1e90 con 0x5570eaf8af50 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf92550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a90fb2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 0x5570eaf8a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf92550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33312/0 (socket says 172.21.15.116:33312) 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3687207336 learned_addr learned my addr 172.21.15.116:0/3687207336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a90fb2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 0x5570eaf8a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:49244/0 (socket says 172.21.15.116:49244) 2024-03-24T09:21:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8af50 0x5570eaf8fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.845+0000 7f3a897fa640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3a780a21d0 0x7f3a780a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3a780a21d0 0x7f3a780a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a897fa640 1 -- 172.21.15.116:0/3719266573 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3a7c0cf8a0 con 0x7f3a8c101820 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3687207336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8af50 msgr2=0x5570eaf8fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8af50 0x5570eaf8fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3687207336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 msgr2=0x5570eaf8a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 0x5570eaf8a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8b7fe640 1 -- 172.21.15.116:0/3687207336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570eaeaed80 con 0x5570eaf900c0 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8af50 0x5570eaf8fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 0x5570eaf8a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf92550 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c098b90 tx=0x7f3a7c0677b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f344e70896d93d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a827fc640 1 -- 172.21.15.116:0/3687207336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c09bb20 con 0x5570eaf900c0 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3a780a21d0 0x7f3a780a4690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5570eaf8b7b0 tx=0x7f3a84004ab0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a827fc640 1 -- 172.21.15.116:0/3687207336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a7c017950 con 0x5570eaf900c0 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.846+0000 7f3a827fc640 1 -- 172.21.15.116:0/3687207336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c0cd140 con 0x5570eaf900c0 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- 172.21.15.116:0/3687207336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 msgr2=0x5570eaf92550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf92550 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c098b90 tx=0x7f3a7c0677b0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- 172.21.15.116:0/3687207336 shutdown_connections 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf92550 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8af50 0x5570eaf8fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- 172.21.15.116:0/3687207336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8a560 0x5570eaf8a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- 172.21.15.116:0/3687207336 >> 172.21.15.116:0/3687207336 conn(0x5570eaf85ad0 msgr2=0x5570eaf85ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- 172.21.15.116:0/3687207336 shutdown_connections 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- 172.21.15.116:0/3687207336 wait complete. 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- start start 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 Processor -- start 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf94470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf949b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf9b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5570eae96d50 con 0x5570eaf900c0 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5570eabc5310 con 0x5570eaf8af50 2024-03-24T09:21:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.847+0000 7f3a955f5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5570eae778f0 con 0x5570eaf8a560 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a955f5740 10 client.0 _ll_register_callbacks cb 0x5570eac02ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf9b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf9b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33316/0 (socket says 172.21.15.116:33316) 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 -- 172.21.15.116:0/2208654692 learned_addr learned my addr 172.21.15.116:0/2208654692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a8b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf949b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf94470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 -- 172.21.15.116:0/2208654692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 msgr2=0x5570eaf94470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf94470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 -- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 msgr2=0x5570eaf949b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf949b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 -- 172.21.15.116:0/2208654692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a8c069f70 con 0x5570eaf900c0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf9b9f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400fcb0 tx=0x7f3a84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54e281f0722d0d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2208654692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a840047d0 con 0x5570eaf900c0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2208654692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3a84021470 con 0x5570eaf900c0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5570eabf16c0 con 0x5570eaf900c0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2208654692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3a84021670 con 0x5570eaf900c0 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a955f5740 10 client.15249 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5570eaef0510 con 0x5570eaf900c0 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.848+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2208654692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5570eabf16c0 con 0x5570eaf900c0 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2208654692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3a840106f0 con 0x5570eaf900c0 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a955f5740 10 client.15249 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a955f5740 10 client.15249 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a955f5740 20 client.15249 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eafa0df0 0x5570eafa2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a955f5740 4 client.15249 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570eac42270 con 0x5570eafa0df0 2024-03-24T09:21:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eafa0df0 0x5570eafa2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a6b7fe640 10 client.15249 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.849+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eafa0df0 0x5570eafa2260 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef9be8d20b38cc5 server_cookie=f91a841d1348d909 in_seq=0 out_seq=0 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2208654692 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570eac42270 con 0x5570eafa0df0 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a6b7fe640 10 client.15249 handle_command_reply: tid=0 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 client.15249 shutdown 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 msgr2=0x5570eaf9b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf9b9f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3a8400fcb0 tx=0x7f3a84004a40 comp rx=0 tx=0).stop 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 shutdown_connections 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eafa0df0 0x5570eafa2260 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf900c0 0x5570eaf9b9f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf949b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2208654692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf94470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a6b7fe640 0 client.15249 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 >> 172.21.15.116:0/2208654692 conn(0x5570eaf85ad0 msgr2=0x5570eaf84ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 shutdown_connections 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 1 -- 172.21.15.116:0/2208654692 wait complete. 2024-03-24T09:21:31.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.850+0000 7f3a955f5740 20 client.15249 trim_cache size 0 max 16384 2024-03-24T09:21:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 Processor -- start 2024-03-24T09:21:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 -- start start 2024-03-24T09:21:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eaf9fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf88d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf89270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5570eaf85990 con 0x5570eaf85ad0 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5570eaf80ea0 con 0x5570eaf8af50 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.868+0000 7f3a955f5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5570eaf9c2b0 con 0x5570eaf8a560 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eaf9fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eaf9fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33328/0 (socket says 172.21.15.116:33328) 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 -- 172.21.15.116:0/1310348692 learned_addr learned my addr 172.21.15.116:0/1310348692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf89270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf88d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 -- 172.21.15.116:0/1310348692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 msgr2=0x5570eaf88d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf88d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 -- 172.21.15.116:0/1310348692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 msgr2=0x5570eaf89270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf89270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 -- 172.21.15.116:0/1310348692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570eae8ca80 con 0x5570eaf85ad0 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf89270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eaf9fd90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3a74005ab0 tx=0x7f3a74009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e6be35760bc2bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a81ffb640 1 -- 172.21.15.116:0/1310348692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7404ab10 con 0x5570eaf85ad0 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a81ffb640 1 -- 172.21.15.116:0/1310348692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a7402de90 con 0x5570eaf85ad0 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.869+0000 7f3a81ffb640 1 -- 172.21.15.116:0/1310348692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74004430 con 0x5570eaf85ad0 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 -- 172.21.15.116:0/1310348692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 msgr2=0x5570eaf9fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eaf9fd90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3a74005ab0 tx=0x7f3a74009260 comp rx=0 tx=0).stop 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 -- 172.21.15.116:0/1310348692 shutdown_connections 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eaf89270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eaf88d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- 172.21.15.116:0/1310348692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eaf9fd90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 -- 172.21.15.116:0/1310348692 >> 172.21.15.116:0/1310348692 conn(0x5570eaf8fbc0 msgr2=0x5570eaf85eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 -- 172.21.15.116:0/1310348692 shutdown_connections 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 -- 172.21.15.116:0/1310348692 wait complete. 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 -- start start 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 Processor -- start 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eb002d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eb0032d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.870+0000 7f3a955f5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eb009d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5570eaf8bb80 con 0x5570eaf85ad0 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5570eafa0950 con 0x5570eaf8af50 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5570eaf86a70 con 0x5570eaf8a560 2024-03-24T09:21:31.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 10 client.0 _ll_register_callbacks cb 0x7f3a840136d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eb002d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eb002d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33330/0 (socket says 172.21.15.116:33330) 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 -- 172.21.15.116:0/2255818057 learned_addr learned my addr 172.21.15.116:0/2255818057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eb0032d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a90fb2640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eb009d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 -- 172.21.15.116:0/2255818057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 msgr2=0x5570eb0032d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eb0032d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 -- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 msgr2=0x5570eb009d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eb009d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 -- 172.21.15.116:0/2255818057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570eaef0510 con 0x5570eaf85ad0 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a8bfff640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eb002d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3a7402c420 tx=0x7f3a7404ab10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970a9b7e16a4d2e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a6b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2255818057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7404adc0 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2255818057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3a74046040 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a7c017950 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2255818057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3a740046c0 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 10 client.15255 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.871+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x7f3a74030070 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2255818057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a7c017950 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2255818057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3a7404a050 con 0x5570eaf85ad0 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 10 client.15255 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:31.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 10 client.15255 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 20 client.15255 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eaf900c0 0x5570eb00e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 4 client.15255 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "status"}) -- 0x5570eac2e610 con 0x5570eaf900c0 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eaf900c0 0x5570eb00e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a6b7fe640 10 client.15255 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a8b7fe640 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eaf900c0 0x5570eb00e460 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=65aa505f0a5d88fd server_cookie=192406a3b3d032d9 in_seq=0 out_seq=0 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a6b7fe640 1 -- 172.21.15.116:0/2255818057 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5570eac2e610 con 0x5570eaf900c0 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a6b7fe640 10 client.15255 handle_command_reply: tid=0 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.872+0000 7f3a955f5740 1 client.15255 shutdown 2024-03-24T09:21:31.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 msgr2=0x5570eb002d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eb002d90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3a7402c420 tx=0x7f3a7404ab10 comp rx=0 tx=0).stop 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 shutdown_connections 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5570eaf900c0 0x5570eb00e460 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5570eaf8af50 0x5570eb009d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5570eaf8a560 0x5570eb0032d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a6b7fe640 0 client.15255 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 --2- 172.21.15.116:0/2255818057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5570eaf85ad0 0x5570eb002d90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 >> 172.21.15.116:0/2255818057 conn(0x5570eaf8fbc0 msgr2=0x5570eaf9fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 shutdown_connections 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 1 -- 172.21.15.116:0/2255818057 wait complete. 2024-03-24T09:21:31.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a955f5740 20 client.15255 trim_cache size 0 max 16384 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 3, 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24443, 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 86.970429871999997, 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 91, 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:31.884 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 139.16620317499999 2024-03-24T09:21:31.885 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3a780a21d0 msgr2=0x7f3a780a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3a780a21d0 0x7f3a780a4690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5570eaf8b7b0 tx=0x7f3a84004ab0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 msgr2=0x7f3a8c196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 0x7f3a8c196c00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00eb00 tx=0x7f3a7c009f20 comp rx=0 tx=0).stop 2024-03-24T09:21:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 shutdown_connections 2024-03-24T09:21:31.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c1021e0 0x7f3a8c19dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a8c101820 0x7f3a8c196c00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f3a780a21d0 0x7f3a780a4690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 --2- 172.21.15.116:0/3719266573 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a8c100630 0x7f3a8c1966c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 >> 172.21.15.116:0/3719266573 conn(0x7f3a8c0fbba0 msgr2=0x7f3a8c104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 shutdown_connections 2024-03-24T09:21:31.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.873+0000 7f3a92a15640 1 -- 172.21.15.116:0/3719266573 wait complete. 2024-03-24T09:21:31.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-24T09:21:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 -- 172.21.15.116:0/1302769260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268101820 msgr2=0x7f8268101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1302769260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268101820 0x7f8268101ca0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8254009900 tx=0x7f825402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 -- 172.21.15.116:0/1302769260 shutdown_connections 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1302769260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82681021e0 0x7f8268108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1302769260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268101820 0x7f8268101ca0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1302769260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268100630 0x7f8268100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 -- 172.21.15.116:0/1302769260 >> 172.21.15.116:0/1302769260 conn(0x7f82680fbbc0 msgr2=0x7f82680fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 -- 172.21.15.116:0/1302769260 shutdown_connections 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.990+0000 7f826ef8a640 1 -- 172.21.15.116:0/1302769260 wait complete. 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 Processor -- start 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 -- start start 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 0x7f826806a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268101820 0x7f826819c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82681021e0 0x7f826819ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f826819f1f0 con 0x7f8268100630 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f826819f360 con 0x7f82681021e0 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f826819f520 con 0x7f8268101820 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 0x7f826806a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 0x7f826806a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33352/0 (socket says 172.21.15.116:33352) 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 -- 172.21.15.116:0/1487088818 learned_addr learned my addr 172.21.15.116:0/1487088818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826d527640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82681021e0 0x7f826819ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f8267fff640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268101820 0x7f826819c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 -- 172.21.15.116:0/1487088818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268101820 msgr2=0x7f826819c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268101820 0x7f826819c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 -- 172.21.15.116:0/1487088818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82681021e0 msgr2=0x7f826819ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82681021e0 0x7f826819ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 -- 172.21.15.116:0/1487088818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8254009560 con 0x7f8268100630 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826cd26640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 0x7f826806a860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9c0 tx=0x7f825c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f8265ffb640 1 -- 172.21.15.116:0/1487088818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c0096e0 con 0x7f8268100630 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f8265ffb640 1 -- 172.21.15.116:0/1487088818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f825c0044e0 con 0x7f8268100630 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826819f800 con 0x7f8268100630 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.991+0000 7f8265ffb640 1 -- 172.21.15.116:0/1487088818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c010430 con 0x7f8268100630 2024-03-24T09:21:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.992+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826819fce0 con 0x7f8268100630 2024-03-24T09:21:32.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.993+0000 7f8265ffb640 1 -- 172.21.15.116:0/1487088818 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f825c003f60 con 0x7f8268100630 2024-03-24T09:21:32.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8265ffb640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f82580a21d0 0x7f82580a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 Processor -- start 2024-03-24T09:21:32.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 -- start start 2024-03-24T09:21:32.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8267fff640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f82580a21d0 0x7f82580a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8265ffb640 1 -- 172.21.15.116:0/1487088818 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f825c014070 con 0x7f8268100630 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31da980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31dfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31e2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5574e3149180 con 0x5574e31daf50 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5574e31466b0 con 0x5574e31da560 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8271b68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5574e3141e90 con 0x5574e31e00c0 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8267fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31dfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8267fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31dfa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33364/0 (socket says 172.21.15.116:33364) 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f8267fff640 1 -- 172.21.15.116:0/40437700 learned_addr learned my addr 172.21.15.116:0/40437700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f826cd26640 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.994+0000 7f826d527640 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31e2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f8267fff640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f82580a21d0 0x7f82580a4690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f825402c170 tx=0x7f82540046c0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f826cd26640 1 -- 172.21.15.116:0/40437700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 msgr2=0x5574e31e2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f826cd26640 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31e2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f826cd26640 1 -- 172.21.15.116:0/40437700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 msgr2=0x5574e31dfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f826cd26640 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31dfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f826cd26640 1 -- 172.21.15.116:0/40437700 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e30fed80 con 0x5574e31da560 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f8267fff640 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31dfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f826cd26640 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31da980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f825c0cbae0 tx=0x7f825c00cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c75df562b5bdb2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f8252ffd640 1 -- 172.21.15.116:0/40437700 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c09ad00 con 0x5574e31da560 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f8252ffd640 1 -- 172.21.15.116:0/40437700 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f825c09ee50 con 0x5574e31da560 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.995+0000 7f8252ffd640 1 -- 172.21.15.116:0/40437700 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c004210 con 0x5574e31da560 2024-03-24T09:21:32.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- 172.21.15.116:0/40437700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 msgr2=0x5574e31da980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31da980 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f825c0cbae0 tx=0x7f825c00cd80 comp rx=0 tx=0).stop 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- 172.21.15.116:0/40437700 shutdown_connections 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31e2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31dfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- 172.21.15.116:0/40437700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31da980 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- 172.21.15.116:0/40437700 >> 172.21.15.116:0/40437700 conn(0x5574e31d5ad0 msgr2=0x5574e31d5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- 172.21.15.116:0/40437700 shutdown_connections 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- 172.21.15.116:0/40437700 wait complete. 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- start start 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 Processor -- start 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31e4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31e4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31eba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5574e30e6d50 con 0x5574e31daf50 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5574e2e15310 con 0x5574e31da560 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5574e30c78f0 con 0x5574e31e00c0 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.996+0000 7f8271b68740 10 client.0 _ll_register_callbacks cb 0x5574e2e52ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8267fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31e4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826d527640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31e4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:49292/0 (socket says 172.21.15.116:49292) 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 -- 172.21.15.116:0/3873403731 learned_addr learned my addr 172.21.15.116:0/3873403731 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826d527640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31eba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41262/0 (socket says 172.21.15.116:41262) 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 -- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 msgr2=0x5574e31eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 -- 172.21.15.116:0/3873403731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 msgr2=0x5574e31e4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31e4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 -- 172.21.15.116:0/3873403731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826819fce0 con 0x5574e31da560 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8267fff640 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f826cd26640 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31e4500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f825c0cbb10 tx=0x7f825c0cbf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27616b5a3c1b524 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8247fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8247fff640 1 -- 172.21.15.116:0/3873403731 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c07f070 con 0x5574e31da560 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8247fff640 1 -- 172.21.15.116:0/3873403731 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f825c0a34a0 con 0x5574e31da560 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5574e2e416c0 con 0x5574e31da560 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8247fff640 1 -- 172.21.15.116:0/3873403731 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f825c0ce200 con 0x5574e31da560 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8271b68740 10 client.25106 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.997+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5574e3140510 con 0x5574e31da560 2024-03-24T09:21:32.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8247fff640 1 -- 172.21.15.116:0/3873403731 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5574e2e416c0 con 0x5574e31da560 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8247fff640 1 -- 172.21.15.116:0/3873403731 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f825c082020 con 0x5574e31da560 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8271b68740 10 client.25106 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8271b68740 10 client.25106 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8271b68740 20 client.25106 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8271b68740 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e31efbe0 0x5574e31f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8271b68740 4 client.25106 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574e2e92270 con 0x5574e31efbe0 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8267fff640 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e31efbe0 0x5574e31f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8247fff640 10 client.25106 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.998+0000 7f8267fff640 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e31efbe0 0x5574e31f1dd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3f3c1e1b8ac9cc42 server_cookie=44776d61e77fcf82 in_seq=0 out_seq=0 2024-03-24T09:21:32.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8247fff640 1 -- 172.21.15.116:0/3873403731 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5574e2e92270 con 0x5574e31efbe0 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8247fff640 10 client.25106 handle_command_reply: tid=0 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 client.25106 shutdown 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 msgr2=0x5574e31e4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31e4500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f825c0cbb10 tx=0x7f825c0cbf60 comp rx=0 tx=0).stop 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 shutdown_connections 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31e4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e31efbe0 0x5574e31f1dd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 --2- 172.21.15.116:0/3873403731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31da560 0x5574e31e4500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8247fff640 0 client.25106 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:32.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 >> 172.21.15.116:0/3873403731 conn(0x5574e31d5ad0 msgr2=0x5574e31d4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 shutdown_connections 2024-03-24T09:21:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 1 -- 172.21.15.116:0/3873403731 wait complete. 2024-03-24T09:21:32.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:31.999+0000 7f8271b68740 20 client.25106 trim_cache size 0 max 16384 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 Processor -- start 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 -- start start 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31da560 0x5574e31ec110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31eba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31e00c0 0x5574e31d5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5574e31f2000 con 0x5574e31daf50 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5574e31e5230 con 0x5574e31e00c0 2024-03-24T09:21:32.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8271b68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5574e31db870 con 0x5574e31da560 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.017+0000 7f8267fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31eba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31eba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33378/0 (socket says 172.21.15.116:33378) 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f826cd26640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31da560 0x5574e31ec110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 -- 172.21.15.116:0/1235284795 learned_addr learned my addr 172.21.15.116:0/1235284795 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f826d527640 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31e00c0 0x5574e31d5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 -- 172.21.15.116:0/1235284795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31da560 msgr2=0x5574e31ec110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31da560 0x5574e31ec110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 -- 172.21.15.116:0/1235284795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31e00c0 msgr2=0x5574e31d5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31e00c0 0x5574e31d5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 -- 172.21.15.116:0/1235284795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e30dca80 con 0x5574e31daf50 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f826cd26640 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31da560 0x5574e31ec110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f8267fff640 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31eba10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f825402c7d0 tx=0x7f8254046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c722c21ed9651511 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f82527fc640 1 -- 172.21.15.116:0/1235284795 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254053070 con 0x5574e31daf50 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f82527fc640 1 -- 172.21.15.116:0/1235284795 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f825402de90 con 0x5574e31daf50 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.018+0000 7f82527fc640 1 -- 172.21.15.116:0/1235284795 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254055810 con 0x5574e31daf50 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- 172.21.15.116:0/1235284795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 msgr2=0x5574e31eba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31eba10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f825402c7d0 tx=0x7f8254046000 comp rx=0 tx=0).stop 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- 172.21.15.116:0/1235284795 shutdown_connections 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31e00c0 0x5574e31d5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31daf50 0x5574e31eba10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- 172.21.15.116:0/1235284795 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31da560 0x5574e31ec110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- 172.21.15.116:0/1235284795 >> 172.21.15.116:0/1235284795 conn(0x5574e31d5ad0 msgr2=0x5574e31d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- 172.21.15.116:0/1235284795 shutdown_connections 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- 172.21.15.116:0/1235284795 wait complete. 2024-03-24T09:21:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- start start 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 Processor -- start 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 0x5574e31e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31daf50 0x5574e31e7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5574e31eacf0 con 0x5574e31da560 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5574e31eae60 con 0x5574e31daf50 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5574e31eb020 con 0x5574e31e00c0 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f8271b68740 10 client.0 _ll_register_callbacks cb 0x7f825c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.019+0000 7f826cd26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 0x5574e31e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826d527640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 0x5574e31e72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33390/0 (socket says 172.21.15.116:33390) 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 -- 172.21.15.116:0/417730108 learned_addr learned my addr 172.21.15.116:0/417730108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8267fff640 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31daf50 0x5574e31e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 -- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 msgr2=0x5574e31ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 -- 172.21.15.116:0/417730108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31daf50 msgr2=0x5574e31e7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31daf50 0x5574e31e7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 -- 172.21.15.116:0/417730108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826819fce0 con 0x5574e31da560 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f826cd26640 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 0x5574e31e72c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f825c0a3380 tx=0x7f825c0038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47815ff9df83ba90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8247fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8247fff640 1 -- 172.21.15.116:0/417730108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c07f030 con 0x5574e31da560 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8247fff640 1 -- 172.21.15.116:0/417730108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f825c080070 con 0x5574e31da560 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f825c09ee50 con 0x5574e31da560 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8247fff640 1 -- 172.21.15.116:0/417730108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f825c0a34a0 con 0x5574e31da560 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8271b68740 10 client.15273 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x7f8268109370 con 0x5574e31da560 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8247fff640 1 -- 172.21.15.116:0/417730108 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f825c09ee50 con 0x5574e31da560 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.020+0000 7f8247fff640 1 -- 172.21.15.116:0/417730108 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f825c082070 con 0x5574e31da560 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 10 client.15273 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 10 client.15273 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 20 client.15273 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e325ab10 0x5574e3259820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 4 client.15273 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "status"}) -- 0x5574e2e7e610 con 0x5574e325ab10 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8267fff640 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e325ab10 0x5574e3259820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8247fff640 10 client.15273 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8267fff640 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e325ab10 0x5574e3259820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dbabfb66815f90f2 server_cookie=665d37014a191a88 in_seq=0 out_seq=0 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8247fff640 1 -- 172.21.15.116:0/417730108 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5574e2e7e610 con 0x5574e325ab10 2024-03-24T09:21:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8247fff640 10 client.15273 handle_command_reply: tid=0 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 client.15273 shutdown 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 msgr2=0x5574e31e72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 0x5574e31e72c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f825c0a3380 tx=0x7f825c0038c0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 shutdown_connections 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5574e31e00c0 0x5574e31ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5574e325ab10 0x5574e3259820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5574e31daf50 0x5574e31e7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8247fff640 0 client.15273 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 --2- 172.21.15.116:0/417730108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5574e31da560 0x5574e31e72c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.021+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 >> 172.21.15.116:0/417730108 conn(0x5574e31d5ad0 msgr2=0x5574e31d5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 shutdown_connections 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f8271b68740 1 -- 172.21.15.116:0/417730108 wait complete. 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f8271b68740 20 client.15273 trim_cache size 0 max 16384 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24451, 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:32.031 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:32.032 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 88.128595293999993, 2024-03-24T09:21:32.032 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:32.032 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 91, 2024-03-24T09:21:32.032 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:32.032 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 136.48246940600001 2024-03-24T09:21:32.032 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f82580a21d0 msgr2=0x7f82580a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f82580a21d0 0x7f82580a4690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f825402c170 tx=0x7f82540046c0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 msgr2=0x7f826806a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 0x7f826806a860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f825c00e9c0 tx=0x7f825c00ee80 comp rx=0 tx=0).stop 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 shutdown_connections 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82681021e0 0x7f826819ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f82580a21d0 0x7f82580a4690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8268101820 0x7f826819c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 --2- 172.21.15.116:0/1487088818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8268100630 0x7f826806a860 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 >> 172.21.15.116:0/1487088818 conn(0x7f82680fbbc0 msgr2=0x7f8268104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 shutdown_connections 2024-03-24T09:21:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.022+0000 7f826ef8a640 1 -- 172.21.15.116:0/1487088818 wait complete. 2024-03-24T09:21:32.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 -- 172.21.15.116:0/519249811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e80ffa00 msgr2=0x7fd1e80ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 --2- 172.21.15.116:0/519249811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e80ffa00 0x7fd1e80ffe20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009900 tx=0x7fd1d802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 -- 172.21.15.116:0/519249811 shutdown_connections 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 --2- 172.21.15.116:0/519249811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e8103390 0x7fd1e8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 --2- 172.21.15.116:0/519249811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1e81029d0 0x7fd1e8102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 --2- 172.21.15.116:0/519249811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e80ffa00 0x7fd1e80ffe20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.137+0000 7fd1ede07640 1 -- 172.21.15.116:0/519249811 >> 172.21.15.116:0/519249811 conn(0x7fd1e80fb310 msgr2=0x7fd1e80fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 -- 172.21.15.116:0/519249811 shutdown_connections 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 -- 172.21.15.116:0/519249811 wait complete. 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 Processor -- start 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 -- start start 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e80ffa00 0x7fd1e8197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1e81029d0 0x7fd1e8197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 0x7fd1e819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e819f220 con 0x7fd1e8103390 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e819f390 con 0x7fd1e81029d0 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.138+0000 7fd1ede07640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e819f550 con 0x7fd1e80ffa00 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 0x7fd1e819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e80ffa00 0x7fd1e8197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1e81029d0 0x7fd1e8197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 0x7fd1e819ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33398/0 (socket says 172.21.15.116:33398) 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e80ffa00 0x7fd1e8197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41302/0 (socket says 172.21.15.116:41302) 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 -- 172.21.15.116:0/2495141065 learned_addr learned my addr 172.21.15.116:0/2495141065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 -- 172.21.15.116:0/2495141065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e80ffa00 msgr2=0x7fd1e8197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e80ffa00 0x7fd1e8197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 -- 172.21.15.116:0/2495141065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1e81029d0 msgr2=0x7fd1e8197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1e81029d0 0x7fd1e8197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 -- 172.21.15.116:0/2495141065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1dc009490 con 0x7fd1e8103390 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e7fff640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 0x7fd1e819ece0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000eb10 tx=0x7fd1e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e4ff9640 1 -- 172.21.15.116:0/2495141065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0004b20 con 0x7fd1e8103390 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e4ff9640 1 -- 172.21.15.116:0/2495141065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1e0004c80 con 0x7fd1e8103390 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d8009560 con 0x7fd1e8103390 2024-03-24T09:21:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1e4ff9640 1 -- 172.21.15.116:0/2495141065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0010690 con 0x7fd1e8103390 2024-03-24T09:21:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.139+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e819fbb0 con 0x7fd1e8103390 2024-03-24T09:21:32.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1e4ff9640 1 -- 172.21.15.116:0/2495141065 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7fd1e0005020 con 0x7fd1e8103390 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 Processor -- start 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 -- start start 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26cd8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 0x561f26cdde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561f26c43180 con 0x561f26cd9360 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561f26c406b0 con 0x561f26cd8970 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561f26c3be90 con 0x561f26cde4d0 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 0x561f26cdde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 0x561f26cdde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33410/0 (socket says 172.21.15.116:33410) 2024-03-24T09:21:32.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1e77fe640 1 -- 172.21.15.116:0/1459913020 learned_addr learned my addr 172.21.15.116:0/1459913020 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26cd8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.142+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e4ff9640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd1d00a2150 0x7fd1d00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e77fe640 1 -- 172.21.15.116:0/1459913020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 msgr2=0x561f26ce0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e77fe640 1 -- 172.21.15.116:0/1459913020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 msgr2=0x561f26cd8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26cd8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e77fe640 1 -- 172.21.15.116:0/1459913020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f26bf8d80 con 0x561f26cd9360 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e7fff640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd1d00a2150 0x7fd1d00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 0x561f26cdde60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009900 tx=0x7fd1d8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c196ce42251584 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1e4ff9640 1 -- 172.21.15.116:0/2495141065 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1e0014070 con 0x7fd1e8103390 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1cdffb640 1 -- 172.21.15.116:0/1459913020 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d802c930 con 0x561f26cd9360 2024-03-24T09:21:32.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1cdffb640 1 -- 172.21.15.116:0/1459913020 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1d802ca90 con 0x561f26cd9360 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.143+0000 7fd1cdffb640 1 -- 172.21.15.116:0/1459913020 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8003830 con 0x561f26cd9360 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- 172.21.15.116:0/1459913020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 msgr2=0x561f26cdde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1e7fff640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd1d00a2150 0x7fd1d00a4610 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0095a00 tx=0x7fd1e006f5a0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 0x561f26cdde60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009900 tx=0x7fd1d8005dd0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- 172.21.15.116:0/1459913020 shutdown_connections 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd9360 0x561f26cdde60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/1459913020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26cd8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- 172.21.15.116:0/1459913020 >> 172.21.15.116:0/1459913020 conn(0x561f26ccfad0 msgr2=0x561f26ccfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- 172.21.15.116:0/1459913020 shutdown_connections 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- 172.21.15.116:0/1459913020 wait complete. 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- start start 2024-03-24T09:21:32.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 Processor -- start 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26ce28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd9360 0x561f26ce2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561f26be0d50 con 0x561f26cd8970 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561f2690f310 con 0x561f26cd9360 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.144+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561f26bc18f0 con 0x561f26cde4d0 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1f09e9740 10 client.0 _ll_register_callbacks cb 0x561f2694cac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26ce28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd9360 0x561f26ce2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26ce28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33424/0 (socket says 172.21.15.116:33424) 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/210922983 learned_addr learned my addr 172.21.15.116:0/210922983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 msgr2=0x561f26ce9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/210922983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd9360 msgr2=0x561f26ce2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd9360 0x561f26ce2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/210922983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e810a460 con 0x561f26cd8970 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd9360 0x561f26ce2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26ce28e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc012870 tx=0x7fd1dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f338cdd660e65a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/210922983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc003dc0 con 0x561f26cd8970 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/210922983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1dc0047a0 con 0x561f26cd8970 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561f2693b6c0 con 0x561f26cd8970 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/210922983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd1dc004960 con 0x561f26cd8970 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1f09e9740 10 client.15285 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:32.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f26c3a510 con 0x561f26cd8970 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.145+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/210922983 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561f2693b6c0 con 0x561f26cd8970 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/210922983 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd1dc020470 con 0x561f26cd8970 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1f09e9740 10 client.15285 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1f09e9740 10 client.15285 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1f09e9740 20 client.15285 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26cef200 0x561f26cf0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1f09e9740 4 client.15285 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f2698c270 con 0x561f26cef200 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26cef200 0x561f26cf0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26cef200 0x561f26cf0670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e956168e8071322c server_cookie=6b3c40bf548e6c19 in_seq=0 out_seq=0 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.146+0000 7fd1c6ffd640 10 client.15285 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:32.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/210922983 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f2698c270 con 0x561f26cef200 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1c6ffd640 10 client.15285 handle_command_reply: tid=0 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 client.15285 shutdown 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 msgr2=0x561f26ce28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26ce28e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc012870 tx=0x7fd1dc004730 comp rx=0 tx=0).stop 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 shutdown_connections 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26cef200 0x561f26cf0670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cde4d0 0x561f26ce9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1c6ffd640 0 client.15285 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd9360 0x561f26ce2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/210922983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26ce28e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 >> 172.21.15.116:0/210922983 conn(0x561f26ccfad0 msgr2=0x561f26ccece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 shutdown_connections 2024-03-24T09:21:32.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 1 -- 172.21.15.116:0/210922983 wait complete. 2024-03-24T09:21:32.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.147+0000 7fd1f09e9740 20 client.15285 trim_cache size 0 max 16384 2024-03-24T09:21:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 Processor -- start 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 -- start start 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26cca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26ce9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cde4d0 0x561f26ccfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561f26cde2b0 con 0x561f26cd8970 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561f26ce1020 con 0x561f26cde4d0 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.165+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561f2698c270 con 0x561f26cd9360 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26cca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26cca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33438/0 (socket says 172.21.15.116:33438) 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3758152028 learned_addr learned my addr 172.21.15.116:0/3758152028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26ce9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cde4d0 0x561f26ccfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3758152028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 msgr2=0x561f26ce9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26ce9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3758152028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cde4d0 msgr2=0x561f26ccfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cde4d0 0x561f26ccfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3758152028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f26bd6a80 con 0x561f26cd8970 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26cca810 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0059f0 tx=0x7fd1dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2522e67518052636 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1cd7fa640 1 -- 172.21.15.116:0/3758152028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc003dc0 con 0x561f26cd8970 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1cd7fa640 1 -- 172.21.15.116:0/3758152028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1dc0203e0 con 0x561f26cd8970 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.166+0000 7fd1cd7fa640 1 -- 172.21.15.116:0/3758152028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc012600 con 0x561f26cd8970 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3758152028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 msgr2=0x561f26cca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26cca810 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0059f0 tx=0x7fd1dc0046c0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3758152028 shutdown_connections 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cde4d0 0x561f26ccfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26ce9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3758152028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cd8970 0x561f26cca810 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3758152028 >> 172.21.15.116:0/3758152028 conn(0x561f26ccfad0 msgr2=0x561f26cee1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3758152028 shutdown_connections 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3758152028 wait complete. 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- start start 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 Processor -- start 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26ce7d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26d50f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 0x561f26d534a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561f26d53a10 con 0x561f26cde4d0 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561f26d53b80 con 0x561f26cd8970 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.167+0000 7fd1f09e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561f26d53d40 con 0x561f26cd9360 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1f09e9740 10 client.0 _ll_register_callbacks cb 0x7fd1dc009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:32.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 0x561f26d534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 0x561f26d534a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33448/0 (socket says 172.21.15.116:33448) 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3523543293 learned_addr learned my addr 172.21.15.116:0/3523543293 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26ce7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26d50f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 msgr2=0x561f26d50f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26d50f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3523543293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 msgr2=0x561f26ce7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26ce7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 -- 172.21.15.116:0/3523543293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f26c3a510 con 0x561f26cde4d0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1e6ffd640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 0x561f26d534a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0058c0 tx=0x7fd1dc004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f0d08ae6552c74 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/3523543293 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc020b10 con 0x561f26cde4d0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/3523543293 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd1dc000e40 con 0x561f26cde4d0 2024-03-24T09:21:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1d802ca90 con 0x561f26cde4d0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/3523543293 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd1dc020c70 con 0x561f26cde4d0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1f09e9740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561f26bfc7e0 con 0x561f26cde4d0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.168+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/3523543293 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1d802ca90 con 0x561f26cde4d0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/3523543293 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd1dc011c30 con 0x561f26cde4d0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 10 client.15291 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 20 client.15291 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26d59260 0x561f26d57f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 4 client.15291 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "status"}) -- 0x561f26978610 con 0x561f26d59260 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26d59260 0x561f26d57f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1c6ffd640 10 client.15291 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1e77fe640 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26d59260 0x561f26d57f70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=64aa3b7c35a75558 server_cookie=fd4a4defbd602d53 in_seq=0 out_seq=0 2024-03-24T09:21:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1c6ffd640 1 -- 172.21.15.116:0/3523543293 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561f26978610 con 0x561f26d59260 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1c6ffd640 10 client.15291 handle_command_reply: tid=0 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 1 client.15291 shutdown 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 msgr2=0x561f26d534a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.169+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 0x561f26d534a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0058c0 tx=0x7fd1dc004470 comp rx=0 tx=0).stop 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 shutdown_connections 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561f26d59260 0x561f26d57f70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f26cde4d0 0x561f26d534a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1c6ffd640 0 client.15291 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561f26cd9360 0x561f26d50f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 --2- 172.21.15.116:0/3523543293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f26cd8970 0x561f26ce7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 >> 172.21.15.116:0/3523543293 conn(0x561f26ccfad0 msgr2=0x561f26cca810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 shutdown_connections 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 1 -- 172.21.15.116:0/3523543293 wait complete. 2024-03-24T09:21:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1f09e9740 20 client.15291 trim_cache size 0 max 16384 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24484, 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 89.149861251999994, 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 91, 2024-03-24T09:21:32.178 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:32.179 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 127.806751218 2024-03-24T09:21:32.179 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:32.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd1d00a2150 msgr2=0x7fd1d00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd1d00a2150 0x7fd1d00a4610 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0095a00 tx=0x7fd1e006f5a0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 msgr2=0x7fd1e819ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 0x7fd1e819ece0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000eb10 tx=0x7fd1e000c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 shutdown_connections 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1e8103390 0x7fd1e819ece0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1e81029d0 0x7fd1e8197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fd1d00a2150 0x7fd1d00a4610 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 --2- 172.21.15.116:0/2495141065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1e80ffa00 0x7fd1e8197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 >> 172.21.15.116:0/2495141065 conn(0x7fd1e80fb310 msgr2=0x7fd1e8105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 shutdown_connections 2024-03-24T09:21:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.170+0000 7fd1ede07640 1 -- 172.21.15.116:0/2495141065 wait complete. 2024-03-24T09:21:32.181 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:32 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2921494115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:32.181 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:32 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1671475334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:32.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T09:21:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:32 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2921494115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:32 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1671475334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 -- 172.21.15.116:0/1456729284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 msgr2=0x7f9e30101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 --2- 172.21.15.116:0/1456729284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30101b40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009900 tx=0x7f9e2402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 -- 172.21.15.116:0/1456729284 shutdown_connections 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 --2- 172.21.15.116:0/1456729284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e301032d0 0x7f9e30109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 --2- 172.21.15.116:0/1456729284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e30102910 0x7f9e30102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 --2- 172.21.15.116:0/1456729284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30101b40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 -- 172.21.15.116:0/1456729284 >> 172.21.15.116:0/1456729284 conn(0x7f9e300fccb0 msgr2=0x7f9e300ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 -- 172.21.15.116:0/1456729284 shutdown_connections 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.283+0000 7f9e3593e640 1 -- 172.21.15.116:0/1456729284 wait complete. 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 Processor -- start 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 -- start start 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e30102910 0x7f9e30197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e301032d0 0x7f9e3019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3019f1f0 con 0x7f9e30101720 2024-03-24T09:21:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3019f360 con 0x7f9e301032d0 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e3593e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3019f520 con 0x7f9e30102910 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33466/0 (socket says 172.21.15.116:33466) 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e2effd640 1 -- 172.21.15.116:0/878934029 learned_addr learned my addr 172.21.15.116:0/878934029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e30102910 0x7f9e30197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.284+0000 7f9e2f7fe640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e301032d0 0x7f9e3019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e2effd640 1 -- 172.21.15.116:0/878934029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e30102910 msgr2=0x7f9e30197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e2effd640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e30102910 0x7f9e30197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e2effd640 1 -- 172.21.15.116:0/878934029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e301032d0 msgr2=0x7f9e3019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e2effd640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e301032d0 0x7f9e3019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e2effd640 1 -- 172.21.15.116:0/878934029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e24009560 con 0x7f9e30101720 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e2effd640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30197700 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9e2402c7d0 tx=0x7f9e24004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e3493c640 1 -- 172.21.15.116:0/878934029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24004500 con 0x7f9e30101720 2024-03-24T09:21:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e3493c640 1 -- 172.21.15.116:0/878934029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e2402de90 con 0x7f9e30101720 2024-03-24T09:21:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3019f7a0 con 0x7f9e30101720 2024-03-24T09:21:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e3493c640 1 -- 172.21.15.116:0/878934029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24005350 con 0x7f9e30101720 2024-03-24T09:21:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.285+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3019fcb0 con 0x7f9e30101720 2024-03-24T09:21:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.287+0000 7f9e3493c640 1 -- 172.21.15.116:0/878934029 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f9e240054b0 con 0x7f9e30101720 2024-03-24T09:21:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e3493c640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9e180a2150 0x7f9e180a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 Processor -- start 2024-03-24T09:21:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 -- start start 2024-03-24T09:21:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9e180a2150 0x7f9e180a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e3493c640 1 -- 172.21.15.116:0/878934029 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9e240f00f0 con 0x7f9e30101720 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe916980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe91ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe91e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649fe885180 con 0x5649fe916f50 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649fe8826b0 con 0x5649fe91c0c0 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e385ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649fe87de90 con 0x5649fe916560 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe91ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe91ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33482/0 (socket says 172.21.15.116:33482) 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2e7fc640 1 -- 172.21.15.116:0/1322407882 learned_addr learned my addr 172.21.15.116:0/1322407882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2f7fe640 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe91e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe916980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9e180a2150 0x7f9e180a4610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e20013570 tx=0x7f9e20013970 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 -- 172.21.15.116:0/1322407882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 msgr2=0x5649fe91e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe91e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 -- 172.21.15.116:0/1322407882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 msgr2=0x5649fe91ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe91ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 -- 172.21.15.116:0/1322407882 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649fe83ad80 con 0x5649fe916560 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe91ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.288+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe916980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9e240ed850 tx=0x7f9e240bc490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d98ac3c8e471bc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e15ffb640 1 -- 172.21.15.116:0/1322407882 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240c0bc0 con 0x5649fe916560 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e15ffb640 1 -- 172.21.15.116:0/1322407882 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e240c5210 con 0x5649fe916560 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e15ffb640 1 -- 172.21.15.116:0/1322407882 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240c4d80 con 0x5649fe916560 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 -- 172.21.15.116:0/1322407882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 msgr2=0x5649fe916980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe916980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9e240ed850 tx=0x7f9e240bc490 comp rx=0 tx=0).stop 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 -- 172.21.15.116:0/1322407882 shutdown_connections 2024-03-24T09:21:32.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe91e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe91ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1322407882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe916980 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 -- 172.21.15.116:0/1322407882 >> 172.21.15.116:0/1322407882 conn(0x5649fe911ad0 msgr2=0x5649fe911ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 -- 172.21.15.116:0/1322407882 shutdown_connections 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.289+0000 7f9e385ff740 1 -- 172.21.15.116:0/1322407882 wait complete. 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 -- start start 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 Processor -- start 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe920330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe916f50 0x5649fe920870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe91c0c0 0x5649fe9278b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649fe822d50 con 0x5649fe916560 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649fe551310 con 0x5649fe916f50 2024-03-24T09:21:32.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649fe8038f0 con 0x5649fe91c0c0 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e385ff740 10 client.0 _ll_register_callbacks cb 0x5649fe58eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe920330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe920330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33484/0 (socket says 172.21.15.116:33484) 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 -- 172.21.15.116:0/1041627315 learned_addr learned my addr 172.21.15.116:0/1041627315 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2f7fe640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe91c0c0 0x5649fe9278b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe916f50 0x5649fe920870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 -- 172.21.15.116:0/1041627315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe91c0c0 msgr2=0x5649fe9278b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe91c0c0 0x5649fe9278b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 -- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe916f50 msgr2=0x5649fe920870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe916f50 0x5649fe920870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.290+0000 7f9e2effd640 1 -- 172.21.15.116:0/1041627315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3019fcb0 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e2effd640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe920330 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9e240edcc0 tx=0x7f9e24003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e9196d2b647381 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e0effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e0effd640 1 -- 172.21.15.116:0/1041627315 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240c0bc0 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e0effd640 1 -- 172.21.15.116:0/1041627315 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9e240c4f90 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649fe57d6c0 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e0effd640 1 -- 172.21.15.116:0/1041627315 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9e240d5270 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 10 client.15303 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649fe87c510 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e0effd640 1 -- 172.21.15.116:0/1041627315 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5649fe57d6c0 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e0effd640 1 -- 172.21.15.116:0/1041627315 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9e2403c070 con 0x5649fe916560 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 10 client.15303 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 10 client.15303 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 20 client.15303 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe92ba10 0x5649fe92dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 4 client.15303 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649fe5ce270 con 0x5649fe92ba10 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.291+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe92ba10 0x5649fe92dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.292+0000 7f9e0effd640 10 client.15303 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.292+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe92ba10 0x5649fe92dad0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc61a5346bc66d19 server_cookie=5158fb6e8c63937b in_seq=0 out_seq=0 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.292+0000 7f9e0effd640 1 -- 172.21.15.116:0/1041627315 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649fe5ce270 con 0x5649fe92ba10 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.292+0000 7f9e0effd640 10 client.15303 handle_command_reply: tid=0 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.292+0000 7f9e385ff740 1 client.15303 shutdown 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 msgr2=0x5649fe920330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe920330 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9e240edcc0 tx=0x7f9e24003ea0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 shutdown_connections 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe91c0c0 0x5649fe9278b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe92ba10 0x5649fe92dad0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe916f50 0x5649fe920870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e0effd640 0 client.15303 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 --2- 172.21.15.116:0/1041627315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe920330 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 >> 172.21.15.116:0/1041627315 conn(0x5649fe911ad0 msgr2=0x5649fe910ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 shutdown_connections 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 1 -- 172.21.15.116:0/1041627315 wait complete. 2024-03-24T09:21:32.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.293+0000 7f9e385ff740 20 client.15303 trim_cache size 0 max 16384 2024-03-24T09:21:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:32 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2921494115' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:32.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:32 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1671475334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.310+0000 7f9e385ff740 1 Processor -- start 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.310+0000 7f9e385ff740 1 -- start start 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe92dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe9207d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe90c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e385ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649fe91f280 con 0x5649fe916f50 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e385ff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649fe912520 con 0x5649fe91c0c0 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e385ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649fe928170 con 0x5649fe916560 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe9207d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe9207d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33496/0 (socket says 172.21.15.116:33496) 2024-03-24T09:21:32.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe92dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 -- 172.21.15.116:0/385893706 learned_addr learned my addr 172.21.15.116:0/385893706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2f7fe640 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe90c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 -- 172.21.15.116:0/385893706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 msgr2=0x5649fe92dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe92dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 -- 172.21.15.116:0/385893706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 msgr2=0x5649fe90c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe90c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 -- 172.21.15.116:0/385893706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649fe818a80 con 0x5649fe916f50 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe9207d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5649fe917350 tx=0x7f9e2001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1e1af11c80d2a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e157fa640 1 -- 172.21.15.116:0/385893706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20018040 con 0x5649fe916f50 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e157fa640 1 -- 172.21.15.116:0/385893706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e20012110 con 0x5649fe916f50 2024-03-24T09:21:32.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.311+0000 7f9e157fa640 1 -- 172.21.15.116:0/385893706 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e200053a0 con 0x5649fe916f50 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 -- 172.21.15.116:0/385893706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 msgr2=0x5649fe9207d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe9207d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5649fe917350 tx=0x7f9e2001f000 comp rx=0 tx=0).stop 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 -- 172.21.15.116:0/385893706 shutdown_connections 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe90c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916f50 0x5649fe9207d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 --2- 172.21.15.116:0/385893706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916560 0x5649fe92dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 -- 172.21.15.116:0/385893706 >> 172.21.15.116:0/385893706 conn(0x5649fe911ad0 msgr2=0x5649fe91b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 -- 172.21.15.116:0/385893706 shutdown_connections 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 -- 172.21.15.116:0/385893706 wait complete. 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 -- start start 2024-03-24T09:21:32.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.312+0000 7f9e385ff740 1 Processor -- start 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe98e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916f50 0x5649fe98eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe995520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649fe995a60 con 0x5649fe916560 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649fe995ba0 con 0x5649fe91c0c0 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649fe995ce0 con 0x5649fe916f50 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 10 client.0 _ll_register_callbacks cb 0x7f9e20009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe98e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe98e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33498/0 (socket says 172.21.15.116:33498) 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 -- 172.21.15.116:0/547001726 learned_addr learned my addr 172.21.15.116:0/547001726 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916f50 0x5649fe98eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2f7fe640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe995520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 -- 172.21.15.116:0/547001726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916f50 msgr2=0x5649fe98eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916f50 0x5649fe98eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 -- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 msgr2=0x5649fe995520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe995520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 -- 172.21.15.116:0/547001726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e20026040 con 0x5649fe916560 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e2effd640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe98e600 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b8bb0 tx=0x7f9e240bc8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45186fbdfa3fd465 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e0effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e0effd640 1 -- 172.21.15.116:0/547001726 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240bbd20 con 0x5649fe916560 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e0effd640 1 -- 172.21.15.116:0/547001726 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9e240eda50 con 0x5649fe916560 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e240c5210 con 0x5649fe916560 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e0effd640 1 -- 172.21.15.116:0/547001726 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9e24003d30 con 0x5649fe916560 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 10 client.15309 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:32.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.313+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649fe83e7e0 con 0x5649fe916560 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e0effd640 1 -- 172.21.15.116:0/547001726 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e240c5210 con 0x5649fe916560 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e0effd640 1 -- 172.21.15.116:0/547001726 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9e2403c040 con 0x5649fe916560 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e385ff740 10 client.15309 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e385ff740 10 client.15309 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e385ff740 20 client.15309 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e385ff740 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe99aff0 0x5649fe999d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e385ff740 4 client.15309 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "status"}) -- 0x5649fe5ba610 con 0x5649fe99aff0 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe99aff0 0x5649fe999d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e0effd640 10 client.15309 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.314+0000 7f9e2e7fc640 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe99aff0 0x5649fe999d00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2809138e37df05c server_cookie=283fb001397a0fdf in_seq=0 out_seq=0 2024-03-24T09:21:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e0effd640 1 -- 172.21.15.116:0/547001726 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5649fe5ba610 con 0x5649fe99aff0 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e0effd640 10 client.15309 handle_command_reply: tid=0 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 client.15309 shutdown 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 msgr2=0x5649fe98e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe98e600 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9e240b8bb0 tx=0x7f9e240bc8d0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 shutdown_connections 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649fe99aff0 0x5649fe999d00 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649fe91c0c0 0x5649fe995520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e0effd640 0 client.15309 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649fe916f50 0x5649fe98eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 --2- 172.21.15.116:0/547001726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649fe916560 0x5649fe98e600 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 >> 172.21.15.116:0/547001726 conn(0x5649fe911ad0 msgr2=0x5649fe90c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 shutdown_connections 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 1 -- 172.21.15.116:0/547001726 wait complete. 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e385ff740 20 client.15309 trim_cache size 0 max 16384 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24515, 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:32.325 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 113.216272121, 2024-03-24T09:21:32.326 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:32.326 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 91, 2024-03-24T09:21:32.326 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:32.326 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 119.357400863 2024-03-24T09:21:32.326 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9e180a2150 msgr2=0x7f9e180a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9e180a2150 0x7f9e180a4610 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9e20013570 tx=0x7f9e20013970 comp rx=0 tx=0).stop 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 msgr2=0x7f9e30197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30197700 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9e2402c7d0 tx=0x7f9e24004040 comp rx=0 tx=0).stop 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 shutdown_connections 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e301032d0 0x7f9e3019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9e180a2150 0x7f9e180a4610 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e30102910 0x7f9e30197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 --2- 172.21.15.116:0/878934029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e30101720 0x7f9e30197700 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.315+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 >> 172.21.15.116:0/878934029 conn(0x7f9e300fccb0 msgr2=0x7f9e30105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.316+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 shutdown_connections 2024-03-24T09:21:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:32.316+0000 7f9e3593e640 1 -- 172.21.15.116:0/878934029 wait complete. 2024-03-24T09:21:32.329 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T09:21:32.329 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:21:32.339 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:21:32.340 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T09:21:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:33 smithi117 ceph-mon[35788]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:33.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:33 smithi116 ceph-mon[32207]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:33 smithi143 ceph-mon[35965]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- 172.21.15.116:0/1774832268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 msgr2=0x7f668c101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1774832268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c101550 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f66800099c0 tx=0x7f6680009cd0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- 172.21.15.116:0/1774832268 shutdown_connections 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1774832268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f668c102cd0 0x7f668c109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1774832268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c101550 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1774832268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f668c102330 0x7f668c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- 172.21.15.116:0/1774832268 >> 172.21.15.116:0/1774832268 conn(0x7f668c0fc6f0 msgr2=0x7f668c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:34.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- 172.21.15.116:0/1774832268 shutdown_connections 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- 172.21.15.116:0/1774832268 wait complete. 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 Processor -- start 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- start start 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c105070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f668c102330 0x7f668c103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f668c102cd0 0x7f668c103c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1043f0 con 0x7f668c101130 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f668c104560 con 0x7f668c102cd0 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.440+0000 7f6692f6f700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f668c1a2270 con 0x7f668c102330 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c105070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c105070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33526/0 (socket says 172.21.15.116:33526) 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 -- 172.21.15.116:0/1790128258 learned_addr learned my addr 172.21.15.116:0/1790128258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f669150c700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f668c102cd0 0x7f668c103c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:34.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f668bfff700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f668c102330 0x7f668c103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 -- 172.21.15.116:0/1790128258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f668c102330 msgr2=0x7f668c103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f668c102330 0x7f668c103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 -- 172.21.15.116:0/1790128258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f668c102cd0 msgr2=0x7f668c103c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f668c102cd0 0x7f668c103c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 -- 172.21.15.116:0/1790128258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66800095f0 con 0x7f668c101130 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6690d0b700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c105070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f66800098e0 tx=0x7f668000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668001d070 con 0x7f668c101130 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6680005940 con 0x7f668c101130 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668c1a24a0 con 0x7f668c101130 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6680022dd0 con 0x7f668c101130 2024-03-24T09:21:34.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.441+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668c1a2940 con 0x7f668c101130 2024-03-24T09:21:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.442+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f668c068ec0 con 0x7f668c101130 2024-03-24T09:21:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.443+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 136808+0+0 (secure 0 0 0) 0x7f6680022570 con 0x7f668c101130 2024-03-24T09:21:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.444+0000 7f6689ffb700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f667c0a21e0 0x7f667c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.444+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f66800cf410 con 0x7f668c101130 2024-03-24T09:21:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.444+0000 7f668bfff700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f667c0a21e0 0x7f667c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:34.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.444+0000 7f668bfff700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f667c0a21e0 0x7f667c0a46a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6678009c60 tx=0x7f6678009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:34.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.448+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6680098040 con 0x7f668c101130 2024-03-24T09:21:34.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.657+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f668c06b0c0 con 0x7f667c0a21e0 2024-03-24T09:21:34.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.658+0000 7f6689ffb700 1 -- 172.21.15.116:0/1790128258 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f668c06b0c0 con 0x7f667c0a21e0 2024-03-24T09:21:34.659 INFO:teuthology.orchestra.run.smithi116.stdout:Backend: cephadm 2024-03-24T09:21:34.659 INFO:teuthology.orchestra.run.smithi116.stdout:Available: Yes 2024-03-24T09:21:34.659 INFO:teuthology.orchestra.run.smithi116.stdout:Paused: No 2024-03-24T09:21:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f667c0a21e0 msgr2=0x7f667c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f667c0a21e0 0x7f667c0a46a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6678009c60 tx=0x7f6678009260 comp rx=0 tx=0).stop 2024-03-24T09:21:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 msgr2=0x7f668c105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:34.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c105070 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f66800098e0 tx=0x7f668000bfd0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 shutdown_connections 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f668c102cd0 0x7f668c103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f667c0a21e0 0x7f667c0a46a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f668c101130 0x7f668c105070 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 --2- 172.21.15.116:0/1790128258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f668c102330 0x7f668c103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 >> 172.21.15.116:0/1790128258 conn(0x7f668c0fc6f0 msgr2=0x7f668c06a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 shutdown_connections 2024-03-24T09:21:34.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:34.661+0000 7f6692f6f700 1 -- 172.21.15.116:0/1790128258 wait complete. 2024-03-24T09:21:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:35 smithi117 ceph-mon[35788]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T09:21:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-24T09:21:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]: dispatch 2024-03-24T09:21:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:35 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-24T09:21:35.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:35 smithi116 ceph-mon[32207]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:35.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T09:21:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-24T09:21:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]: dispatch 2024-03-24T09:21:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:35 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-24T09:21:35.489 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T09:21:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:35 smithi143 ceph-mon[35965]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-24T09:21:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T09:21:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-03-24T09:21:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]: dispatch 2024-03-24T09:21:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:35 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-24T09:21:35.728 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:36.105 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:36.105 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T09:21:36.105 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-24T09:21:36.105 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]': finished 2024-03-24T09:21:36.106 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-24T09:21:36.106 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T09:21:36.106 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T09:21:36.106 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:36 smithi116 ceph-mon[32207]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]': finished 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T09:21:36.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:36 smithi117 ceph-mon[35788]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]': finished 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: from='mgr.14148 172.21.15.116:0/1228178175' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 307 B/s wr, 10 op/s 2024-03-24T09:21:36.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:36 smithi143 ceph-mon[35965]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T09:21:37.199 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:37 smithi116 ceph-mon[32207]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T09:21:37.199 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:37 smithi116 ceph-mon[32207]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T09:21:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:37 smithi117 ceph-mon[35788]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T09:21:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:37 smithi117 ceph-mon[35788]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.429+0000 7f124f423700 1 -- 172.21.15.116:0/3599657895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248101190 msgr2=0x7f12481015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.429+0000 7f124f423700 1 --2- 172.21.15.116:0/3599657895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248101190 0x7f12481015b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f12440099c0 tx=0x7f1244009cd0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- 172.21.15.116:0/3599657895 shutdown_connections 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 --2- 172.21.15.116:0/3599657895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1248102d30 0x7f12481095e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 --2- 172.21.15.116:0/3599657895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248101190 0x7f12481015b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 --2- 172.21.15.116:0/3599657895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 0x7f12481027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- 172.21.15.116:0/3599657895 >> 172.21.15.116:0/3599657895 conn(0x7f12480fc750 msgr2=0x7f12480feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:37.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- 172.21.15.116:0/3599657895 shutdown_connections 2024-03-24T09:21:37.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- 172.21.15.116:0/3599657895 wait complete. 2024-03-24T09:21:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 Processor -- start 2024-03-24T09:21:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- start start 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1248101190 0x7f12481981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 0x7f12481986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 0x7f124819f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f124819fce0 con 0x7f1248102d30 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f124819fe20 con 0x7f1248102390 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.430+0000 7f124f423700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f124819ffb0 con 0x7f1248101190 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124d9c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 0x7f124819f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124d9c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 0x7f124819f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37908/0 (socket says 172.21.15.116:37908) 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124d9c0700 1 -- 172.21.15.116:0/1118729025 learned_addr learned my addr 172.21.15.116:0/1118729025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 0x7f12481986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124d1bf700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1248101190 0x7f12481981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 -- 172.21.15.116:0/1118729025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1248101190 msgr2=0x7f12481981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1248101190 0x7f12481981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 -- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 msgr2=0x7f124819f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 0x7f124819f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 -- 172.21.15.116:0/1118729025 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12440095f0 con 0x7f1248102390 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124d9c0700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 0x7f124819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124c9be700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 0x7f12481986e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f123800e9f0 tx=0x7f123800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f123800c9c0 con 0x7f1248102390 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12481a0290 con 0x7f1248102390 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1238010430 con 0x7f1248102390 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1238018680 con 0x7f1248102390 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.431+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12481a0790 con 0x7f1248102390 2024-03-24T09:21:37.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.432+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1248068ec0 con 0x7f1248102390 2024-03-24T09:21:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.433+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f12380105a0 con 0x7f1248102390 2024-03-24T09:21:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.434+0000 7f123e7fc700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12340a2150 0x7f12340a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.434+0000 7f124d1bf700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12340a2150 0x7f12340a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.434+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f1238014070 con 0x7f1248102390 2024-03-24T09:21:37.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.434+0000 7f124d1bf700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12340a2150 0x7f12340a4610 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1244009990 tx=0x7f1244005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.442+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1238096ff0 con 0x7f1248102390 2024-03-24T09:21:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:37 smithi143 ceph-mon[35965]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T09:21:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:37 smithi143 ceph-mon[35965]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T09:21:37.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.645+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f12480fe980 con 0x7f12340a2150 2024-03-24T09:21:37.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.658+0000 7f123e7fc700 1 -- 172.21.15.116:0/1118729025 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7f12480fe980 con 0x7f12340a2150 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.a smithi116 running (2m) 32s ago 2m 18.4M - 19.0.0-1855-g21b3bd67 553463a90fd5 5c1a4d775300 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.b smithi117 running (2m) 57s ago 2m 25.4M - 19.0.0-1855-g21b3bd67 553463a90fd5 551dc8fe39be 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.c smithi143 running (2m) 58s ago 2m 16.0M - 19.0.0-1855-g21b3bd67 553463a90fd5 8b385a11dbd3 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.d smithi116 running (2m) 32s ago 2m 16.1M - 19.0.0-1855-g21b3bd67 553463a90fd5 e02a1bbc9768 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.e smithi117 running (2m) 57s ago 2m 17.1M - 19.0.0-1855-g21b3bd67 553463a90fd5 b6d6ed44dcd8 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.f smithi143 running (2m) 58s ago 2m 16.7M - 19.0.0-1855-g21b3bd67 553463a90fd5 f3d09d2ba8ce 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.g smithi116 running (2m) 32s ago 2m 16.2M - 19.0.0-1855-g21b3bd67 553463a90fd5 fc2441da97ec 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.h smithi117 running (2m) 57s ago 2m 18.5M - 19.0.0-1855-g21b3bd67 553463a90fd5 e0c634a61ae5 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.i smithi143 running (2m) 58s ago 2m 17.8M - 19.0.0-1855-g21b3bd67 553463a90fd5 317ad529fd26 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.j smithi116 running (2m) 32s ago 2m 13.1M - 19.0.0-1855-g21b3bd67 553463a90fd5 e47f948a299c 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.k smithi117 running (2m) 57s ago 2m 17.2M - 19.0.0-1855-g21b3bd67 553463a90fd5 6b2a3e1261f6 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mds.l smithi143 running (2m) 58s ago 2m 15.7M - 19.0.0-1855-g21b3bd67 553463a90fd5 8034a12199b3 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi116 *:9283,8765 running (9m) 32s ago 9m 513M - 19.0.0-1855-g21b3bd67 553463a90fd5 00e781c18f7d 2024-03-24T09:21:37.659 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi117 *:8443,8765 running (7m) 57s ago 7m 453M - 19.0.0-1855-g21b3bd67 553463a90fd5 fca04d5a0987 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.z smithi143 *:8443,8765 running (7m) 58s ago 7m 452M - 19.0.0-1855-g21b3bd67 553463a90fd5 83938f718d47 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (9m) 32s ago 9m 51.7M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 b745d26f56b3 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi117 running (8m) 57s ago 8m 42.0M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 b766b95c8640 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi143 running (8m) 58s ago 8m 43.3M 2048M 19.0.0-1855-g21b3bd67 553463a90fd5 f7132b5183bb 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7m) 32s ago 7m 57.6M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 13e96d2a2442 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi117 running (7m) 57s ago 7m 47.8M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 b54f015d43e1 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi143 running (6m) 58s ago 6m 47.6M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 bb325f71c020 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (6m) 32s ago 6m 57.0M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 83fbf11889e8 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi117 running (5m) 57s ago 5m 54.3M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 b5ba8fe69a04 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi143 running (5m) 58s ago 5m 58.0M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 b4b6ea48e2fe 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi116 running (5m) 32s ago 5m 42.3M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 59ee1ddbc4af 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi117 running (4m) 57s ago 4m 55.4M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 d33530d726f9 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 smithi143 running (4m) 58s ago 4m 55.3M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 645493ef080f 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 smithi116 running (3m) 32s ago 3m 57.3M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 ba1b894136d2 2024-03-24T09:21:37.660 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 smithi117 running (3m) 57s ago 3m 56.0M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 854b23418c5b 2024-03-24T09:21:37.661 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 smithi143 running (2m) 58s ago 2m 50.0M 4272M 19.0.0-1855-g21b3bd67 553463a90fd5 99a4803132a9 2024-03-24T09:21:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12340a2150 msgr2=0x7f12340a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:37.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12340a2150 0x7f12340a4610 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1244009990 tx=0x7f1244005fb0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 msgr2=0x7f12481986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 0x7f12481986e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f123800e9f0 tx=0x7f123800ed00 comp rx=0 tx=0).stop 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 shutdown_connections 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f12340a2150 0x7f12340a4610 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1248102d30 0x7f124819f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1248101190 0x7f12481981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 --2- 172.21.15.116:0/1118729025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1248102390 0x7f12481986e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 >> 172.21.15.116:0/1118729025 conn(0x7f12480fc750 msgr2=0x7f12480fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 shutdown_connections 2024-03-24T09:21:37.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:37.660+0000 7f124f423700 1 -- 172.21.15.116:0/1118729025 wait complete. 2024-03-24T09:21:38.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:38 smithi117 ceph-mon[35788]: from='client.25160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:38.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:38 smithi117 ceph-mon[35788]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 10 op/s 2024-03-24T09:21:38.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:38 smithi116 ceph-mon[32207]: from='client.25160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:38.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:38 smithi116 ceph-mon[32207]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 10 op/s 2024-03-24T09:21:38.420 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T09:21:38.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:38 smithi143 ceph-mon[35965]: from='client.25160 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:38.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:38 smithi143 ceph-mon[35965]: pgmap v355: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 895 B/s wr, 10 op/s 2024-03-24T09:21:38.659 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:40.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 -- 172.21.15.116:0/3187982811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda30101420 msgr2=0x7fda30101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:40.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 --2- 172.21.15.116:0/3187982811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda30101420 0x7fda30101880 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fda18009990 tx=0x7fda18009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 -- 172.21.15.116:0/3187982811 shutdown_connections 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 --2- 172.21.15.116:0/3187982811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 0x7fda30109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 --2- 172.21.15.116:0/3187982811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda301026e0 0x7fda30102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 --2- 172.21.15.116:0/3187982811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda30101420 0x7fda30101880 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 -- 172.21.15.116:0/3187982811 >> 172.21.15.116:0/3187982811 conn(0x7fda300fbe50 msgr2=0x7fda300fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 -- 172.21.15.116:0/3187982811 shutdown_connections 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 -- 172.21.15.116:0/3187982811 wait complete. 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 Processor -- start 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.331+0000 7fda35420700 1 -- start start 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda30101420 0x7fda30100e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 0x7fda3019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 0x7fda3019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fda3019fc60 con 0x7fda301026e0 2024-03-24T09:21:40.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fda3019fdd0 con 0x7fda30101dc0 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fda3019ff90 con 0x7fda30101420 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 0x7fda3019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 0x7fda3019f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37928/0 (socket says 172.21.15.116:37928) 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2effd700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda30101420 0x7fda30100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda267fc700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 0x7fda3019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 -- 172.21.15.116:0/2079188444 learned_addr learned my addr 172.21.15.116:0/2079188444 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 -- 172.21.15.116:0/2079188444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda30101420 msgr2=0x7fda30100e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda30101420 0x7fda30100e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 -- 172.21.15.116:0/2079188444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 msgr2=0x7fda3019ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 0x7fda3019ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 -- 172.21.15.116:0/2079188444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda180095f0 con 0x7fda301026e0 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda267fc700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 0x7fda3019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2f7fe700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 0x7fda3019f4c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fda2800ef30 tx=0x7fda2800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:40.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28018070 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda301a0210 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fda2800f040 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2800cd20 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.332+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda301a0730 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.333+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda301086c0 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.334+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fda28003a00 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.335+0000 7fda2cff9700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fda1c0a2150 0x7fda1c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.335+0000 7fda2effd700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fda1c0a2150 0x7fda1c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.335+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fda280cdb00 con 0x7fda301026e0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.335+0000 7fda2effd700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fda1c0a2150 0x7fda1c0a4610 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda3006a700 tx=0x7fda20009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:40.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.339+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fda28096790 con 0x7fda301026e0 2024-03-24T09:21:40.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.542+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fda3002db60 con 0x7fda1c0a2150 2024-03-24T09:21:40.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.548+0000 7fda2cff9700 1 -- 172.21.15.116:0/2079188444 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fda3002db60 con 0x7fda1c0a2150 2024-03-24T09:21:40.549 INFO:teuthology.orchestra.run.smithi116.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T09:21:40.549 INFO:teuthology.orchestra.run.smithi116.stdout:mds.all 12/12 61s ago 2m smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi143=c;smithi143=f;smithi143=i;smithi143=l;count:12 2024-03-24T09:21:40.549 INFO:teuthology.orchestra.run.smithi116.stdout:mgr 3/3 61s ago 7m smithi116=x;smithi117=y;smithi143=z;count:3 2024-03-24T09:21:40.549 INFO:teuthology.orchestra.run.smithi116.stdout:mon 3/3 61s ago 8m smithi116:172.21.15.116=a;smithi117:172.21.15.117=b;smithi143:172.21.15.143=c;count:3 2024-03-24T09:21:40.549 INFO:teuthology.orchestra.run.smithi116.stdout:osd 12 61s ago - 2024-03-24T09:21:40.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fda1c0a2150 msgr2=0x7fda1c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:40.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fda1c0a2150 0x7fda1c0a4610 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fda3006a700 tx=0x7fda20009260 comp rx=0 tx=0).stop 2024-03-24T09:21:40.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 msgr2=0x7fda3019f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 0x7fda3019f4c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fda2800ef30 tx=0x7fda2800c3a0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 shutdown_connections 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fda1c0a2150 0x7fda1c0a4610 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda301026e0 0x7fda3019f4c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda30101420 0x7fda30100e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 --2- 172.21.15.116:0/2079188444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda30101dc0 0x7fda3019ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 >> 172.21.15.116:0/2079188444 conn(0x7fda300fbe50 msgr2=0x7fda30105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 shutdown_connections 2024-03-24T09:21:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:40.551+0000 7fda35420700 1 -- 172.21.15.116:0/2079188444 wait complete. 2024-03-24T09:21:41.234 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:41 smithi116 ceph-mon[32207]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:41.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T09:21:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:41 smithi143 ceph-mon[35965]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:41 smithi117 ceph-mon[35788]: pgmap v356: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 895 B/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:41.502 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:42.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:42 smithi143 ceph-mon[35965]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:42.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:42 smithi117 ceph-mon[35788]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:42.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:42 smithi116 ceph-mon[32207]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 -- 172.21.15.116:0/2469174280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c102cd0 msgr2=0x7f508c109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2469174280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c102cd0 0x7f508c109580 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f508400b0d0 tx=0x7f508400b3e0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 -- 172.21.15.116:0/2469174280 shutdown_connections 2024-03-24T09:21:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2469174280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c102cd0 0x7f508c109580 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2469174280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c101130 0x7f508c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2469174280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 0x7f508c102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 -- 172.21.15.116:0/2469174280 >> 172.21.15.116:0/2469174280 conn(0x7f508c0fc6f0 msgr2=0x7f508c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 -- 172.21.15.116:0/2469174280 shutdown_connections 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.182+0000 7f5090e3a700 1 -- 172.21.15.116:0/2469174280 wait complete. 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 Processor -- start 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 -- start start 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 0x7f508c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 0x7f508c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:43.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c102cd0 0x7f508c19f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f508c19fd90 con 0x7f508c101130 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f508c19fed0 con 0x7f508c102330 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5090e3a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f508c1a0060 con 0x7f508c102cd0 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 0x7f508c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 0x7f508c198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37958/0 (socket says 172.21.15.116:37958) 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 -- 172.21.15.116:0/2278500255 learned_addr learned my addr 172.21.15.116:0/2278500255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508ad9d700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c102cd0 0x7f508c19f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5089d9b700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 0x7f508c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 -- 172.21.15.116:0/2278500255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c102cd0 msgr2=0x7f508c19f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c102cd0 0x7f508c19f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 -- 172.21.15.116:0/2278500255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 msgr2=0x7f508c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 0x7f508c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508a59c700 1 -- 172.21.15.116:0/2278500255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5084009d00 con 0x7f508c101130 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f508ad9d700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c102cd0 0x7f508c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.183+0000 7f5089d9b700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 0x7f508c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.184+0000 7f508a59c700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 0x7f508c198250 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f507c00e9a0 tx=0x7f507c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.184+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c009790 con 0x7f508c101130 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.184+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508c1a0340 con 0x7f508c101130 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.184+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f507c00c810 con 0x7f508c101130 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.184+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c003c40 con 0x7f508c101130 2024-03-24T09:21:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.184+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508c1a0840 con 0x7f508c101130 2024-03-24T09:21:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.185+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508c068ec0 con 0x7f508c101130 2024-03-24T09:21:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.186+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f507c00c980 con 0x7f508c101130 2024-03-24T09:21:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.186+0000 7f507b7fe700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f50740a2150 0x7f50740a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.187+0000 7f5089d9b700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f50740a2150 0x7f50740a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.187+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f507c014070 con 0x7f508c101130 2024-03-24T09:21:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.187+0000 7f5089d9b700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f50740a2150 0x7f50740a4610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5080011490 tx=0x7f5080009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:43.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.191+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f507c096e00 con 0x7f508c101130 2024-03-24T09:21:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:43 smithi143 ceph-mon[35965]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 895 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:43 smithi143 ceph-mon[35965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T09:21:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:43 smithi143 ceph-mon[35965]: Cluster is now healthy 2024-03-24T09:21:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:43 smithi117 ceph-mon[35788]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 895 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:43 smithi117 ceph-mon[35788]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T09:21:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:43 smithi117 ceph-mon[35788]: Cluster is now healthy 2024-03-24T09:21:43.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:43 smithi116 ceph-mon[32207]: pgmap v357: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 895 B/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:43 smithi116 ceph-mon[32207]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T09:21:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:43 smithi116 ceph-mon[32207]: Cluster is now healthy 2024-03-24T09:21:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.397+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f508c06b030 con 0x7f50740a2150 2024-03-24T09:21:43.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.399+0000 7f507b7fe700 1 -- 172.21.15.116:0/2278500255 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f508c06b030 con 0x7f50740a2150 2024-03-24T09:21:43.400 INFO:teuthology.orchestra.run.smithi116.stdout:HOST ADDR LABELS STATUS 2024-03-24T09:21:43.400 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 172.21.15.116 2024-03-24T09:21:43.400 INFO:teuthology.orchestra.run.smithi116.stdout:smithi117 172.21.15.117 2024-03-24T09:21:43.400 INFO:teuthology.orchestra.run.smithi116.stdout:smithi143 172.21.15.143 2024-03-24T09:21:43.400 INFO:teuthology.orchestra.run.smithi116.stdout:3 hosts in cluster 2024-03-24T09:21:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f50740a2150 msgr2=0x7f50740a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f50740a2150 0x7f50740a4610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5080011490 tx=0x7f5080009260 comp rx=0 tx=0).stop 2024-03-24T09:21:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 msgr2=0x7f508c198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 0x7f508c198250 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f507c00e9a0 tx=0x7f507c00ecb0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 shutdown_connections 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f508c102cd0 0x7f508c19f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f50740a2150 0x7f50740a4610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f508c101130 0x7f508c198250 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 --2- 172.21.15.116:0/2278500255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f508c102330 0x7f508c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 >> 172.21.15.116:0/2278500255 conn(0x7f508c0fc6f0 msgr2=0x7f508c06a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 shutdown_connections 2024-03-24T09:21:43.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:43.401+0000 7f5090e3a700 1 -- 172.21.15.116:0/2278500255 wait complete. 2024-03-24T09:21:44.096 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T09:21:44.344 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:45.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:45 smithi116 ceph-mon[32207]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:45.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:45 smithi116 ceph-mon[32207]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 459 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:45 smithi143 ceph-mon[35965]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:45 smithi143 ceph-mon[35965]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 459 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:45 smithi117 ceph-mon[35788]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:45 smithi117 ceph-mon[35788]: pgmap v358: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 459 B/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 -- 172.21.15.116:0/1784611098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 msgr2=0x7fdc1c101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 --2- 172.21.15.116:0/1784611098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c101880 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009990 tx=0x7fdc04009ca0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 -- 172.21.15.116:0/1784611098 shutdown_connections 2024-03-24T09:21:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 --2- 172.21.15.116:0/1784611098 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1c101dc0 0x7fdc1c109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 --2- 172.21.15.116:0/1784611098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 0x7fdc1c102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 --2- 172.21.15.116:0/1784611098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c101880 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 -- 172.21.15.116:0/1784611098 >> 172.21.15.116:0/1784611098 conn(0x7fdc1c0fbe50 msgr2=0x7fdc1c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 -- 172.21.15.116:0/1784611098 shutdown_connections 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.048+0000 7fdc21a78700 1 -- 172.21.15.116:0/1784611098 wait complete. 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 Processor -- start 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 -- start start 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c1051c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1c101dc0 0x7fdc1c103810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 0x7fdc1c103d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1044d0 con 0x7fdc1c101420 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c104640 con 0x7fdc1c1026e0 2024-03-24T09:21:46.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc21a78700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1a21b0 con 0x7fdc1c101dc0 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc1b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c1051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc1b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c1051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37970/0 (socket says 172.21.15.116:37970) 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc1bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 0x7fdc1c103d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc1b7fe700 1 -- 172.21.15.116:0/3023409511 learned_addr learned my addr 172.21.15.116:0/3023409511 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.049+0000 7fdc1affd700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1c101dc0 0x7fdc1c103810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1b7fe700 1 -- 172.21.15.116:0/3023409511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1c101dc0 msgr2=0x7fdc1c103810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1b7fe700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1c101dc0 0x7fdc1c103810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1b7fe700 1 -- 172.21.15.116:0/3023409511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 msgr2=0x7fdc1c103d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1b7fe700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 0x7fdc1c103d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1b7fe700 1 -- 172.21.15.116:0/3023409511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc040095f0 con 0x7fdc1c101420 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1bfff700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 0x7fdc1c103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:46.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc1b7fe700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c1051c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00efc0 tx=0x7fdc0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:46.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c018070 con 0x7fdc1c101420 2024-03-24T09:21:46.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdc0c00f040 con 0x7fdc1c101420 2024-03-24T09:21:46.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c1a2430 con 0x7fdc1c101420 2024-03-24T09:21:46.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c004280 con 0x7fdc1c101420 2024-03-24T09:21:46.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.050+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1a2980 con 0x7fdc1c101420 2024-03-24T09:21:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.051+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1c068ec0 con 0x7fdc1c101420 2024-03-24T09:21:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.052+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fdc0c003a00 con 0x7fdc1c101420 2024-03-24T09:21:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.053+0000 7fdc18ff9700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdc080a21d0 0x7fdc080a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.053+0000 7fdc1affd700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdc080a21d0 0x7fdc080a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.053+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdc0c0cde50 con 0x7fdc1c101420 2024-03-24T09:21:46.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.054+0000 7fdc1affd700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdc080a21d0 0x7fdc080a4690 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400b2e0 tx=0x7fdc0400b750 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.058+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdc0c096ae0 con 0x7fdc1c101420 2024-03-24T09:21:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.263+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 --> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fdc1c100a20 con 0x7fdc080a21d0 2024-03-24T09:21:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.264+0000 7fdc18ff9700 1 -- 172.21.15.116:0/3023409511 <== mgr.14148 v2:172.21.15.116:6800/2363601294 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fdc1c100a20 con 0x7fdc080a21d0 2024-03-24T09:21:46.265 INFO:teuthology.orchestra.run.smithi116.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T09:21:46.265 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T09:21:46.266 INFO:teuthology.orchestra.run.smithi116.stdout:smithi117 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V6400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T09:21:46.266 INFO:teuthology.orchestra.run.smithi116.stdout:smithi143 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330073400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T09:21:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.266+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdc080a21d0 msgr2=0x7fdc080a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdc080a21d0 0x7fdc080a4690 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400b2e0 tx=0x7fdc0400b750 comp rx=0 tx=0).stop 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 msgr2=0x7fdc1c1051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c1051c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00efc0 tx=0x7fdc0c00c3a0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 shutdown_connections 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c1026e0 0x7fdc1c103d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdc080a21d0 0x7fdc080a4690 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc1c101420 0x7fdc1c1051c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 --2- 172.21.15.116:0/3023409511 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc1c101dc0 0x7fdc1c103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 >> 172.21.15.116:0/3023409511 conn(0x7fdc1c0fbe50 msgr2=0x7fdc1c100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 shutdown_connections 2024-03-24T09:21:46.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:46.267+0000 7fdc21a78700 1 -- 172.21.15.116:0/3023409511 wait complete. 2024-03-24T09:21:47.001 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T09:21:47.010 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:21:47.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T09:21:47.253 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:47 smithi143 ceph-mon[35965]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 409 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:47 smithi117 ceph-mon[35788]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 409 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:47 smithi116 ceph-mon[32207]: pgmap v359: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 409 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:48.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:48 smithi143 ceph-mon[35965]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:48 smithi117 ceph-mon[35788]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:48.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:48 smithi116 ceph-mon[32207]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:21:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1999412411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102330 msgr2=0x7fa824102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:49.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1999412411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102330 0x7fa824102790 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa8140099c0 tx=0x7fa814009cd0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1999412411 shutdown_connections 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1999412411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa824102cd0 0x7fa824109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1999412411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa824101130 0x7fa824101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1999412411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102330 0x7fa824102790 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1999412411 >> 172.21.15.116:0/1999412411 conn(0x7fa8240fc6d0 msgr2=0x7fa8240feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1999412411 shutdown_connections 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.088+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1999412411 wait complete. 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 Processor -- start 2024-03-24T09:21:49.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 -- start start 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa824101130 0x7fa824198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa824102330 0x7fa824198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 0x7fa82419f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa82419fce0 con 0x7fa824102cd0 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa82419fe20 con 0x7fa824102330 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82bd2c700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa82419ffb0 con 0x7fa824101130 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82a2c9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 0x7fa82419f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82a2c9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 0x7fa82419f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59096/0 (socket says 172.21.15.116:59096) 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa82a2c9700 1 -- 172.21.15.116:0/1605431365 learned_addr learned my addr 172.21.15.116:0/1605431365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:49.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa8292c7700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa824102330 0x7fa824198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.089+0000 7fa829ac8700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa824101130 0x7fa824198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82a2c9700 1 -- 172.21.15.116:0/1605431365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa824101130 msgr2=0x7fa824198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82a2c9700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa824101130 0x7fa824198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82a2c9700 1 -- 172.21.15.116:0/1605431365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa824102330 msgr2=0x7fa824198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82a2c9700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa824102330 0x7fa824198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82a2c9700 1 -- 172.21.15.116:0/1605431365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8140095f0 con 0x7fa824102cd0 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82a2c9700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 0x7fa82419f710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00ef60 tx=0x7fa81c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c018070 con 0x7fa824102cd0 2024-03-24T09:21:49.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa81c00f040 con 0x7fa824102cd0 2024-03-24T09:21:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8241a0230 con 0x7fa824102cd0 2024-03-24T09:21:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c00ccf0 con 0x7fa824102cd0 2024-03-24T09:21:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.090+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8241a0730 con 0x7fa824102cd0 2024-03-24T09:21:49.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.091+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa824108630 con 0x7fa824102cd0 2024-03-24T09:21:49.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.092+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fa81c004ad0 con 0x7fa824102cd0 2024-03-24T09:21:49.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.092+0000 7fa81affd700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa8100a21d0 0x7fa8100a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:49.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.093+0000 7fa829ac8700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa8100a21d0 0x7fa8100a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:49.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.093+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fa81c0cdaa0 con 0x7fa824102cd0 2024-03-24T09:21:49.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.093+0000 7fa829ac8700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa8100a21d0 0x7fa8100a4690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa824102190 tx=0x7fa820009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:49.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.097+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa81c096730 con 0x7fa824102cd0 2024-03-24T09:21:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:49 smithi143 ceph-mon[35965]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 343 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:49 smithi117 ceph-mon[35788]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 343 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:49.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:49 smithi116 ceph-mon[32207]: pgmap v360: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 343 B/s wr, 10 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.332+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fa824068ec0 con 0x7fa824102cd0 2024-03-24T09:21:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.332+0000 7fa81affd700 1 -- 172.21.15.116:0/1605431365 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3106 (secure 0 0 0) 0x7fa81c099df0 con 0x7fa824102cd0 2024-03-24T09:21:49.333 INFO:teuthology.orchestra.run.smithi116.stdout:e28 2024-03-24T09:21:49.333 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T09:21:49.333 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T09:21:49.333 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 28 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-24T09:19:38.431545+0000 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-24T09:20:06.925494+0000 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 300 2024-03-24T09:21:49.334 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 0 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 5 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:in 0,1,2,3,4 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24515,1=24484,2=24451,3=24443,4=24422} 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-24T09:21:49.335 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3,4] 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 0 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:qdb_cluster leader: 24515 members: 24484,24515,24451,24443,24422 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b{0:24515} state up:active seq 3 addr [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.h{0:24473} state up:standby-replay seq 1 addr [v2:172.21.15.117:6836/309438710,v1:172.21.15.117:6837/309438710] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.i{1:24484} state up:active seq 11 addr [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c{1:24469} state up:standby-replay seq 1 addr [v2:172.21.15.143:6836/3300095096,v1:172.21.15.143:6837/3300095096] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.336 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.f{2:24451} state up:active seq 14 addr [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.l{2:24412} state up:standby-replay seq 1 addr [v2:172.21.15.143:6832/2011428167,v1:172.21.15.143:6833/2011428167] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.e{3:24443} state up:active seq 15 addr [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a{3:14649} state up:standby-replay seq 1 addr [v2:172.21.15.116:6840/891659164,v1:172.21.15.116:6841/891659164] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.k{4:24422} state up:active seq 17 addr [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.g{4:14631} state up:standby-replay seq 1 addr [v2:172.21.15.116:6838/1877964932,v1:172.21.15.116:6839/1877964932] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.j{-1:14562} state up:standby seq 1 addr [v2:172.21.15.116:6834/203590866,v1:172.21.15.116:6835/203590866] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.337 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d{-1:14574} state up:standby seq 1 addr [v2:172.21.15.116:6836/1967493794,v1:172.21.15.116:6837/1967493794] compat {c=[1],r=[1],i=[1fff]}] 2024-03-24T09:21:49.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.334+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa8100a21d0 msgr2=0x7fa8100a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.334+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa8100a21d0 0x7fa8100a4690 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa824102190 tx=0x7fa820009260 comp rx=0 tx=0).stop 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.334+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 msgr2=0x7fa82419f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.334+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 0x7fa82419f710 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00ef60 tx=0x7fa81c00ef90 comp rx=0 tx=0).stop 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.334+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 shutdown_connections 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fa8100a21d0 0x7fa8100a4690 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa824102cd0 0x7fa82419f710 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa824101130 0x7fa824198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 --2- 172.21.15.116:0/1605431365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa824102330 0x7fa824198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 >> 172.21.15.116:0/1605431365 conn(0x7fa8240fc6d0 msgr2=0x7fa824105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:49.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 shutdown_connections 2024-03-24T09:21:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:49.335+0000 7fa82bd2c700 1 -- 172.21.15.116:0/1605431365 wait complete. 2024-03-24T09:21:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:21:49.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-24T09:21:50.217 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:50.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:50 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1605431365' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:21:50.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:50 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1605431365' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:21:50.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:50 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1605431365' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T09:21:51.289 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:51 smithi116 ceph-mon[32207]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:51 smithi117 ceph-mon[35788]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:51 smithi143 ceph-mon[35965]: pgmap v361: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 -- 172.21.15.116:0/3299651521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8102cd0 msgr2=0x7fbfa8109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 --2- 172.21.15.116:0/3299651521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8102cd0 0x7fbfa8109580 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000b0d0 tx=0x7fbfa000b3e0 comp rx=0 tx=0).stop 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 -- 172.21.15.116:0/3299651521 shutdown_connections 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 --2- 172.21.15.116:0/3299651521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8102cd0 0x7fbfa8109580 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 --2- 172.21.15.116:0/3299651521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8101130 0x7fbfa8101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 --2- 172.21.15.116:0/3299651521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa8102330 0x7fbfa8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:51.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 -- 172.21.15.116:0/3299651521 >> 172.21.15.116:0/3299651521 conn(0x7fbfa80fc6d0 msgr2=0x7fbfa80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 -- 172.21.15.116:0/3299651521 shutdown_connections 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.826+0000 7fbfad99a700 1 -- 172.21.15.116:0/3299651521 wait complete. 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 Processor -- start 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 -- start start 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 0x7fbfa8198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8102330 0x7fbfa8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa8102cd0 0x7fbfa819f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa819fe70 con 0x7fbfa8101130 2024-03-24T09:21:51.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa819ffb0 con 0x7fbfa8102330 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfad99a700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa81a0140 con 0x7fbfa8102cd0 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfa6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 0x7fbfa8198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.827+0000 7fbfa6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 0x7fbfa8198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59124/0 (socket says 172.21.15.116:59124) 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 -- 172.21.15.116:0/2888103310 learned_addr learned my addr 172.21.15.116:0/2888103310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa77fe700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa8102cd0 0x7fbfa819f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa67fc700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8102330 0x7fbfa8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 -- 172.21.15.116:0/2888103310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa8102cd0 msgr2=0x7fbfa819f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa8102cd0 0x7fbfa819f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 -- 172.21.15.116:0/2888103310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8102330 msgr2=0x7fbfa8198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:51.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8102330 0x7fbfa8198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 -- 172.21.15.116:0/2888103310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa0009d00 con 0x7fbfa8101130 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa67fc700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8102330 0x7fbfa8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfa6ffd700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 0x7fbfa8198330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9a0 tx=0x7fbf9800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98009790 con 0x7fbfa8101130 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa81a0420 con 0x7fbfa8101130 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbf9800c810 con 0x7fbfa8101130 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf98003c40 con 0x7fbfa8101130 2024-03-24T09:21:51.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.828+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa81a0920 con 0x7fbfa8101130 2024-03-24T09:21:51.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.829+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa8068ec0 con 0x7fbfa8101130 2024-03-24T09:21:51.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.830+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fbf9800c980 con 0x7fbfa8101130 2024-03-24T09:21:51.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.831+0000 7fbfac998700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbf940a2150 0x7fbf940a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:51.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.831+0000 7fbfa67fc700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbf940a2150 0x7fbf940a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:51.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.831+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fbf98014070 con 0x7fbfa8101130 2024-03-24T09:21:51.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.831+0000 7fbfa67fc700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbf940a2150 0x7fbf940a4610 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbf90011490 tx=0x7fbf90009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:51.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:51.835+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbf98096e00 con 0x7fbfa8101130 2024-03-24T09:21:52.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.032+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fbfa81a0e90 con 0x7fbfa8101130 2024-03-24T09:21:52.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.033+0000 7fbfac998700 1 -- 172.21.15.116:0/2888103310 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5734 (secure 0 0 0) 0x7fbf9809a4c0 con 0x7fbfa8101130 2024-03-24T09:21:52.033 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 93 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-24T09:11:38.627995+0000 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-24T09:21:36.028172+0000 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:crush_version 27 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:full_ratio 0.95 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:backfillfull_ratio 0.9 2024-03-24T09:21:52.034 INFO:teuthology.orchestra.run.smithi116.stdout:nearfull_ratio 0.85 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:require_min_compat_client luminous 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:min_compat_client luminous 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:require_osd_release squid 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:stretch_mode_enabled false 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:max_osd 12 2024-03-24T09:21:52.035 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6802/2942581569,v1:172.21.15.116:6803/2942581569] [v2:172.21.15.116:6804/2942581569,v1:172.21.15.116:6805/2942581569] exists,up 8b186fcf-17c6-409c-9ca6-7c8373830eea 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/2635729600,v1:172.21.15.117:6801/2635729600] [v2:172.21.15.117:6802/2635729600,v1:172.21.15.117:6803/2635729600] exists,up 62462f7a-4125-4c73-beac-3aee728ab940 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6800/541711081,v1:172.21.15.143:6801/541711081] [v2:172.21.15.143:6802/541711081,v1:172.21.15.143:6803/541711081] exists,up 9f4d76a1-5218-4904-8b3e-0d782ada3628 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6810/2981524673,v1:172.21.15.116:6811/2981524673] [v2:172.21.15.116:6812/2981524673,v1:172.21.15.116:6813/2981524673] exists,up 3f2f25e8-51f7-4048-a704-c27813a60056 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/3606274523,v1:172.21.15.117:6809/3606274523] [v2:172.21.15.117:6810/3606274523,v1:172.21.15.117:6811/3606274523] exists,up 2a944b4a-1532-4713-a537-792acac8425c 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6808/1015586525,v1:172.21.15.143:6809/1015586525] [v2:172.21.15.143:6810/1015586525,v1:172.21.15.143:6811/1015586525] exists,up aaadd9c9-b1bc-4ca0-bb20-760ee737abf9 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/254462190,v1:172.21.15.116:6819/254462190] [v2:172.21.15.116:6820/254462190,v1:172.21.15.116:6821/254462190] exists,up 09e3ad38-6286-4d42-9ddc-ccfa80ace379 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/3523258939,v1:172.21.15.117:6817/3523258939] [v2:172.21.15.117:6818/3523258939,v1:172.21.15.117:6819/3523258939] exists,up fc83d789-61b0-4c26-93ff-c7ddcba3cc21 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 up in weight 1 up_from 59 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6816/3224917064,v1:172.21.15.143:6817/3224917064] [v2:172.21.15.143:6818/3224917064,v1:172.21.15.143:6819/3224917064] exists,up b9ce166d-03cd-41ec-a11e-f93e301cfb42 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6826/2856079084,v1:172.21.15.116:6827/2856079084] [v2:172.21.15.116:6828/2856079084,v1:172.21.15.116:6829/2856079084] exists,up 5c25c107-ff19-45fb-84e3-c994805faa36 2024-03-24T09:21:52.036 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/3860527713,v1:172.21.15.117:6825/3860527713] [v2:172.21.15.117:6826/3860527713,v1:172.21.15.117:6827/3860527713] exists,up 33d2b62d-960d-458f-aa10-364ffb11ee53 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.143:6824/2884681125,v1:172.21.15.143:6825/2884681125] [v2:172.21.15.143:6826/2884681125,v1:172.21.15.143:6827/2884681125] exists,up 0397d0cd-1f13-475b-85e6-332bfd32d77d 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.a [2,11] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.b [8,11] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.c [3,9] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.10 [5,9] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.14 [1,9] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.17 [4,6] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.1a [5,7] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.1b [3,6] 2024-03-24T09:21:52.037 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.1d [4,7] 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.3a [4,8] 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.a [5,8] 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.f [9,8] 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.12 [9,10] 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 4.18 [9,8] 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3305720445 expires 2024-03-25T09:12:33.839806+0000 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3598626147 expires 2024-03-25T09:12:33.839806+0000 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/3871815735 expires 2024-03-25T09:12:33.839806+0000 2024-03-24T09:21:52.038 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/3871815735 expires 2024-03-25T09:12:33.839806+0000 2024-03-24T09:21:52.039 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/285563997 expires 2024-03-25T09:12:07.749841+0000 2024-03-24T09:21:52.039 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3855909741 expires 2024-03-25T09:12:07.749841+0000 2024-03-24T09:21:52.039 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/448304506 expires 2024-03-25T09:12:07.749841+0000 2024-03-24T09:21:52.039 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/285563997 expires 2024-03-25T09:12:07.749841+0000 2024-03-24T09:21:52.039 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3108228013 expires 2024-03-25T09:12:07.749841+0000 2024-03-24T09:21:52.039 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2688022316 expires 2024-03-25T09:12:33.839806+0000 2024-03-24T09:21:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbf940a2150 msgr2=0x7fbf940a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbf940a2150 0x7fbf940a4610 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbf90011490 tx=0x7fbf90009260 comp rx=0 tx=0).stop 2024-03-24T09:21:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 msgr2=0x7fbfa8198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 0x7fbfa8198330 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9a0 tx=0x7fbf9800ecb0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 shutdown_connections 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbfa8102cd0 0x7fbfa819f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fbf940a2150 0x7fbf940a4610 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfa8101130 0x7fbfa8198330 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 --2- 172.21.15.116:0/2888103310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfa8102330 0x7fbfa8198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 >> 172.21.15.116:0/2888103310 conn(0x7fbfa80fc6d0 msgr2=0x7fbfa80fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 shutdown_connections 2024-03-24T09:21:52.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.036+0000 7fbfad99a700 1 -- 172.21.15.116:0/2888103310 wait complete. 2024-03-24T09:21:52.718 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-24T09:21:52.733 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 -- 172.21.15.116:0/1373175158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed38107420 msgr2=0x7fed38109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 --2- 172.21.15.116:0/1373175158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed38107420 0x7fed38109870 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fed30009ca0 tx=0x7fed3002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 -- 172.21.15.116:0/1373175158 shutdown_connections 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 --2- 172.21.15.116:0/1373175158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed38107420 0x7fed38109870 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 --2- 172.21.15.116:0/1373175158 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed38104af0 0x7fed38106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 --2- 172.21.15.116:0/1373175158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed381021c0 0x7fed381045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 -- 172.21.15.116:0/1373175158 >> 172.21.15.116:0/1373175158 conn(0x7fed380fbbf0 msgr2=0x7fed380fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 -- 172.21.15.116:0/1373175158 shutdown_connections 2024-03-24T09:21:52.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.839+0000 7fed3ce23640 1 -- 172.21.15.116:0/1373175158 wait complete. 2024-03-24T09:21:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 Processor -- start 2024-03-24T09:21:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 -- start start 2024-03-24T09:21:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 0x7fed381976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed38104af0 0x7fed38197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed38107420 0x7fed3819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:52.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fed3819f1a0 con 0x7fed381021c0 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed3819f310 con 0x7fed38107420 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3ce23640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fed3819f4d0 con 0x7fed38104af0 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 0x7fed381976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 0x7fed381976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59142/0 (socket says 172.21.15.116:59142) 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed3659c640 1 -- 172.21.15.116:0/2697951856 learned_addr learned my addr 172.21.15.116:0/2697951856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed2ffff640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed38104af0 0x7fed38197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.840+0000 7fed36d9d640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed38107420 0x7fed3819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3659c640 1 -- 172.21.15.116:0/2697951856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed38104af0 msgr2=0x7fed38197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3659c640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed38104af0 0x7fed38197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3659c640 1 -- 172.21.15.116:0/2697951856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed38107420 msgr2=0x7fed3819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3659c640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed38107420 0x7fed3819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3659c640 1 -- 172.21.15.116:0/2697951856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed30009900 con 0x7fed381021c0 2024-03-24T09:21:52.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3659c640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 0x7fed381976b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fed2800e9f0 tx=0x7fed2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28009700 con 0x7fed381021c0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fed2800cd80 con 0x7fed381021c0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3819f7b0 con 0x7fed381021c0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28018840 con 0x7fed381021c0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.841+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3819fcc0 con 0x7fed381021c0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.842+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed380fd6e0 con 0x7fed381021c0 2024-03-24T09:21:52.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.843+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fed280189a0 con 0x7fed381021c0 2024-03-24T09:21:52.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.844+0000 7fed2f7fe640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fed240a20c0 0x7fed240a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:52.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.844+0000 7fed2ffff640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fed240a20c0 0x7fed240a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:52.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.844+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fed28014070 con 0x7fed381021c0 2024-03-24T09:21:52.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.844+0000 7fed2ffff640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fed240a20c0 0x7fed240a4580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fed38198bc0 tx=0x7fed2000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:52.847+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fed280974b0 con 0x7fed381021c0 2024-03-24T09:21:53.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.025+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed3810d170 con 0x7fed381021c0 2024-03-24T09:21:53.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.026+0000 7fed2f7fe640 1 -- 172.21.15.116:0/2697951856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24911 (secure 0 0 0) 0x7fed3810d170 con 0x7fed381021c0 2024-03-24T09:21:53.027 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:53.027 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":93,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:21:36.028172+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fed240a20c0 msgr2=0x7fed240a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fed240a20c0 0x7fed240a4580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fed38198bc0 tx=0x7fed2000b410 comp rx=0 tx=0).stop 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 msgr2=0x7fed381976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 0x7fed381976b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fed2800e9f0 tx=0x7fed2800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 shutdown_connections 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed38107420 0x7fed3819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fed240a20c0 0x7fed240a4580 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed38104af0 0x7fed38197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 --2- 172.21.15.116:0/2697951856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed381021c0 0x7fed381976b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 >> 172.21.15.116:0/2697951856 conn(0x7fed380fbbf0 msgr2=0x7fed38102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 shutdown_connections 2024-03-24T09:21:53.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.029+0000 7fed3ce23640 1 -- 172.21.15.116:0/2697951856 wait complete. 2024-03-24T09:21:53.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:21:53.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:21:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 -- 172.21.15.116:0/4041884699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 msgr2=0x7fb408101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 --2- 172.21.15.116:0/4041884699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408101ca0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009900 tx=0x7fb3f802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 -- 172.21.15.116:0/4041884699 shutdown_connections 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 --2- 172.21.15.116:0/4041884699 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4081021e0 0x7fb408108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 --2- 172.21.15.116:0/4041884699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408101ca0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 --2- 172.21.15.116:0/4041884699 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 0x7fb408100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 -- 172.21.15.116:0/4041884699 >> 172.21.15.116:0/4041884699 conn(0x7fb4080fbbe0 msgr2=0x7fb4080fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 -- 172.21.15.116:0/4041884699 shutdown_connections 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 -- 172.21.15.116:0/4041884699 wait complete. 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 Processor -- start 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.146+0000 7fb40e62e640 1 -- start start 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 0x7fb408196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4081021e0 0x7fb40819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb40819e070 con 0x7fb408101820 2024-03-24T09:21:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb40819e1e0 con 0x7fb408100630 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb40819e3a0 con 0x7fb4081021e0 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 0x7fb408196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59164/0 (socket says 172.21.15.116:59164) 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb4077fe640 1 -- 172.21.15.116:0/420490246 learned_addr learned my addr 172.21.15.116:0/420490246 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40cbcb640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4081021e0 0x7fb40819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 -- 172.21.15.116:0/420490246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4081021e0 msgr2=0x7fb40819db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4081021e0 0x7fb40819db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 -- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 msgr2=0x7fb408196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 -- 172.21.15.116:0/420490246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc009490 con 0x7fb408100630 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb4077fe640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb407fff640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 0x7fb408196580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00ead0 tx=0x7fb3fc00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc013070 con 0x7fb408100630 2024-03-24T09:21:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3fc00bba0 con 0x7fb408100630 2024-03-24T09:21:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.147+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f8009560 con 0x7fb408100630 2024-03-24T09:21:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.148+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc016610 con 0x7fb408100630 2024-03-24T09:21:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.148+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb40806a120 con 0x7fb408100630 2024-03-24T09:21:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.148+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb40806ab60 con 0x7fb408100630 2024-03-24T09:21:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.149+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fb3fc00bd10 con 0x7fb408100630 2024-03-24T09:21:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.150+0000 7fb4057fa640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb3f00a20c0 0x7fb3f00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.150+0000 7fb4077fe640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb3f00a20c0 0x7fb3f00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.150+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fb3fc0cf7d0 con 0x7fb408100630 2024-03-24T09:21:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.150+0000 7fb4077fe640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb3f00a20c0 0x7fb3f00a4580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb408197a90 tx=0x7fb3f802c770 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:53.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.155+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb3fc098480 con 0x7fb408100630 2024-03-24T09:21:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:53 smithi117 ceph-mon[35788]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-24T09:21:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2888103310' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T09:21:53.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:53 smithi116 ceph-mon[32207]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-24T09:21:53.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2888103310' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T09:21:53.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.329+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb4081073c0 con 0x7fb408100630 2024-03-24T09:21:53.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.329+0000 7fb4057fa640 1 -- 172.21.15.116:0/420490246 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fb3fc09bb40 con 0x7fb408100630 2024-03-24T09:21:53.330 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:21:53.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb3f00a20c0 msgr2=0x7fb3f00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb3f00a20c0 0x7fb3f00a4580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb408197a90 tx=0x7fb3f802c770 comp rx=0 tx=0).stop 2024-03-24T09:21:53.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 msgr2=0x7fb408196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 0x7fb408196580 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00ead0 tx=0x7fb3fc00ef90 comp rx=0 tx=0).stop 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 shutdown_connections 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fb3f00a20c0 0x7fb3f00a4580 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4081021e0 0x7fb40819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb408101820 0x7fb408196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 --2- 172.21.15.116:0/420490246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb408100630 0x7fb408196580 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 >> 172.21.15.116:0/420490246 conn(0x7fb4080fbbe0 msgr2=0x7fb408104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 shutdown_connections 2024-03-24T09:21:53.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.331+0000 7fb40e62e640 1 -- 172.21.15.116:0/420490246 wait complete. 2024-03-24T09:21:53.343 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 -- 172.21.15.116:0/1497890509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30101800 msgr2=0x7f4a30101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 --2- 172.21.15.116:0/1497890509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30101800 0x7f4a30101c80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4a18009900 tx=0x7f4a1802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 -- 172.21.15.116:0/1497890509 shutdown_connections 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 --2- 172.21.15.116:0/1497890509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a301021c0 0x7f4a30108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 --2- 172.21.15.116:0/1497890509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30101800 0x7f4a30101c80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 --2- 172.21.15.116:0/1497890509 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a30100610 0x7f4a30100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 -- 172.21.15.116:0/1497890509 >> 172.21.15.116:0/1497890509 conn(0x7f4a300fbba0 msgr2=0x7f4a300fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 -- 172.21.15.116:0/1497890509 shutdown_connections 2024-03-24T09:21:53.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.447+0000 7f4a359dd640 1 -- 172.21.15.116:0/1497890509 wait complete. 2024-03-24T09:21:53.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 Processor -- start 2024-03-24T09:21:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 -- start start 2024-03-24T09:21:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 0x7f4a30197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a30101800 0x7f4a30197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a301021c0 0x7f4a3019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3019f1f0 con 0x7f4a30100610 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3019f360 con 0x7f4a301021c0 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a359dd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3019f520 con 0x7f4a30101800 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 0x7f4a30197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a2effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 0x7f4a30197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59176/0 (socket says 172.21.15.116:59176) 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a2effd640 1 -- 172.21.15.116:0/332036792 learned_addr learned my addr 172.21.15.116:0/332036792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a30101800 0x7f4a30197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.448+0000 7f4a2f7fe640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a301021c0 0x7f4a3019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a2effd640 1 -- 172.21.15.116:0/332036792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a30101800 msgr2=0x7f4a30197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a2effd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a30101800 0x7f4a30197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a2effd640 1 -- 172.21.15.116:0/332036792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a301021c0 msgr2=0x7f4a3019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a2effd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a301021c0 0x7f4a3019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a2effd640 1 -- 172.21.15.116:0/332036792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a18009560 con 0x7f4a30100610 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a2effd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 0x7f4a30197700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000eb70 tx=0x7f4a2000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20003b70 con 0x7f4a30100610 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4a2000ee70 con 0x7f4a30100610 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3019f800 con 0x7f4a30100610 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20010690 con 0x7f4a30100610 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.449+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3019fc90 con 0x7f4a30100610 2024-03-24T09:21:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.450+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a301079a0 con 0x7f4a30100610 2024-03-24T09:21:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.451+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f4a20004020 con 0x7f4a30100610 2024-03-24T09:21:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.451+0000 7f4a349db640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a1c0a20c0 0x7f4a1c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.452+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a1c0a20c0 0x7f4a1c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.452+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4a20014070 con 0x7f4a30100610 2024-03-24T09:21:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.452+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a1c0a20c0 0x7f4a1c0a4580 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a18000c00 tx=0x7f4a1802c770 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.456+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4a200972a0 con 0x7f4a30100610 2024-03-24T09:21:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:53 smithi143 ceph-mon[35965]: pgmap v362: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-03-24T09:21:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2888103310' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T09:21:53.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.629+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a3002d200 con 0x7f4a30100610 2024-03-24T09:21:53.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.629+0000 7f4a349db640 1 -- 172.21.15.116:0/332036792 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f4a20003e60 con 0x7f4a30100610 2024-03-24T09:21:53.629 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a1c0a20c0 msgr2=0x7f4a1c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a1c0a20c0 0x7f4a1c0a4580 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a18000c00 tx=0x7f4a1802c770 comp rx=0 tx=0).stop 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 msgr2=0x7f4a30197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 0x7f4a30197700 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4a2000eb70 tx=0x7f4a2000c6a0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 shutdown_connections 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a301021c0 0x7f4a3019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f4a1c0a20c0 0x7f4a1c0a4580 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4a30101800 0x7f4a30197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 --2- 172.21.15.116:0/332036792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a30100610 0x7f4a30197700 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 >> 172.21.15.116:0/332036792 conn(0x7f4a300fbba0 msgr2=0x7f4a30104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 shutdown_connections 2024-03-24T09:21:53.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.631+0000 7f4a359dd640 1 -- 172.21.15.116:0/332036792 wait complete. 2024-03-24T09:21:53.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:21:53.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 -- 172.21.15.116:0/1630109407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c1021a0 msgr2=0x7efe7c104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 --2- 172.21.15.116:0/1630109407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c1021a0 0x7efe7c104590 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efe74009ca0 tx=0x7efe7402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 -- 172.21.15.116:0/1630109407 shutdown_connections 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 --2- 172.21.15.116:0/1630109407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe7c107400 0x7efe7c109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 --2- 172.21.15.116:0/1630109407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c104ad0 0x7efe7c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 --2- 172.21.15.116:0/1630109407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c1021a0 0x7efe7c104590 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 -- 172.21.15.116:0/1630109407 >> 172.21.15.116:0/1630109407 conn(0x7efe7c0fbbb0 msgr2=0x7efe7c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 -- 172.21.15.116:0/1630109407 shutdown_connections 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.746+0000 7efe83423640 1 -- 172.21.15.116:0/1630109407 wait complete. 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 Processor -- start 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 -- start start 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c1021a0 0x7efe7c1976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe7c104ad0 0x7efe7c197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 0x7efe7c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe7c19f1b0 con 0x7efe7c107400 2024-03-24T09:21:53.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efe7c19f320 con 0x7efe7c1021a0 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe83423640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efe7c19f4e0 con 0x7efe7c104ad0 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 0x7efe7c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 0x7efe7c19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59196/0 (socket says 172.21.15.116:59196) 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe811bf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c1021a0 0x7efe7c1976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 -- 172.21.15.116:0/3054953522 learned_addr learned my addr 172.21.15.116:0/3054953522 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe809be640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe7c104ad0 0x7efe7c197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 -- 172.21.15.116:0/3054953522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe7c104ad0 msgr2=0x7efe7c197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe7c104ad0 0x7efe7c197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 -- 172.21.15.116:0/3054953522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c1021a0 msgr2=0x7efe7c1976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c1021a0 0x7efe7c1976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe819c0640 1 -- 172.21.15.116:0/3054953522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe74009900 con 0x7efe7c107400 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.747+0000 7efe811bf640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c1021a0 0x7efe7c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.748+0000 7efe819c0640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 0x7efe7c19ec70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efe7000e970 tx=0x7efe7000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.748+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe70009720 con 0x7efe7c107400 2024-03-24T09:21:53.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.748+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efe700054d0 con 0x7efe7c107400 2024-03-24T09:21:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.748+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe7c19f7c0 con 0x7efe7c107400 2024-03-24T09:21:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.748+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe70010430 con 0x7efe7c107400 2024-03-24T09:21:53.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.748+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe7c19fcd0 con 0x7efe7c107400 2024-03-24T09:21:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.749+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe7c0fd6a0 con 0x7efe7c107400 2024-03-24T09:21:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.750+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7efe70004f10 con 0x7efe7c107400 2024-03-24T09:21:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.751+0000 7efe6e7fc640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efe680a20c0 0x7efe680a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.751+0000 7efe811bf640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efe680a20c0 0x7efe680a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.751+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efe70014070 con 0x7efe7c107400 2024-03-24T09:21:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.751+0000 7efe811bf640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efe680a20c0 0x7efe680a4580 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efe74006010 tx=0x7efe740325d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.755+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efe70096fb0 con 0x7efe7c107400 2024-03-24T09:21:53.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.930+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efe7c02d1d0 con 0x7efe7c107400 2024-03-24T09:21:53.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.930+0000 7efe6e7fc640 1 -- 172.21.15.116:0/3054953522 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7efe7009a670 con 0x7efe7c107400 2024-03-24T09:21:53.931 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efe680a20c0 msgr2=0x7efe680a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efe680a20c0 0x7efe680a4580 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efe74006010 tx=0x7efe740325d0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 msgr2=0x7efe7c19ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 0x7efe7c19ec70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7efe7000e970 tx=0x7efe7000ed70 comp rx=0 tx=0).stop 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 shutdown_connections 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe7c107400 0x7efe7c19ec70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efe680a20c0 0x7efe680a4580 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efe7c104ad0 0x7efe7c197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 --2- 172.21.15.116:0/3054953522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe7c1021a0 0x7efe7c1976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 >> 172.21.15.116:0/3054953522 conn(0x7efe7c0fbbb0 msgr2=0x7efe7c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 shutdown_connections 2024-03-24T09:21:53.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:53.932+0000 7efe83423640 1 -- 172.21.15.116:0/3054953522 wait complete. 2024-03-24T09:21:53.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:21:54.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 -- 172.21.15.116:0/2624309291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4100630 msgr2=0x7f67e4100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 --2- 172.21.15.116:0/2624309291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4100630 0x7f67e4100a50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009900 tx=0x7f67d802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:54.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 -- 172.21.15.116:0/2624309291 shutdown_connections 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 --2- 172.21.15.116:0/2624309291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e41021e0 0x7f67e4108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 --2- 172.21.15.116:0/2624309291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67e4101820 0x7f67e4101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 --2- 172.21.15.116:0/2624309291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4100630 0x7f67e4100a50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 -- 172.21.15.116:0/2624309291 >> 172.21.15.116:0/2624309291 conn(0x7f67e40fbbc0 msgr2=0x7f67e40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 -- 172.21.15.116:0/2624309291 shutdown_connections 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.052+0000 7f67eaf13640 1 -- 172.21.15.116:0/2624309291 wait complete. 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 Processor -- start 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 -- start start 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e4100630 0x7f67e4197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 0x7f67e4197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67e41021e0 0x7f67e419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67e419f230 con 0x7f67e4101820 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f67e419f3a0 con 0x7f67e41021e0 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f67e419f560 con 0x7f67e4100630 2024-03-24T09:21:54.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 0x7f67e4197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 0x7f67e4197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59210/0 (socket says 172.21.15.116:59210) 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 -- 172.21.15.116:0/115568276 learned_addr learned my addr 172.21.15.116:0/115568276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e94b0640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67e41021e0 0x7f67e419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e8caf640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e4100630 0x7f67e4197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 -- 172.21.15.116:0/115568276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e4100630 msgr2=0x7f67e4197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e4100630 0x7f67e4197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 -- 172.21.15.116:0/115568276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67e41021e0 msgr2=0x7f67e419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67e41021e0 0x7f67e419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 -- 172.21.15.116:0/115568276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d8009560 con 0x7f67e4101820 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e3fff640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 0x7f67e4197c80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f67d000e9f0 tx=0x7f67d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0009700 con 0x7f67e4101820 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67d000cd80 con 0x7f67e4101820 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e419f840 con 0x7f67e4101820 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.053+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0018840 con 0x7f67e4101820 2024-03-24T09:21:54.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.054+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e419fd50 con 0x7f67e4101820 2024-03-24T09:21:54.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.055+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67e410bef0 con 0x7f67e4101820 2024-03-24T09:21:54.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.056+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f67d00189a0 con 0x7f67e4101820 2024-03-24T09:21:54.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.056+0000 7f67e1ffb640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f67d40a20c0 0x7f67d40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.056+0000 7f67e8caf640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f67d40a20c0 0x7f67d40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.056+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f67d0014070 con 0x7f67e4101820 2024-03-24T09:21:54.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.057+0000 7f67e8caf640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f67d40a20c0 0x7f67d40a4580 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f67d802c7d0 tx=0x7f67d8005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:54.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.061+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f67d0053280 con 0x7f67e4101820 2024-03-24T09:21:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.236+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f67e41a01e0 con 0x7f67e4101820 2024-03-24T09:21:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.236+0000 7f67e1ffb640 1 -- 172.21.15.116:0/115568276 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f67d0097050 con 0x7f67e4101820 2024-03-24T09:21:54.237 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.238+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f67d40a20c0 msgr2=0x7f67d40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.238+0000 7f67eaf13640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f67d40a20c0 0x7f67d40a4580 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f67d802c7d0 tx=0x7f67d8005fd0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.238+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 msgr2=0x7f67e4197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.238+0000 7f67eaf13640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 0x7f67e4197c80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f67d000e9f0 tx=0x7f67d000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 shutdown_connections 2024-03-24T09:21:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67e41021e0 0x7f67e419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67e4101820 0x7f67e4197c80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f67d40a20c0 0x7f67d40a4580 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 --2- 172.21.15.116:0/115568276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67e4100630 0x7f67e4197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 >> 172.21.15.116:0/115568276 conn(0x7f67e40fbbc0 msgr2=0x7f67e4069d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:54.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 shutdown_connections 2024-03-24T09:21:54.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.239+0000 7f67eaf13640 1 -- 172.21.15.116:0/115568276 wait complete. 2024-03-24T09:21:54.249 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2697951856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:54.249 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/420490246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.249 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/332036792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.249 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3054953522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:21:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2697951856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/420490246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/332036792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3054953522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 -- 172.21.15.116:0/159747382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63941029d0 msgr2=0x7f6394102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 --2- 172.21.15.116:0/159747382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63941029d0 0x7f6394102e50 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f6388009900 tx=0x7f638802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:54.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 -- 172.21.15.116:0/159747382 shutdown_connections 2024-03-24T09:21:54.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 --2- 172.21.15.116:0/159747382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6394103390 0x7f6394109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 --2- 172.21.15.116:0/159747382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f63941029d0 0x7f6394102e50 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 --2- 172.21.15.116:0/159747382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63940ffa00 0x7f63940ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 -- 172.21.15.116:0/159747382 >> 172.21.15.116:0/159747382 conn(0x7f63940fb330 msgr2=0x7f63940fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 -- 172.21.15.116:0/159747382 shutdown_connections 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 -- 172.21.15.116:0/159747382 wait complete. 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 Processor -- start 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.352+0000 7f639a69d640 1 -- start start 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63940ffa00 0x7f6394197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63941029d0 0x7f6394197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 0x7f639419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f639419f1f0 con 0x7f6394103390 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f639419f360 con 0x7f63941029d0 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f639419f520 con 0x7f63940ffa00 2024-03-24T09:21:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 0x7f639419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 0x7f639419ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59238/0 (socket says 172.21.15.116:59238) 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 -- 172.21.15.116:0/1563647924 learned_addr learned my addr 172.21.15.116:0/1563647924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f63937fe640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63941029d0 0x7f6394197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6393fff640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63940ffa00 0x7f6394197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 -- 172.21.15.116:0/1563647924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63940ffa00 msgr2=0x7f6394197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63940ffa00 0x7f6394197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 -- 172.21.15.116:0/1563647924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63941029d0 msgr2=0x7f6394197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63941029d0 0x7f6394197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 -- 172.21.15.116:0/1563647924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6388009560 con 0x7f6394103390 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f63937fe640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63941029d0 0x7f6394197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f6398c3a640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 0x7f639419ecb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f638c00eb10 tx=0x7f638c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c004b20 con 0x7f6394103390 2024-03-24T09:21:54.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f638c004c80 con 0x7f6394103390 2024-03-24T09:21:54.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c010690 con 0x7f6394103390 2024-03-24T09:21:54.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639419f800 con 0x7f6394103390 2024-03-24T09:21:54.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.353+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639419fce0 con 0x7f6394103390 2024-03-24T09:21:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.354+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6394108c40 con 0x7f6394103390 2024-03-24T09:21:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.355+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f638c005020 con 0x7f6394103390 2024-03-24T09:21:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.356+0000 7f63917fa640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f637c0a20c0 0x7f637c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.356+0000 7f6393fff640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f637c0a20c0 0x7f637c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.356+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f638c014070 con 0x7f6394103390 2024-03-24T09:21:54.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.356+0000 7f6393fff640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f637c0a20c0 0x7f637c0a4580 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6394069b40 tx=0x7f6384005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.360+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f638c0971d0 con 0x7f6394103390 2024-03-24T09:21:54.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2697951856' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:54.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/420490246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/332036792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3054953522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.568+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f639410d000 con 0x7f6394103390 2024-03-24T09:21:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.569+0000 7f63917fa640 1 -- 172.21.15.116:0/1563647924 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f638c09a890 con 0x7f6394103390 2024-03-24T09:21:54.569 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:54.569 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:20:06.925494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:21:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:21:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.570+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f637c0a20c0 msgr2=0x7f637c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.570+0000 7f639a69d640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f637c0a20c0 0x7f637c0a4580 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6394069b40 tx=0x7f6384005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.570+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 msgr2=0x7f639419ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.570+0000 7f639a69d640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 0x7f639419ecb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f638c00eb10 tx=0x7f638c00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 shutdown_connections 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6394103390 0x7f639419ecb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63941029d0 0x7f6394197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f637c0a20c0 0x7f637c0a4580 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 --2- 172.21.15.116:0/1563647924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f63940ffa00 0x7f6394197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 >> 172.21.15.116:0/1563647924 conn(0x7f63940fb330 msgr2=0x7f6394105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 shutdown_connections 2024-03-24T09:21:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.571+0000 7f639a69d640 1 -- 172.21.15.116:0/1563647924 wait complete. 2024-03-24T09:21:54.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:21:54.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 -- 172.21.15.116:0/2735356131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1141021b0 msgr2=0x7ff1141045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 --2- 172.21.15.116:0/2735356131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1141021b0 0x7ff1141045a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff104009860 tx=0x7ff10402bb70 comp rx=0 tx=0).stop 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 -- 172.21.15.116:0/2735356131 shutdown_connections 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 --2- 172.21.15.116:0/2735356131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff114107410 0x7ff114109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 --2- 172.21.15.116:0/2735356131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff114104ae0 0x7ff114106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 --2- 172.21.15.116:0/2735356131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1141021b0 0x7ff1141045a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 -- 172.21.15.116:0/2735356131 >> 172.21.15.116:0/2735356131 conn(0x7ff1140fbba0 msgr2=0x7ff1140fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.688+0000 7ff11ae09640 1 -- 172.21.15.116:0/2735356131 shutdown_connections 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 -- 172.21.15.116:0/2735356131 wait complete. 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 Processor -- start 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 -- start start 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 0x7ff1141976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff114104ae0 0x7ff114197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff114107410 0x7ff11419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff11419f1e0 con 0x7ff1141021b0 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff11419f350 con 0x7ff114107410 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff11ae09640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff11419f510 con 0x7ff114104ae0 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff118ba5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 0x7ff1141976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff118ba5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 0x7ff1141976f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59256/0 (socket says 172.21.15.116:59256) 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff118ba5640 1 -- 172.21.15.116:0/367657527 learned_addr learned my addr 172.21.15.116:0/367657527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff1193a6640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff114107410 0x7ff11419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.689+0000 7ff113fff640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff114104ae0 0x7ff114197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff118ba5640 1 -- 172.21.15.116:0/367657527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff114104ae0 msgr2=0x7ff114197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff118ba5640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff114104ae0 0x7ff114197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff118ba5640 1 -- 172.21.15.116:0/367657527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff114107410 msgr2=0x7ff11419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff118ba5640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff114107410 0x7ff11419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff118ba5640 1 -- 172.21.15.116:0/367657527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff108009560 con 0x7ff1141021b0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff118ba5640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 0x7ff1141976f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff104005940 tx=0x7ff104032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10402c510 con 0x7ff1141021b0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff104009490 con 0x7ff1141021b0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff10402ce90 con 0x7ff1141021b0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104003e20 con 0x7ff1141021b0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.690+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff11419fb00 con 0x7ff1141021b0 2024-03-24T09:21:54.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.691+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1140fd690 con 0x7ff1141021b0 2024-03-24T09:21:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.692+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7ff10403b040 con 0x7ff1141021b0 2024-03-24T09:21:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.692+0000 7ff111ffb640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff0fc0a20c0 0x7ff0fc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.693+0000 7ff113fff640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff0fc0a20c0 0x7ff0fc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.693+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7ff1040efba0 con 0x7ff1141021b0 2024-03-24T09:21:54.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.693+0000 7ff113fff640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff0fc0a20c0 0x7ff0fc0a4580 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff114198c00 tx=0x7ff108009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:54.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.697+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff1040b8850 con 0x7ff1141021b0 2024-03-24T09:21:54.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.874+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff11402d160 con 0x7ff1141021b0 2024-03-24T09:21:54.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.876+0000 7ff111ffb640 1 -- 172.21.15.116:0/367657527 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24911 (secure 0 0 0) 0x7ff1040bbf10 con 0x7ff1141021b0 2024-03-24T09:21:54.877 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:54.877 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":93,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:21:36.028172+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:21:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff0fc0a20c0 msgr2=0x7ff0fc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff0fc0a20c0 0x7ff0fc0a4580 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff114198c00 tx=0x7ff108009260 comp rx=0 tx=0).stop 2024-03-24T09:21:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 msgr2=0x7ff1141976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 0x7ff1141976f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff104005940 tx=0x7ff104032470 comp rx=0 tx=0).stop 2024-03-24T09:21:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 shutdown_connections 2024-03-24T09:21:54.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff114107410 0x7ff11419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7ff0fc0a20c0 0x7ff0fc0a4580 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff114104ae0 0x7ff114197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 --2- 172.21.15.116:0/367657527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1141021b0 0x7ff1141976f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 >> 172.21.15.116:0/367657527 conn(0x7ff1140fbba0 msgr2=0x7ff114102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 shutdown_connections 2024-03-24T09:21:54.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.878+0000 7ff11ae09640 1 -- 172.21.15.116:0/367657527 wait complete. 2024-03-24T09:21:54.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:21:54.891 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:21:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 -- 172.21.15.116:0/1407143751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc00101800 msgr2=0x7fdc00101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1407143751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc00101800 0x7fdc00101c80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009900 tx=0x7fdbe802bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 -- 172.21.15.116:0/1407143751 shutdown_connections 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1407143751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc001021c0 0x7fdc00108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1407143751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc00101800 0x7fdc00101c80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1407143751 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc00100610 0x7fdc00100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 -- 172.21.15.116:0/1407143751 >> 172.21.15.116:0/1407143751 conn(0x7fdc000fbb80 msgr2=0x7fdc000fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 -- 172.21.15.116:0/1407143751 shutdown_connections 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.996+0000 7fdc050c3640 1 -- 172.21.15.116:0/1407143751 wait complete. 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 Processor -- start 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 -- start start 2024-03-24T09:21:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc00100610 0x7fdc001965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc00101800 0x7fdc00196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 0x7fdc0019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0019e0e0 con 0x7fdc001021c0 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0019e250 con 0x7fdc00100610 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdc0019e410 con 0x7fdc00101800 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 0x7fdc0019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbfed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc00100610 0x7fdc001965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 0x7fdc0019dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59274/0 (socket says 172.21.15.116:59274) 2024-03-24T09:21:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbf7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc00101800 0x7fdc00196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 -- 172.21.15.116:0/1799316352 learned_addr learned my addr 172.21.15.116:0/1799316352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 -- 172.21.15.116:0/1799316352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc00101800 msgr2=0x7fdc00196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc00101800 0x7fdc00196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 -- 172.21.15.116:0/1799316352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc00100610 msgr2=0x7fdc001965f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc00100610 0x7fdc001965f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 -- 172.21.15.116:0/1799316352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe8009560 con 0x7fdc001021c0 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbfed9d640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc00100610 0x7fdc001965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbff59e640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 0x7fdc0019dba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800eb10 tx=0x7fdbf800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8004b20 con 0x7fdc001021c0 2024-03-24T09:21:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdbf8004c80 con 0x7fdc001021c0 2024-03-24T09:21:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.997+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc0019e6f0 con 0x7fdc001021c0 2024-03-24T09:21:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.998+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8010690 con 0x7fdc001021c0 2024-03-24T09:21:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.998+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc0019ebd0 con 0x7fdc001021c0 2024-03-24T09:21:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.998+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc00107a20 con 0x7fdc001021c0 2024-03-24T09:21:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:54.999+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7fdbf8005020 con 0x7fdc001021c0 2024-03-24T09:21:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.000+0000 7fdbfcd99640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdbec0a2140 0x7fdbec0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.000+0000 7fdbfed9d640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdbec0a2140 0x7fdbec0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.000+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fdbf8014070 con 0x7fdc001021c0 2024-03-24T09:21:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.001+0000 7fdbfed9d640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdbec0a2140 0x7fdbec0a4600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdc00101660 tx=0x7fdbf0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.004+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdbf80971e0 con 0x7fdc001021c0 2024-03-24T09:21:55.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.183+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdc0002d1d0 con 0x7fdc001021c0 2024-03-24T09:21:55.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.183+0000 7fdbfcd99640 1 -- 172.21.15.116:0/1799316352 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fdbf809a8a0 con 0x7fdc001021c0 2024-03-24T09:21:55.184 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:21:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdbec0a2140 msgr2=0x7fdbec0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdbec0a2140 0x7fdbec0a4600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdc00101660 tx=0x7fdbf0005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 msgr2=0x7fdc0019dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 0x7fdc0019dba0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800eb10 tx=0x7fdbf800c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 shutdown_connections 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc001021c0 0x7fdc0019dba0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fdbec0a2140 0x7fdbec0a4600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc00101800 0x7fdc00196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 --2- 172.21.15.116:0/1799316352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc00100610 0x7fdc001965f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 >> 172.21.15.116:0/1799316352 conn(0x7fdc000fbb80 msgr2=0x7fdc00104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 shutdown_connections 2024-03-24T09:21:55.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.186+0000 7fdc050c3640 1 -- 172.21.15.116:0/1799316352 wait complete. 2024-03-24T09:21:55.196 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:55 smithi116 ceph-mon[32207]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T09:21:55.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:55 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/115568276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:55.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:55 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1563647924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:55.197 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:55 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/367657527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:55.198 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 -- 172.21.15.116:0/3060988533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc80ffa00 msgr2=0x7f0bc80ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/3060988533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc80ffa00 0x7f0bc80ffe20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009900 tx=0x7f0bc402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 -- 172.21.15.116:0/3060988533 shutdown_connections 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/3060988533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc8103390 0x7f0bc8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/3060988533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bc81029d0 0x7f0bc8102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/3060988533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc80ffa00 0x7f0bc80ffe20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 -- 172.21.15.116:0/3060988533 >> 172.21.15.116:0/3060988533 conn(0x7f0bc80fb310 msgr2=0x7f0bc80fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 -- 172.21.15.116:0/3060988533 shutdown_connections 2024-03-24T09:21:55.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.301+0000 7f0bcf82e640 1 -- 172.21.15.116:0/3060988533 wait complete. 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 Processor -- start 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 -- start start 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 0x7f0bc8102340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bc81029d0 0x7f0bc8100980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 0x7f0bc8100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc8101620 con 0x7f0bc8103390 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc8101790 con 0x7f0bc81029d0 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcf82e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc81a1780 con 0x7f0bc80ffa00 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 0x7f0bc8100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcd5ca640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 0x7f0bc8102340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 0x7f0bc8100ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59294/0 (socket says 172.21.15.116:59294) 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcd5ca640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 0x7f0bc8102340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:43146/0 (socket says 172.21.15.116:43146) 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 -- 172.21.15.116:0/861374073 learned_addr learned my addr 172.21.15.116:0/861374073 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bccdc9640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bc81029d0 0x7f0bc8100980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 -- 172.21.15.116:0/861374073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 msgr2=0x7f0bc8102340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 0x7f0bc8102340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 -- 172.21.15.116:0/861374073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bc81029d0 msgr2=0x7f0bc8100980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bc81029d0 0x7f0bc8100980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.302+0000 7f0bcddcb640 1 -- 172.21.15.116:0/861374073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc4009560 con 0x7f0bc8103390 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bcd5ca640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 0x7f0bc8102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bcddcb640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 0x7f0bc8100ec0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000eb40 tx=0x7f0bc000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0004b20 con 0x7f0bc8103390 2024-03-24T09:21:55.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0bc0004c80 con 0x7f0bc8103390 2024-03-24T09:21:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bc81a1a30 con 0x7f0bc8103390 2024-03-24T09:21:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc0010690 con 0x7f0bc8103390 2024-03-24T09:21:55.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.303+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bc81a1ef0 con 0x7f0bc8103390 2024-03-24T09:21:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.304+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bc8108b20 con 0x7f0bc8103390 2024-03-24T09:21:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.305+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f0bc0005020 con 0x7f0bc8103390 2024-03-24T09:21:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.305+0000 7f0bbe7fc640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0bb40a20c0 0x7f0bb40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.305+0000 7f0bcd5ca640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0bb40a20c0 0x7f0bb40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.306+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f0bc0014070 con 0x7f0bc8103390 2024-03-24T09:21:55.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.306+0000 7f0bcd5ca640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0bb40a20c0 0x7f0bb40a4580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0bc402c7d0 tx=0x7f0bc4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.310+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0bc00971d0 con 0x7f0bc8103390 2024-03-24T09:21:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:55 smithi117 ceph-mon[35788]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T09:21:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:55 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/115568276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:55 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1563647924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:55 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/367657527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.483+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0bc802d1d0 con 0x7f0bc8103390 2024-03-24T09:21:55.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.484+0000 7f0bbe7fc640 1 -- 172.21.15.116:0/861374073 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f0bc009a890 con 0x7f0bc8103390 2024-03-24T09:21:55.484 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.485+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0bb40a20c0 msgr2=0x7f0bb40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0bb40a20c0 0x7f0bb40a4580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0bc402c7d0 tx=0x7f0bc4005fd0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 msgr2=0x7f0bc8100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 0x7f0bc8100ec0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0bc000eb40 tx=0x7f0bc000c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 shutdown_connections 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc8103390 0x7f0bc8100ec0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bc81029d0 0x7f0bc8100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f0bb40a20c0 0x7f0bb40a4580 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 --2- 172.21.15.116:0/861374073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0bc80ffa00 0x7f0bc8102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 >> 172.21.15.116:0/861374073 conn(0x7f0bc80fb310 msgr2=0x7f0bc8105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 shutdown_connections 2024-03-24T09:21:55.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.486+0000 7f0bcf82e640 1 -- 172.21.15.116:0/861374073 wait complete. 2024-03-24T09:21:55.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:21:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:55 smithi143 ceph-mon[35965]: pgmap v363: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T09:21:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:55 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/115568276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:55 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1563647924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:55 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/367657527' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 -- 172.21.15.116:0/1984855353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14107410 msgr2=0x7efd14109860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 --2- 172.21.15.116:0/1984855353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14107410 0x7efd14109860 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009ca0 tx=0x7efd0c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 -- 172.21.15.116:0/1984855353 shutdown_connections 2024-03-24T09:21:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 --2- 172.21.15.116:0/1984855353 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14107410 0x7efd14109860 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 --2- 172.21.15.116:0/1984855353 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd14104ae0 0x7efd14106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 --2- 172.21.15.116:0/1984855353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd141021b0 0x7efd141045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 -- 172.21.15.116:0/1984855353 >> 172.21.15.116:0/1984855353 conn(0x7efd140fbbe0 msgr2=0x7efd140fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 -- 172.21.15.116:0/1984855353 shutdown_connections 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.601+0000 7efd18bd0640 1 -- 172.21.15.116:0/1984855353 wait complete. 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 Processor -- start 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 -- start start 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 0x7efd14101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14104ae0 0x7efd14100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd14107410 0x7efd141006a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd14100e00 con 0x7efd141021b0 2024-03-24T09:21:55.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd14100f70 con 0x7efd14107410 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd18bd0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efd141a16b0 con 0x7efd14104ae0 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd1259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 0x7efd14101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.602+0000 7efd1259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 0x7efd14101b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59312/0 (socket says 172.21.15.116:59312) 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 -- 172.21.15.116:0/4204876470 learned_addr learned my addr 172.21.15.116:0/4204876470 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd11d9b640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14104ae0 0x7efd14100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd12d9d640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd14107410 0x7efd141006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 -- 172.21.15.116:0/4204876470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14104ae0 msgr2=0x7efd14100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14104ae0 0x7efd14100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 -- 172.21.15.116:0/4204876470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd14107410 msgr2=0x7efd141006a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd14107410 0x7efd141006a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 -- 172.21.15.116:0/4204876470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c009900 con 0x7efd141021b0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd11d9b640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14104ae0 0x7efd14100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd1259c640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 0x7efd14101b20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efd0400ea20 tx=0x7efd0400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04009720 con 0x7efd141021b0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd141a1960 con 0x7efd141021b0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efd0400cd80 con 0x7efd141021b0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04010430 con 0x7efd141021b0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.603+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd141a1e70 con 0x7efd141021b0 2024-03-24T09:21:55.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.604+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd140fd6d0 con 0x7efd141021b0 2024-03-24T09:21:55.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.605+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7efd04004ff0 con 0x7efd141021b0 2024-03-24T09:21:55.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.606+0000 7efd037fe640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efcfc0a20c0 0x7efcfc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.606+0000 7efd11d9b640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efcfc0a20c0 0x7efcfc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.606+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7efd04014070 con 0x7efd141021b0 2024-03-24T09:21:55.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.606+0000 7efd11d9b640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efcfc0a20c0 0x7efcfc0a4580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efd14101890 tx=0x7efd08005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.610+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7efd04096fe0 con 0x7efd141021b0 2024-03-24T09:21:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.783+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efd1410cff0 con 0x7efd141021b0 2024-03-24T09:21:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.784+0000 7efd037fe640 1 -- 172.21.15.116:0/4204876470 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7efd0409a6a0 con 0x7efd141021b0 2024-03-24T09:21:55.784 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:21:55.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efcfc0a20c0 msgr2=0x7efcfc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efcfc0a20c0 0x7efcfc0a4580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7efd14101890 tx=0x7efd08005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:55.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 msgr2=0x7efd14101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 0x7efd14101b20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efd0400ea20 tx=0x7efd0400ee20 comp rx=0 tx=0).stop 2024-03-24T09:21:55.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 shutdown_connections 2024-03-24T09:21:55.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd14107410 0x7efd141006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7efcfc0a20c0 0x7efcfc0a4580 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efd14104ae0 0x7efd14100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 --2- 172.21.15.116:0/4204876470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd141021b0 0x7efd14101b20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 >> 172.21.15.116:0/4204876470 conn(0x7efd140fbbe0 msgr2=0x7efd14102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 shutdown_connections 2024-03-24T09:21:55.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.786+0000 7efd18bd0640 1 -- 172.21.15.116:0/4204876470 wait complete. 2024-03-24T09:21:55.797 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1469059861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c100610 msgr2=0x7f2e9c100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1469059861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c100610 0x7f2e9c100a30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c009ca0 tx=0x7f2e8c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1469059861 shutdown_connections 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1469059861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e9c1021c0 0x7f2e9c108a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1469059861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1469059861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c100610 0x7f2e9c100a30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1469059861 >> 172.21.15.116:0/1469059861 conn(0x7f2e9c0fbba0 msgr2=0x7f2e9c0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1469059861 shutdown_connections 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.902+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1469059861 wait complete. 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 Processor -- start 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 -- start start 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e9c100610 0x7f2e9c195530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c195a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c1021c0 0x7f2e9c19cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c19d020 con 0x7f2e9c101800 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c19d190 con 0x7f2e9c1021c0 2024-03-24T09:21:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2ea13dd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9c19d350 con 0x7f2e9c100610 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2e9a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c195a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2e9a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c195a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59332/0 (socket says 172.21.15.116:59332) 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2e9a7fc640 1 -- 172.21.15.116:0/1705634225 learned_addr learned my addr 172.21.15.116:0/1705634225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2e9affd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e9c100610 0x7f2e9c195530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.903+0000 7f2e9b7fe640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c1021c0 0x7f2e9c19cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9a7fc640 1 -- 172.21.15.116:0/1705634225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e9c100610 msgr2=0x7f2e9c195530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9a7fc640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e9c100610 0x7f2e9c195530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9a7fc640 1 -- 172.21.15.116:0/1705634225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c1021c0 msgr2=0x7f2e9c19cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9a7fc640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c1021c0 0x7f2e9c19cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9a7fc640 1 -- 172.21.15.116:0/1705634225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e8c009900 con 0x7f2e9c101800 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9b7fe640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c1021c0 0x7f2e9c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e9a7fc640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c195a70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2e90010ad0 tx=0x7f2e90010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9000b910 con 0x7f2e9c101800 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9c19d630 con 0x7f2e9c101800 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2e90011040 con 0x7f2e9c101800 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e90015640 con 0x7f2e9c101800 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.904+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e9c19db40 con 0x7f2e9c101800 2024-03-24T09:21:55.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.905+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e9c107af0 con 0x7f2e9c101800 2024-03-24T09:21:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.906+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f2e9000ec30 con 0x7f2e9c101800 2024-03-24T09:21:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.906+0000 7f2e83fff640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2e840a20c0 0x7f2e840a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.907+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f2e900d06b0 con 0x7f2e9c101800 2024-03-24T09:21:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.907+0000 7f2e9affd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2e840a20c0 0x7f2e840a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.907+0000 7f2e9affd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2e840a20c0 0x7f2e840a4580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c02c700 tx=0x7f2e8c0325d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:55.911+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2e900992b0 con 0x7f2e9c101800 2024-03-24T09:21:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:56 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1799316352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:56 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/861374073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:56 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4204876470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:56 smithi116 ceph-mon[32207]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-24T09:21:56.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.086+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2e9c02d130 con 0x7f2e9c101800 2024-03-24T09:21:56.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.087+0000 7f2e83fff640 1 -- 172.21.15.116:0/1705634225 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f2e9009c970 con 0x7f2e9c101800 2024-03-24T09:21:56.087 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:21:56.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2e840a20c0 msgr2=0x7f2e840a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2e840a20c0 0x7f2e840a4580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2e8c02c700 tx=0x7f2e8c0325d0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 msgr2=0x7f2e9c195a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c195a70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2e90010ad0 tx=0x7f2e90010f90 comp rx=0 tx=0).stop 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 shutdown_connections 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2e9c1021c0 0x7f2e9c19cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e9c101800 0x7f2e9c195a70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f2e840a20c0 0x7f2e840a4580 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 --2- 172.21.15.116:0/1705634225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2e9c100610 0x7f2e9c195530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 >> 172.21.15.116:0/1705634225 conn(0x7f2e9c0fbba0 msgr2=0x7f2e9c104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 shutdown_connections 2024-03-24T09:21:56.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.089+0000 7f2ea13dd640 1 -- 172.21.15.116:0/1705634225 wait complete. 2024-03-24T09:21:56.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:21:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 -- 172.21.15.116:0/2145385586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c100650 msgr2=0x7f348c100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 --2- 172.21.15.116:0/2145385586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c100650 0x7f348c100a70 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f347c009900 tx=0x7f347c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 -- 172.21.15.116:0/2145385586 shutdown_connections 2024-03-24T09:21:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 --2- 172.21.15.116:0/2145385586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c102200 0x7f348c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 --2- 172.21.15.116:0/2145385586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f348c101840 0x7f348c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 --2- 172.21.15.116:0/2145385586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c100650 0x7f348c100a70 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 -- 172.21.15.116:0/2145385586 >> 172.21.15.116:0/2145385586 conn(0x7f348c0fbbc0 msgr2=0x7f348c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 -- 172.21.15.116:0/2145385586 shutdown_connections 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 -- 172.21.15.116:0/2145385586 wait complete. 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.202+0000 7f3491ed4640 1 Processor -- start 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 -- start start 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c100650 0x7f348c196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f348c101840 0x7f348c196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 0x7f348c19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f348c19e110 con 0x7f348c102200 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f348c19e280 con 0x7f348c101840 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3491ed4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f348c19e440 con 0x7f348c100650 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 0x7f348c19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 0x7f348c19dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59350/0 (socket says 172.21.15.116:59350) 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 -- 172.21.15.116:0/3597666901 learned_addr learned my addr 172.21.15.116:0/3597666901 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348b7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c100650 0x7f348c196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348affd640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f348c101840 0x7f348c196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 -- 172.21.15.116:0/3597666901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c100650 msgr2=0x7f348c196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c100650 0x7f348c196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 -- 172.21.15.116:0/3597666901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f348c101840 msgr2=0x7f348c196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f348c101840 0x7f348c196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 -- 172.21.15.116:0/3597666901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c009560 con 0x7f348c102200 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f348bfff640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 0x7f348c19dbd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f348400eb10 tx=0x7f348400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.203+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484004b20 con 0x7f348c102200 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.204+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3484004c80 con 0x7f348c102200 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.204+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f348c19e720 con 0x7f348c102200 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.204+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484010690 con 0x7f348c102200 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.204+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348c19ec30 con 0x7f348c102200 2024-03-24T09:21:56.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.205+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f348c107a10 con 0x7f348c102200 2024-03-24T09:21:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.206+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f3484005020 con 0x7f348c102200 2024-03-24T09:21:56.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.206+0000 7f3488ff9640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f34740a20c0 0x7f34740a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.206+0000 7f348b7fe640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f34740a20c0 0x7f34740a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.206+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f3484014070 con 0x7f348c102200 2024-03-24T09:21:56.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.207+0000 7f348b7fe640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f34740a20c0 0x7f34740a4580 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f347c02c7d0 tx=0x7f347c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.211+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f34840971d0 con 0x7f348c102200 2024-03-24T09:21:56.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:56 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1799316352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:56 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/861374073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:56 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4204876470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:56 smithi117 ceph-mon[35788]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-24T09:21:56.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.421+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f348c02d1d0 con 0x7f348c102200 2024-03-24T09:21:56.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.421+0000 7f3488ff9640 1 -- 172.21.15.116:0/3597666901 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11953 (secure 0 0 0) 0x7f348409a890 con 0x7f348c102200 2024-03-24T09:21:56.422 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:56.422 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:20:06.925494+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:21:56.423 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-03-24T09:21:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f34740a20c0 msgr2=0x7f34740a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f34740a20c0 0x7f34740a4580 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f347c02c7d0 tx=0x7f347c005fd0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 msgr2=0x7f348c19dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 0x7f348c19dbd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f348400eb10 tx=0x7f348400c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 shutdown_connections 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f348c102200 0x7f348c19dbd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f348c101840 0x7f348c196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f34740a20c0 0x7f34740a4580 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 --2- 172.21.15.116:0/3597666901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f348c100650 0x7f348c196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 >> 172.21.15.116:0/3597666901 conn(0x7f348c0fbbc0 msgr2=0x7f348c104be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.423+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 shutdown_connections 2024-03-24T09:21:56.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.424+0000 7f3491ed4640 1 -- 172.21.15.116:0/3597666901 wait complete. 2024-03-24T09:21:56.436 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:21:56.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:56 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1799316352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:56 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/861374073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:56 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4204876470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:21:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:56 smithi143 ceph-mon[35965]: pgmap v364: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/1375933410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88104af0 msgr2=0x7f8b88106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/1375933410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88104af0 0x7f8b88106ee0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009900 tx=0x7f8b7c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/1375933410 shutdown_connections 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/1375933410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b88107420 0x7f8b88109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/1375933410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88104af0 0x7f8b88106ee0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/1375933410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/1375933410 >> 172.21.15.116:0/1375933410 conn(0x7f8b880fbbf0 msgr2=0x7f8b880fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/1375933410 shutdown_connections 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.537+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/1375933410 wait complete. 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 Processor -- start 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 -- start start 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b88104af0 0x7f8b88197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88107420 0x7f8b8819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8819f1d0 con 0x7f8b881021c0 2024-03-24T09:21:56.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8819f340 con 0x7f8b88104af0 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b8dcc0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8819f500 con 0x7f8b88107420 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b86ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b88104af0 0x7f8b88197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39610/0 (socket says 172.21.15.116:39610) 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b877fe640 1 -- 172.21.15.116:0/628618932 learned_addr learned my addr 172.21.15.116:0/628618932 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.538+0000 7f8b87fff640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88107420 0x7f8b8819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b877fe640 1 -- 172.21.15.116:0/628618932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88107420 msgr2=0x7f8b8819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b877fe640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88107420 0x7f8b8819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b877fe640 1 -- 172.21.15.116:0/628618932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b88104af0 msgr2=0x7f8b88197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b877fe640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b88104af0 0x7f8b88197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b877fe640 1 -- 172.21.15.116:0/628618932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b78009490 con 0x7f8b881021c0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b877fe640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881976e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800ea60 tx=0x7f8b7800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7800cde0 con 0x7f8b881021c0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8b780044d0 con 0x7f8b881021c0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7c009560 con 0x7f8b881021c0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78010690 con 0x7f8b881021c0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.539+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8819fb30 con 0x7f8b881021c0 2024-03-24T09:21:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.540+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b880fd6e0 con 0x7f8b881021c0 2024-03-24T09:21:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.541+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f8b78003e40 con 0x7f8b881021c0 2024-03-24T09:21:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.542+0000 7f8b84ff9640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8b700a20c0 0x7f8b700a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.542+0000 7f8b86ffd640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8b700a20c0 0x7f8b700a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.542+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f8b78014070 con 0x7f8b881021c0 2024-03-24T09:21:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.542+0000 7f8b86ffd640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8b700a20c0 0x7f8b700a4580 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c02c7d0 tx=0x7f8b7c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.546+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f8b78097220 con 0x7f8b881021c0 2024-03-24T09:21:56.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.720+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b8810d1d0 con 0x7f8b881021c0 2024-03-24T09:21:56.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.721+0000 7f8b84ff9640 1 -- 172.21.15.116:0/628618932 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24911 (secure 0 0 0) 0x7f8b8810d1d0 con 0x7f8b881021c0 2024-03-24T09:21:56.722 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:21:56.723 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":93,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:21:36.028172+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:21:56.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8b700a20c0 msgr2=0x7f8b700a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8b700a20c0 0x7f8b700a4580 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c02c7d0 tx=0x7f8b7c005fd0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 msgr2=0x7f8b881976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881976e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800ea60 tx=0x7f8b7800ef20 comp rx=0 tx=0).stop 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 shutdown_connections 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f8b700a20c0 0x7f8b700a4580 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8b88107420 0x7f8b8819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b88104af0 0x7f8b88197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 --2- 172.21.15.116:0/628618932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b881021c0 0x7f8b881976e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 >> 172.21.15.116:0/628618932 conn(0x7f8b880fbbf0 msgr2=0x7f8b88102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.723+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 shutdown_connections 2024-03-24T09:21:56.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.724+0000 7f8b8dcc0640 1 -- 172.21.15.116:0/628618932 wait complete. 2024-03-24T09:21:56.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T09:19:38.431545+0000', 'modified': '2024-03-24T09:20:06.925494+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24515, 'mds_1': 24484, 'mds_2': 24451, 'mds_3': 24443, 'mds_4': 24422}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14631': {'gid': 14631, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1877964932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1877964932}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1877964932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14649': {'gid': 14649, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6841/891659164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6840', 'nonce': 891659164}, {'type': 'v1', 'addr': '172.21.15.116:6841', 'nonce': 891659164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24412': {'gid': 24412, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6833/2011428167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6832', 'nonce': 2011428167}, {'type': 'v1', 'addr': '172.21.15.143:6833', 'nonce': 2011428167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24422': {'gid': 24422, 'name': 'k', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.117:6833/3849440293', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6832', 'nonce': 3849440293}, {'type': 'v1', 'addr': '172.21.15.117:6833', 'nonce': 3849440293}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.117:6835/1932277434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1932277434}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1932277434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24451': {'gid': 24451, 'name': 'f', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.143:6835/1036797529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6834', 'nonce': 1036797529}, {'type': 'v1', 'addr': '172.21.15.143:6835', 'nonce': 1036797529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.143:6837/3300095096', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6836', 'nonce': 3300095096}, {'type': 'v1', 'addr': '172.21.15.143:6837', 'nonce': 3300095096}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6837/309438710', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 309438710}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 309438710}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.143:6839/3139487666', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.143:6838', 'nonce': 3139487666}, {'type': 'v1', 'addr': '172.21.15.143:6839', 'nonce': 3139487666}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6839/3950675281', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3950675281}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3950675281}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24515, 'qdb_cluster': [24484, 24515, 24451, 24443, 24422]} 2024-03-24T09:21:56.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T09:21:56.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.k status 2024-03-24T09:21:56.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.839+0000 7f5464ba2640 1 -- 172.21.15.116:0/240277391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601004f0 msgr2=0x7f5460100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.839+0000 7f5464ba2640 1 --2- 172.21.15.116:0/240277391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601004f0 0x7f5460100910 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5450009900 tx=0x7f545002bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 -- 172.21.15.116:0/240277391 shutdown_connections 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 --2- 172.21.15.116:0/240277391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54601020a0 0x7f5460108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 --2- 172.21.15.116:0/240277391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54601016e0 0x7f5460101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 --2- 172.21.15.116:0/240277391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601004f0 0x7f5460100910 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 -- 172.21.15.116:0/240277391 >> 172.21.15.116:0/240277391 conn(0x7f54600fba80 msgr2=0x7f54600fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 -- 172.21.15.116:0/240277391 shutdown_connections 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 -- 172.21.15.116:0/240277391 wait complete. 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 Processor -- start 2024-03-24T09:21:56.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 -- start start 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54601004f0 0x7f54600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54601016e0 0x7f546019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.840+0000 7f5464ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 0x7f546019e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f5464ba2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f546019f110 con 0x7f54601020a0 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f5464ba2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f546019f280 con 0x7f54601004f0 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f5464ba2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f546019f440 con 0x7f54601016e0 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 0x7f546019e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 0x7f546019e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39628/0 (socket says 172.21.15.116:39628) 2024-03-24T09:21:56.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 -- 172.21.15.116:0/1952288226 learned_addr learned my addr 172.21.15.116:0/1952288226 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545f7fe640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54601004f0 0x7f54600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f5456ffd640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54601016e0 0x7f546019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 -- 172.21.15.116:0/1952288226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54601016e0 msgr2=0x7f546019c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54601016e0 0x7f546019c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 -- 172.21.15.116:0/1952288226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54601004f0 msgr2=0x7f54600693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54601004f0 0x7f54600693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 -- 172.21.15.116:0/1952288226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5450009560 con 0x7f54601020a0 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545ffff640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 0x7f546019e930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f545800eb10 tx=0x7f545800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545d7fa640 1 -- 172.21.15.116:0/1952288226 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458004b20 con 0x7f54601020a0 2024-03-24T09:21:56.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545d7fa640 1 -- 172.21.15.116:0/1952288226 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5458004c80 con 0x7f54601020a0 2024-03-24T09:21:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546019f6c0 con 0x7f54601020a0 2024-03-24T09:21:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f545d7fa640 1 -- 172.21.15.116:0/1952288226 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5458010690 con 0x7f54601020a0 2024-03-24T09:21:56.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.841+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546019fbd0 con 0x7f54601020a0 2024-03-24T09:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f545d7fa640 1 -- 172.21.15.116:0/1952288226 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f5458005020 con 0x7f54601020a0 2024-03-24T09:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 Processor -- start 2024-03-24T09:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 -- start start 2024-03-24T09:21:56.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec23b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec28c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec2b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b9eb92180 con 0x559b9ec24150 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559b9eb8f6b0 con 0x559b9ec292c0 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f54689db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559b9eb8ae90 con 0x559b9ec23760 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f545f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec28c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f545f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec28c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39636/0 (socket says 172.21.15.116:39636) 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f545f7fe640 1 -- 172.21.15.116:0/222367874 learned_addr learned my addr 172.21.15.116:0/222367874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f5456ffd640 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec23b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f545d7fa640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f544c0a20c0 0x7f544c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.844+0000 7f5456ffd640 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec2b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545ffff640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f544c0a20c0 0x7f544c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545f7fe640 1 -- 172.21.15.116:0/222367874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 msgr2=0x559b9ec23b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545f7fe640 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec23b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545f7fe640 1 -- 172.21.15.116:0/222367874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 msgr2=0x559b9ec2b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545f7fe640 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec2b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545f7fe640 1 -- 172.21.15.116:0/222367874 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b9eb47d80 con 0x559b9ec24150 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545d7fa640 1 -- 172.21.15.116:0/1952288226 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5458014070 con 0x7f54601020a0 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545f7fe640 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec28c50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5450009900 tx=0x7f5450005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8be56042d0ba36f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f5455ffb640 1 -- 172.21.15.116:0/222367874 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545002ca30 con 0x559b9ec24150 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f545ffff640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f544c0a20c0 0x7f544c0a4580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f54580cc070 tx=0x7f545806dff0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f5455ffb640 1 -- 172.21.15.116:0/222367874 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f545002de90 con 0x559b9ec24150 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.845+0000 7f5455ffb640 1 -- 172.21.15.116:0/222367874 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54500037f0 con 0x559b9ec24150 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- 172.21.15.116:0/222367874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 msgr2=0x559b9ec28c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec28c50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5450009900 tx=0x7f5450005dd0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- 172.21.15.116:0/222367874 shutdown_connections 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec2b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec28c50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- 172.21.15.116:0/222367874 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec23b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- 172.21.15.116:0/222367874 >> 172.21.15.116:0/222367874 conn(0x559b9ec1ecd0 msgr2=0x559b9ec1f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- 172.21.15.116:0/222367874 shutdown_connections 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- 172.21.15.116:0/222367874 wait complete. 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- start start 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 Processor -- start 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec23760 0x559b9ec2d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec2dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec292c0 0x559b9ec34c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b9eb2fd50 con 0x559b9ec24150 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559b9e85e310 con 0x559b9ec23760 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.846+0000 7f54689db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559b9eb108f0 con 0x559b9ec292c0 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f54689db740 10 client.0 _ll_register_callbacks cb 0x559b9e89bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec2dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec23760 0x559b9ec2d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec2dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39642/0 (socket says 172.21.15.116:39642) 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 -- 172.21.15.116:0/2843245230 learned_addr learned my addr 172.21.15.116:0/2843245230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f5456ffd640 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec292c0 0x559b9ec34c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 -- 172.21.15.116:0/2843245230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec292c0 msgr2=0x559b9ec34c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec292c0 0x559b9ec34c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 -- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec23760 msgr2=0x559b9ec2d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec23760 0x559b9ec2d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 -- 172.21.15.116:0/2843245230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5460109230 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f545f7fe640 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec2dbf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5450036040 tx=0x7f545002cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b595776af9530d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f543effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f543effd640 1 -- 172.21.15.116:0/2843245230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54500051d0 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f543effd640 1 -- 172.21.15.116:0/2843245230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f545004d070 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b9e88a6c0 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f543effd640 1 -- 172.21.15.116:0/2843245230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5450005330 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f54689db740 10 client.15441 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b9eb89510 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.847+0000 7f543effd640 1 -- 172.21.15.116:0/2843245230 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559b9e88a6c0 con 0x559b9ec24150 2024-03-24T09:21:56.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f543effd640 1 -- 172.21.15.116:0/2843245230 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f545003c050 con 0x559b9ec24150 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f54689db740 10 client.15441 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f54689db740 10 client.15441 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f54689db740 20 client.15441 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f54689db740 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9ec38d90 0x559b9ec3af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f54689db740 4 client.15441 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b9e8db270 con 0x559b9ec38d90 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f5456ffd640 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9ec38d90 0x559b9ec3af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f5456ffd640 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9ec38d90 0x559b9ec3af50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=17284017ca6f2858 server_cookie=e563d8c4e6ccf3bb in_seq=0 out_seq=0 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.848+0000 7f543effd640 10 client.15441 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.849+0000 7f543effd640 1 -- 172.21.15.116:0/2843245230 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559b9e8db270 con 0x559b9ec38d90 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.849+0000 7f543effd640 10 client.15441 handle_command_reply: tid=0 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.849+0000 7f54689db740 1 client.15441 shutdown 2024-03-24T09:21:56.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.849+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 msgr2=0x559b9ec2dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.849+0000 7f54689db740 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec2dbf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5450036040 tx=0x7f545002cc60 comp rx=0 tx=0).stop 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 shutdown_connections 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec292c0 0x559b9ec34c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9ec38d90 0x559b9ec3af50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec24150 0x559b9ec2dbf0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 --2- 172.21.15.116:0/2843245230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec23760 0x559b9ec2d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f543effd640 0 client.15441 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 >> 172.21.15.116:0/2843245230 conn(0x559b9ec1ecd0 msgr2=0x559b9ec1ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 shutdown_connections 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 1 -- 172.21.15.116:0/2843245230 wait complete. 2024-03-24T09:21:56.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.850+0000 7f54689db740 20 client.15441 trim_cache size 0 max 16384 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.867+0000 7f54689db740 1 Processor -- start 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 -- start start 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec24150 0x559b9ec34bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 0x559b9ec249b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b9ec39d10 con 0x559b9ec292c0 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559b9ec1a520 con 0x559b9ec24150 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54689db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559b9e88a6c0 con 0x559b9ec23760 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 0x559b9ec249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 0x559b9ec249b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39656/0 (socket says 172.21.15.116:39656) 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 -- 172.21.15.116:0/3301945383 learned_addr learned my addr 172.21.15.116:0/3301945383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f545f7fe640 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec24150 0x559b9ec34bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 -- 172.21.15.116:0/3301945383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 msgr2=0x559b9ec352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec352c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 -- 172.21.15.116:0/3301945383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec24150 msgr2=0x559b9ec34bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec24150 0x559b9ec34bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 -- 172.21.15.116:0/3301945383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b9eb25a80 con 0x559b9ec292c0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f5456ffd640 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 0x559b9ec249b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5448009fd0 tx=0x7f54480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1549cfefecbe42dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54557fa640 1 -- 172.21.15.116:0/3301945383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448016040 con 0x559b9ec292c0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54557fa640 1 -- 172.21.15.116:0/3301945383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f544802d070 con 0x559b9ec292c0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.868+0000 7f54557fa640 1 -- 172.21.15.116:0/3301945383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448005480 con 0x559b9ec292c0 2024-03-24T09:21:56.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 -- 172.21.15.116:0/3301945383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 msgr2=0x559b9ec249b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 0x559b9ec249b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5448009fd0 tx=0x7f54480046c0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 -- 172.21.15.116:0/3301945383 shutdown_connections 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec292c0 0x559b9ec249b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec24150 0x559b9ec34bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 --2- 172.21.15.116:0/3301945383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec23760 0x559b9ec352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 -- 172.21.15.116:0/3301945383 >> 172.21.15.116:0/3301945383 conn(0x559b9ec1ecd0 msgr2=0x559b9ec227c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 -- 172.21.15.116:0/3301945383 shutdown_connections 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 -- 172.21.15.116:0/3301945383 wait complete. 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.869+0000 7f54689db740 1 -- start start 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 Processor -- start 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 0x559b9ec301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec24150 0x559b9ec30730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec33750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b9ec33c90 con 0x559b9ec23760 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559b9ec33e00 con 0x559b9ec292c0 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559b9ec33fc0 con 0x559b9ec24150 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 10 client.0 _ll_register_callbacks cb 0x7f5448009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 0x559b9ec301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 0x559b9ec301f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39670/0 (socket says 172.21.15.116:39670) 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 -- 172.21.15.116:0/4085442117 learned_addr learned my addr 172.21.15.116:0/4085442117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f545f7fe640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec24150 0x559b9ec30730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec33750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 -- 172.21.15.116:0/4085442117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec24150 msgr2=0x559b9ec30730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec24150 0x559b9ec30730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 -- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 msgr2=0x559b9ec33750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec33750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 -- 172.21.15.116:0/4085442117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5448019040 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f5456ffd640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 0x559b9ec301f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5448011990 tx=0x7f5448011bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f098a35fa7d4fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f543effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f543effd640 1 -- 172.21.15.116:0/4085442117 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448016070 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f543effd640 1 -- 172.21.15.116:0/4085442117 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f5448000e40 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f545002de90 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f543effd640 1 -- 172.21.15.116:0/4085442117 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5448011020 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 10 client.15447 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.870+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b9eb4b7e0 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f543effd640 1 -- 172.21.15.116:0/4085442117 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f544803d080 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f543effd640 1 -- 172.21.15.116:0/4085442117 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f54480049d0 con 0x559b9ec23760 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f54689db740 10 client.15447 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:56.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f54689db740 10 client.15447 resolve_mds: resolved name 'k' to daemon mds.k 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f54689db740 20 client.15447 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f54689db740 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9eca3e10 0x559b9eca2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f54689db740 4 client.15447 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "status"}) -- 0x559b9e8c7610 con 0x559b9eca3e10 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f545f7fe640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9eca3e10 0x559b9eca2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f543effd640 10 client.15447 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.871+0000 7f545f7fe640 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9eca3e10 0x559b9eca2b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1ac20c18884865ae server_cookie=2d04885455135767 in_seq=0 out_seq=0 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f543effd640 1 -- 172.21.15.116:0/4085442117 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559b9e8c7610 con 0x559b9eca3e10 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f543effd640 10 client.15447 handle_command_reply: tid=0 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 client.15447 shutdown 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 msgr2=0x559b9ec301f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 0x559b9ec301f0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5448011990 tx=0x7f5448011bd0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 shutdown_connections 2024-03-24T09:21:56.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b9ec292c0 0x559b9ec33750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559b9ec24150 0x559b9ec30730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x559b9eca3e10 0x559b9eca2b20 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 --2- 172.21.15.116:0/4085442117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b9ec23760 0x559b9ec301f0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f543effd640 0 client.15447 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 >> 172.21.15.116:0/4085442117 conn(0x559b9ec1ecd0 msgr2=0x559b9ec1f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 shutdown_connections 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 1 -- 172.21.15.116:0/4085442117 wait complete. 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f54689db740 20 client.15447 trim_cache size 0 max 16384 2024-03-24T09:21:56.880 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 4, 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24422, 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 110.96144593299999, 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 93, 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 172.85356707700001 2024-03-24T09:21:56.881 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f544c0a20c0 msgr2=0x7f544c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f544c0a20c0 0x7f544c0a4580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f54580cc070 tx=0x7f545806dff0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 msgr2=0x7f546019e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 0x7f546019e930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f545800eb10 tx=0x7f545800c2d0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 shutdown_connections 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54601020a0 0x7f546019e930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f544c0a20c0 0x7f544c0a4580 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f54601016e0 0x7f546019c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 --2- 172.21.15.116:0/1952288226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54601004f0 0x7f54600693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.872+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 >> 172.21.15.116:0/1952288226 conn(0x7f54600fba80 msgr2=0x7f5460104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.873+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 shutdown_connections 2024-03-24T09:21:56.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.873+0000 7f5464ba2640 1 -- 172.21.15.116:0/1952288226 wait complete. 2024-03-24T09:21:56.884 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.986+0000 7f4837e34640 1 -- 172.21.15.116:0/2429783578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 msgr2=0x7f4830100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.986+0000 7f4837e34640 1 --2- 172.21.15.116:0/2429783578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830100a50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f4828009ca0 tx=0x7f482802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- 172.21.15.116:0/2429783578 shutdown_connections 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 --2- 172.21.15.116:0/2429783578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48301021e0 0x7f4830108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 --2- 172.21.15.116:0/2429783578 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4830101820 0x7f4830101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 --2- 172.21.15.116:0/2429783578 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830100a50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- 172.21.15.116:0/2429783578 >> 172.21.15.116:0/2429783578 conn(0x7f48300fbbc0 msgr2=0x7f48300fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- 172.21.15.116:0/2429783578 shutdown_connections 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- 172.21.15.116:0/2429783578 wait complete. 2024-03-24T09:21:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 Processor -- start 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- start start 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4830101820 0x7f4830196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48301021e0 0x7f483019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f483019e160 con 0x7f4830100630 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f483019e2d0 con 0x7f4830101820 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4837e34640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f483019e490 con 0x7f48301021e0 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4835bd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4835bd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39686/0 (socket says 172.21.15.116:39686) 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f4835bd0640 1 -- 172.21.15.116:0/4084030006 learned_addr learned my addr 172.21.15.116:0/4084030006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f48363d1640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48301021e0 0x7f483019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.987+0000 7f48353cf640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4830101820 0x7f4830196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4835bd0640 1 -- 172.21.15.116:0/4084030006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48301021e0 msgr2=0x7f483019dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4835bd0640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48301021e0 0x7f483019dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4835bd0640 1 -- 172.21.15.116:0/4084030006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4830101820 msgr2=0x7f4830196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4835bd0640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4830101820 0x7f4830196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4835bd0640 1 -- 172.21.15.116:0/4084030006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4828009900 con 0x7f4830100630 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4835bd0640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830196670 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4828037040 tx=0x7f4828032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f482effd640 1 -- 172.21.15.116:0/4084030006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4828034c70 con 0x7f4830100630 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f482effd640 1 -- 172.21.15.116:0/4084030006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4828003e20 con 0x7f4830100630 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f482effd640 1 -- 172.21.15.116:0/4084030006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280044a0 con 0x7f4830100630 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4830069ce0 con 0x7f4830100630 2024-03-24T09:21:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.988+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f483006a1e0 con 0x7f4830100630 2024-03-24T09:21:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.990+0000 7f482effd640 1 -- 172.21.15.116:0/4084030006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f482803b070 con 0x7f4830100630 2024-03-24T09:21:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 Processor -- start 2024-03-24T09:21:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 -- start start 2024-03-24T09:21:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f482effd640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48180a2140 0x7f48180a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b06d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b07360 0x55f6c5b0be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5a71180 con 0x55f6c5b0c4d0 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5a6e6b0 con 0x55f6c5b06970 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f483aa0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5a69e90 con 0x55f6c5b07360 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f48363d1640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48180a2140 0x7f48180a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f482effd640 1 -- 172.21.15.116:0/4084030006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f48280f26d0 con 0x7f4830100630 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b06d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f48353cf640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b07360 0x55f6c5b0be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f48353cf640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b07360 0x55f6c5b0be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:39654/0 (socket says 172.21.15.116:39654) 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b06d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57452/0 (socket says 172.21.15.116:57452) 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f48353cf640 1 -- 172.21.15.116:0/789404323 learned_addr learned my addr 172.21.15.116:0/789404323 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:56.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f48363d1640 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 -- 172.21.15.116:0/789404323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b07360 msgr2=0x55f6c5b0be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b07360 0x55f6c5b0be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 -- 172.21.15.116:0/789404323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b0c4d0 msgr2=0x55f6c5b0e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.991+0000 7f4835bd0640 1 -- 172.21.15.116:0/789404323 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6c5a26d80 con 0x55f6c5b06970 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f48363d1640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48180a2140 0x7f48180a4600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4820005fd0 tx=0x7f482000ba90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f4835bd0640 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b06d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4828000f80 tx=0x7f4828032340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3403cea1fbd7597b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f4817fff640 1 -- 172.21.15.116:0/789404323 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280bf0e0 con 0x55f6c5b06970 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f4817fff640 1 -- 172.21.15.116:0/789404323 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48280c3180 con 0x55f6c5b06970 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f4817fff640 1 -- 172.21.15.116:0/789404323 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280bb750 con 0x55f6c5b06970 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f483aa0f740 1 -- 172.21.15.116:0/789404323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 msgr2=0x55f6c5b06d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.992+0000 7f483aa0f740 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b06d90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4828000f80 tx=0x7f4828032340 comp rx=0 tx=0).stop 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- 172.21.15.116:0/789404323 shutdown_connections 2024-03-24T09:21:57.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b07360 0x55f6c5b0be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 --2- 172.21.15.116:0/789404323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b06d90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- 172.21.15.116:0/789404323 >> 172.21.15.116:0/789404323 conn(0x55f6c5afdad0 msgr2=0x55f6c5afded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- 172.21.15.116:0/789404323 shutdown_connections 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- 172.21.15.116:0/789404323 wait complete. 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- start start 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 Processor -- start 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b10610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b10b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b17b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5a0ed50 con 0x55f6c5b07360 2024-03-24T09:21:57.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f6c573d310 con 0x55f6c5b06970 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f6c59ef8f0 con 0x55f6c5b0c4d0 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f48353cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b10b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.993+0000 7f483aa0f740 10 client.0 _ll_register_callbacks cb 0x55f6c577aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f4835bd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b10610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b10b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39714/0 (socket says 172.21.15.116:39714) 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 -- 172.21.15.116:0/2767967154 learned_addr learned my addr 172.21.15.116:0/2767967154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48363d1640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b17b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 -- 172.21.15.116:0/2767967154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 msgr2=0x55f6c5b17b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b17b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 -- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 msgr2=0x55f6c5b10610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b10610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 -- 172.21.15.116:0/2767967154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f483006a1e0 con 0x55f6c5b07360 2024-03-24T09:21:57.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b10610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48353cf640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b10b50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4824012700 tx=0x7f4824012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5964746a3e9d8894 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48157fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48157fa640 1 -- 172.21.15.116:0/2767967154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824017070 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f6c57696c0 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48157fa640 1 -- 172.21.15.116:0/2767967154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4824004050 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48157fa640 1 -- 172.21.15.116:0/2767967154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4824004210 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f483aa0f740 10 client.15459 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f6c5a68510 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.994+0000 7f48157fa640 1 -- 172.21.15.116:0/2767967154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f6c57696c0 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f48157fa640 1 -- 172.21.15.116:0/2767967154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4824003930 con 0x55f6c5b07360 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f483aa0f740 10 client.15459 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f483aa0f740 10 client.15459 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f483aa0f740 20 client.15459 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b1cf50 0x55f6c5b1e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f483aa0f740 4 client.15459 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6c57ba270 con 0x55f6c5b1cf50 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b1cf50 0x55f6c5b1e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f48157fa640 10 client.15459 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.995+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b1cf50 0x55f6c5b1e3c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5825db9e5a54f855 server_cookie=c696bfda0bc35f35 in_seq=0 out_seq=0 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f48157fa640 1 -- 172.21.15.116:0/2767967154 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6c57ba270 con 0x55f6c5b1cf50 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f48157fa640 10 client.15459 handle_command_reply: tid=0 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 client.15459 shutdown 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 msgr2=0x55f6c5b10b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b10b50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4824012700 tx=0x7f4824012bc0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 shutdown_connections 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b1cf50 0x55f6c5b1e3c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b17b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b10b50 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2767967154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b10610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f48157fa640 0 client.15459 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 >> 172.21.15.116:0/2767967154 conn(0x55f6c5afdad0 msgr2=0x55f6c5afcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 shutdown_connections 2024-03-24T09:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.996+0000 7f483aa0f740 1 -- 172.21.15.116:0/2767967154 wait complete. 2024-03-24T09:21:57.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:56.997+0000 7f483aa0f740 20 client.15459 trim_cache size 0 max 16384 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 Processor -- start 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 -- start start 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b06970 0x55f6c5b0f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5b0fac0 con 0x55f6c5b07360 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5b11600 con 0x55f6c5b0c4d0 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.014+0000 7f483aa0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5af8e80 con 0x55f6c5b06970 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39730/0 (socket says 172.21.15.116:39730) 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 -- 172.21.15.116:0/4153666952 learned_addr learned my addr 172.21.15.116:0/4153666952 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f4835bd0640 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b06970 0x55f6c5b0f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48363d1640 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 -- 172.21.15.116:0/4153666952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b06970 msgr2=0x55f6c5b0f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b06970 0x55f6c5b0f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 -- 172.21.15.116:0/4153666952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b0c4d0 msgr2=0x55f6c5b0bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 -- 172.21.15.116:0/4153666952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6c5a04a80 con 0x55f6c5b07360 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48353cf640 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17b50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4824012420 tx=0x7f48240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98a6958b5ef750 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48177fe640 1 -- 172.21.15.116:0/4153666952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824017040 con 0x55f6c5b07360 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48177fe640 1 -- 172.21.15.116:0/4153666952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4824003aa0 con 0x55f6c5b07360 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.015+0000 7f48177fe640 1 -- 172.21.15.116:0/4153666952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824005570 con 0x55f6c5b07360 2024-03-24T09:21:57.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- 172.21.15.116:0/4153666952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 msgr2=0x55f6c5b17b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17b50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4824012420 tx=0x7f48240057f0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- 172.21.15.116:0/4153666952 shutdown_connections 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b0bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17b50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- 172.21.15.116:0/4153666952 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b06970 0x55f6c5b0f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- 172.21.15.116:0/4153666952 >> 172.21.15.116:0/4153666952 conn(0x55f6c5afdad0 msgr2=0x55f6c5b1bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- 172.21.15.116:0/4153666952 shutdown_connections 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- 172.21.15.116:0/4153666952 wait complete. 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- start start 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 Processor -- start 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b0afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b80cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5b81390 con 0x55f6c5b07360 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5b81500 con 0x55f6c5b06970 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f6c5b816c0 con 0x55f6c5b0c4d0 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f483aa0f740 10 client.0 _ll_register_callbacks cb 0x7f4824009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.016+0000 7f48353cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39734/0 (socket says 172.21.15.116:39734) 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 -- 172.21.15.116:0/2770266048 learned_addr learned my addr 172.21.15.116:0/2770266048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b0afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48363d1640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b80cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 -- 172.21.15.116:0/2770266048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 msgr2=0x55f6c5b80cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b80cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 -- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 msgr2=0x55f6c5b0afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b0afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 -- 172.21.15.116:0/2770266048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6c5a68510 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b0afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48353cf640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17ab0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f48240038a0 tx=0x7f48240038d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9c0004971b02ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48157fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48157fa640 1 -- 172.21.15.116:0/2770266048 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4824017070 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48157fa640 1 -- 172.21.15.116:0/2770266048 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f4824003f30 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48280c3180 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48157fa640 1 -- 172.21.15.116:0/2770266048 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4824004110 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f483aa0f740 10 client.15465 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f6c5a2a7e0 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.017+0000 7f48157fa640 1 -- 172.21.15.116:0/2770266048 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f48280c3180 con 0x55f6c5b07360 2024-03-24T09:21:57.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f48157fa640 1 -- 172.21.15.116:0/2770266048 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4824030030 con 0x55f6c5b07360 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 10 client.15465 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 10 client.15465 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 20 client.15465 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b86b50 0x55f6c5b85860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 4 client.15465 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "status"}) -- 0x55f6c57a6610 con 0x55f6c5b86b50 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b86b50 0x55f6c5b85860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f48157fa640 10 client.15465 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f4835bd0640 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b86b50 0x55f6c5b85860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=65b356ebab535323 server_cookie=3d297f9f6a4e44a4 in_seq=0 out_seq=0 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f48157fa640 1 -- 172.21.15.116:0/2770266048 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f6c57a6610 con 0x55f6c5b86b50 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f48157fa640 10 client.15465 handle_command_reply: tid=0 2024-03-24T09:21:57.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 1 client.15465 shutdown 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 msgr2=0x55f6c5b17ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.018+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17ab0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f48240038a0 tx=0x7f48240038d0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 shutdown_connections 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55f6c5b86b50 0x55f6c5b85860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f6c5b0c4d0 0x55f6c5b80cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f48157fa640 0 client.15465 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f6c5b07360 0x55f6c5b17ab0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 --2- 172.21.15.116:0/2770266048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f6c5b06970 0x55f6c5b0afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 >> 172.21.15.116:0/2770266048 conn(0x55f6c5afdad0 msgr2=0x55f6c5afdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 shutdown_connections 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 1 -- 172.21.15.116:0/2770266048 wait complete. 2024-03-24T09:21:57.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f483aa0f740 20 client.15465 trim_cache size 0 max 16384 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 3, 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24443, 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 112.116322723, 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 93, 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 164.31209914799999 2024-03-24T09:21:57.028 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48180a2140 msgr2=0x7f48180a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48180a2140 0x7f48180a4600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4820005fd0 tx=0x7f482000ba90 comp rx=0 tx=0).stop 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 msgr2=0x7f4830196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830196670 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4828037040 tx=0x7f4828032980 comp rx=0 tx=0).stop 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 shutdown_connections 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f48180a2140 0x7f48180a4600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f48301021e0 0x7f483019dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4830101820 0x7f4830196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 --2- 172.21.15.116:0/4084030006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4830100630 0x7f4830196670 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 >> 172.21.15.116:0/4084030006 conn(0x7f48300fbbc0 msgr2=0x7f4830104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 shutdown_connections 2024-03-24T09:21:57.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.019+0000 7f4837e34640 1 -- 172.21.15.116:0/4084030006 wait complete. 2024-03-24T09:21:57.031 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-24T09:21:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.134+0000 7f709d842640 1 -- 172.21.15.116:0/3532026986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7098102190 msgr2=0x7f7098104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.134+0000 7f709d842640 1 --2- 172.21.15.116:0/3532026986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7098102190 0x7f7098104580 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f7088009ca0 tx=0x7f708802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.134+0000 7f709d842640 1 -- 172.21.15.116:0/3532026986 shutdown_connections 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.134+0000 7f709d842640 1 --2- 172.21.15.116:0/3532026986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70981073f0 0x7f7098109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.134+0000 7f709d842640 1 --2- 172.21.15.116:0/3532026986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7098104ac0 0x7f7098106eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 --2- 172.21.15.116:0/3532026986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7098102190 0x7f7098104580 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- 172.21.15.116:0/3532026986 >> 172.21.15.116:0/3532026986 conn(0x7f70980fbba0 msgr2=0x7f70980fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- 172.21.15.116:0/3532026986 shutdown_connections 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- 172.21.15.116:0/3532026986 wait complete. 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 Processor -- start 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- start start 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7098102190 0x7f7098197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7098104ac0 0x7f7098197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 0x7f709819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f709819f220 con 0x7f70981073f0 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f709819f390 con 0x7f7098102190 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.135+0000 7f709d842640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f709819f550 con 0x7f7098104ac0 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 0x7f709819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 0x7f709819ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39756/0 (socket says 172.21.15.116:39756) 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 -- 172.21.15.116:0/3058021622 learned_addr learned my addr 172.21.15.116:0/3058021622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f7096ffd640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7098102190 0x7f7098197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70967fc640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7098104ac0 0x7f7098197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 -- 172.21.15.116:0/3058021622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7098104ac0 msgr2=0x7f7098197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7098104ac0 0x7f7098197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 -- 172.21.15.116:0/3058021622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7098102190 msgr2=0x7f7098197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7098102190 0x7f7098197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 -- 172.21.15.116:0/3058021622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7088009900 con 0x7f70981073f0 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f70977fe640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 0x7f709819ece0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f709000e970 tx=0x7f709000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f709c840640 1 -- 172.21.15.116:0/3058021622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090009720 con 0x7f70981073f0 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f709c840640 1 -- 172.21.15.116:0/3058021622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70900054d0 con 0x7f70981073f0 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709819f830 con 0x7f70981073f0 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f709c840640 1 -- 172.21.15.116:0/3058021622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090010430 con 0x7f70981073f0 2024-03-24T09:21:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.136+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709819fcf0 con 0x7f70981073f0 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 Processor -- start 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f709c840640 1 -- 172.21.15.116:0/3058021622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f7090004f30 con 0x7f70981073f0 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 -- start start 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa896970 0x55b2fa896d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa897360 0x55b2fa89be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 0x55b2fa89e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa801180 con 0x55b2fa89c4d0 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa7fe6b0 con 0x55b2fa896970 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70a0419740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa7f9e90 con 0x55b2fa897360 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 0x55b2fa89e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 0x55b2fa89e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39770/0 (socket says 172.21.15.116:39770) 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f7096ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa897360 0x55b2fa89be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70967fc640 1 -- 172.21.15.116:0/3212694339 learned_addr learned my addr 172.21.15.116:0/3212694339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.139+0000 7f70967fc640 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa896970 0x55b2fa896d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f709c840640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f70800a20c0 0x7f70800a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70967fc640 1 -- 172.21.15.116:0/3212694339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa897360 msgr2=0x55b2fa89be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70967fc640 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa897360 0x55b2fa89be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70967fc640 1 -- 172.21.15.116:0/3212694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa896970 msgr2=0x55b2fa896d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70967fc640 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa896970 0x55b2fa896d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70967fc640 1 -- 172.21.15.116:0/3212694339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2fa7b6d80 con 0x55b2fa89c4d0 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70977fe640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f70800a20c0 0x7f70800a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f70967fc640 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 0x55b2fa89e960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f708c009fd0 tx=0x7f708c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b1e0614940819 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f709c840640 1 -- 172.21.15.116:0/3058021622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f7090014070 con 0x7f70981073f0 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f707dffb640 1 -- 172.21.15.116:0/3212694339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c004020 con 0x55b2fa89c4d0 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f707dffb640 1 -- 172.21.15.116:0/3212694339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f708c005020 con 0x55b2fa89c4d0 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.140+0000 7f707dffb640 1 -- 172.21.15.116:0/3212694339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c004ac0 con 0x55b2fa89c4d0 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- 172.21.15.116:0/3212694339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 msgr2=0x55b2fa89e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70977fe640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f70800a20c0 0x7f70800a4580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f70900971e0 tx=0x7f70900c5440 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 0x55b2fa89e960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f708c009fd0 tx=0x7f708c005850 comp rx=0 tx=0).stop 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- 172.21.15.116:0/3212694339 shutdown_connections 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa89c4d0 0x55b2fa89e960 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa897360 0x55b2fa89be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- 172.21.15.116:0/3212694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa896970 0x55b2fa896d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- 172.21.15.116:0/3212694339 >> 172.21.15.116:0/3212694339 conn(0x55b2fa88dad0 msgr2=0x55b2fa88ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- 172.21.15.116:0/3212694339 shutdown_connections 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- 172.21.15.116:0/3212694339 wait complete. 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- start start 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 Processor -- start 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 0x55b2fa8a0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa897360 0x55b2fa8a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa89c4d0 0x55b2fa8a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa79ed50 con 0x55b2fa896970 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa4cd310 con 0x55b2fa897360 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.141+0000 7f70a0419740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa77f8f0 con 0x55b2fa89c4d0 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70a0419740 10 client.0 _ll_register_callbacks cb 0x55b2fa50aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 0x55b2fa8a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f7096ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa897360 0x55b2fa8a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 0x55b2fa8a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39774/0 (socket says 172.21.15.116:39774) 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 -- 172.21.15.116:0/1862809653 learned_addr learned my addr 172.21.15.116:0/1862809653 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa89c4d0 0x55b2fa8a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 -- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa89c4d0 msgr2=0x55b2fa8a7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa89c4d0 0x55b2fa8a7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 -- 172.21.15.116:0/1862809653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa897360 msgr2=0x55b2fa8a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa897360 0x55b2fa8a0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 -- 172.21.15.116:0/1862809653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708c019040 con 0x55b2fa896970 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70967fc640 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 0x55b2fa8a0940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f708c005bc0 tx=0x7f708c004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd6c6585b9563dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f7076ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f7076ffd640 1 -- 172.21.15.116:0/1862809653 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0048a0 con 0x55b2fa896970 2024-03-24T09:21:57.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f7076ffd640 1 -- 172.21.15.116:0/1862809653 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f708c000e40 con 0x55b2fa896970 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b2fa4f96c0 con 0x55b2fa896970 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f7076ffd640 1 -- 172.21.15.116:0/1862809653 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f708c004a00 con 0x55b2fa896970 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70a0419740 10 client.15477 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2fa7f8510 con 0x55b2fa896970 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.142+0000 7f7076ffd640 1 -- 172.21.15.116:0/1862809653 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b2fa4f96c0 con 0x55b2fa896970 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f7076ffd640 1 -- 172.21.15.116:0/1862809653 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f708c0035f0 con 0x55b2fa896970 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f70a0419740 10 client.15477 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f70a0419740 10 client.15477 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f70a0419740 20 client.15477 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f70a0419740 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa8acfe0 0x55b2fa8ae450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f70a0419740 4 client.15477 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2fa54a270 con 0x55b2fa8acfe0 2024-03-24T09:21:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f7096ffd640 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa8acfe0 0x55b2fa8ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f7096ffd640 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa8acfe0 0x55b2fa8ae450 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3587476fb2a9b095 server_cookie=3cf35af325e3a6b5 in_seq=0 out_seq=0 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.143+0000 7f7076ffd640 10 client.15477 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f7076ffd640 1 -- 172.21.15.116:0/1862809653 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2fa54a270 con 0x55b2fa8acfe0 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f7076ffd640 10 client.15477 handle_command_reply: tid=0 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 client.15477 shutdown 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 msgr2=0x55b2fa8a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 0x55b2fa8a0940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f708c005bc0 tx=0x7f708c004700 comp rx=0 tx=0).stop 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 shutdown_connections 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa89c4d0 0x55b2fa8a7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa8acfe0 0x55b2fa8ae450 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa897360 0x55b2fa8a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f7076ffd640 0 client.15477 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:57.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 --2- 172.21.15.116:0/1862809653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa896970 0x55b2fa8a0940 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 >> 172.21.15.116:0/1862809653 conn(0x55b2fa88dad0 msgr2=0x55b2fa88cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 shutdown_connections 2024-03-24T09:21:57.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.144+0000 7f70a0419740 1 -- 172.21.15.116:0/1862809653 wait complete. 2024-03-24T09:21:57.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.145+0000 7f70a0419740 20 client.15477 trim_cache size 0 max 16384 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 Processor -- start 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 -- start start 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa89f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa89fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa8acb40 con 0x55b2fa897360 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa8a1550 con 0x55b2fa88dad0 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70a0419740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa888e80 con 0x55b2fa896970 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.162+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa89fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa89fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39786/0 (socket says 172.21.15.116:39786) 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 -- 172.21.15.116:0/3850635660 learned_addr learned my addr 172.21.15.116:0/3850635660 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f7096ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa89f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 -- 172.21.15.116:0/3850635660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 msgr2=0x55b2fa8a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 -- 172.21.15.116:0/3850635660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 msgr2=0x55b2fa89f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa89f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 -- 172.21.15.116:0/3850635660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2fa794a80 con 0x55b2fa897360 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f70967fc640 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa89fbe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f708c0059f0 tx=0x7f708c005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42c6937c6ae060ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f707d7fa640 1 -- 172.21.15.116:0/3850635660 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0097d0 con 0x55b2fa897360 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f707d7fa640 1 -- 172.21.15.116:0/3850635660 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f708c023070 con 0x55b2fa897360 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.163+0000 7f707d7fa640 1 -- 172.21.15.116:0/3850635660 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c00f9f0 con 0x55b2fa897360 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- 172.21.15.116:0/3850635660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 msgr2=0x55b2fa89fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa89fbe0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f708c0059f0 tx=0x7f708c005db0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- 172.21.15.116:0/3850635660 shutdown_connections 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa89fbe0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- 172.21.15.116:0/3850635660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa89f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- 172.21.15.116:0/3850635660 >> 172.21.15.116:0/3850635660 conn(0x55b2fa89bfd0 msgr2=0x55b2fa88deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- 172.21.15.116:0/3850635660 shutdown_connections 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- 172.21.15.116:0/3850635660 wait complete. 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- start start 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 Processor -- start 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa8a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa8a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa8a7180 con 0x55b2fa897360 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa8a72f0 con 0x55b2fa88dad0 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b2fa8a74b0 con 0x55b2fa896970 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70a0419740 10 client.0 _ll_register_callbacks cb 0x7f708c009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.164+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa8a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa8a6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39796/0 (socket says 172.21.15.116:39796) 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 -- 172.21.15.116:0/2568185554 learned_addr learned my addr 172.21.15.116:0/2568185554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f7096ffd640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa8a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 -- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 msgr2=0x55b2fa8a3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 -- 172.21.15.116:0/2568185554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 msgr2=0x55b2fa8a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa8a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 -- 172.21.15.116:0/2568185554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2fa7f8510 con 0x55b2fa897360 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70967fc640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa8a6c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f708c004150 tx=0x7f708c0045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10304afb648f295 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f7076ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f7076ffd640 1 -- 172.21.15.116:0/2568185554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c003820 con 0x55b2fa897360 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f7076ffd640 1 -- 172.21.15.116:0/2568185554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f708c000e40 con 0x55b2fa897360 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f708c005020 con 0x55b2fa897360 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f7076ffd640 1 -- 172.21.15.116:0/2568185554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f708c003e00 con 0x55b2fa897360 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70a0419740 10 client.15483 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b2fa7ba7e0 con 0x55b2fa897360 2024-03-24T09:21:57.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.165+0000 7f7076ffd640 1 -- 172.21.15.116:0/2568185554 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f708c005020 con 0x55b2fa897360 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f7076ffd640 1 -- 172.21.15.116:0/2568185554 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f708c045030 con 0x55b2fa897360 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 10 client.15483 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 10 client.15483 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 20 client.15483 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa89c4d0 0x55b2fa915d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 4 client.15483 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "status"}) -- 0x55b2fa536610 con 0x55b2fa89c4d0 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f7096ffd640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa89c4d0 0x55b2fa915d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f7076ffd640 10 client.15483 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f7096ffd640 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa89c4d0 0x55b2fa915d70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=832508245be11878 server_cookie=8e39d9ae371f53ba in_seq=0 out_seq=0 2024-03-24T09:21:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f7076ffd640 1 -- 172.21.15.116:0/2568185554 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b2fa536610 con 0x55b2fa89c4d0 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f7076ffd640 10 client.15483 handle_command_reply: tid=0 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 1 client.15483 shutdown 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 msgr2=0x55b2fa8a6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.166+0000 7f70a0419740 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa8a6c40 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f708c004150 tx=0x7f708c0045d0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 shutdown_connections 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b2fa897360 0x55b2fa8a6c40 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b2fa896970 0x55b2fa8a3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b2fa89c4d0 0x55b2fa915d70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 --2- 172.21.15.116:0/2568185554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b2fa88dad0 0x55b2fa8a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f7076ffd640 0 client.15483 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 >> 172.21.15.116:0/2568185554 conn(0x55b2fa89bfd0 msgr2=0x55b2fa89f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 shutdown_connections 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 1 -- 172.21.15.116:0/2568185554 wait complete. 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f70a0419740 20 client.15483 trim_cache size 0 max 16384 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24451, 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:57.176 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 113.273641548, 2024-03-24T09:21:57.177 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:57.177 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 93, 2024-03-24T09:21:57.177 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:57.177 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 161.62751658900001 2024-03-24T09:21:57.177 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f70800a20c0 msgr2=0x7f70800a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f70800a20c0 0x7f70800a4580 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f70900971e0 tx=0x7f70900c5440 comp rx=0 tx=0).stop 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 msgr2=0x7f709819ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 0x7f709819ece0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f709000e970 tx=0x7f709000ed70 comp rx=0 tx=0).stop 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 shutdown_connections 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70981073f0 0x7f709819ece0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f70800a20c0 0x7f70800a4580 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7098104ac0 0x7f7098197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 --2- 172.21.15.116:0/3058021622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7098102190 0x7f7098197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 >> 172.21.15.116:0/3058021622 conn(0x7f70980fbba0 msgr2=0x7f7098104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 shutdown_connections 2024-03-24T09:21:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.167+0000 7f709d842640 1 -- 172.21.15.116:0/3058021622 wait complete. 2024-03-24T09:21:57.180 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 -- 172.21.15.116:0/852337200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c1008d0 msgr2=0x7f9f9c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 --2- 172.21.15.116:0/852337200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c100d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009860 tx=0x7f9f9002bb70 comp rx=0 tx=0).stop 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 -- 172.21.15.116:0/852337200 shutdown_connections 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 --2- 172.21.15.116:0/852337200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c101290 0x7f9f9c108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 --2- 172.21.15.116:0/852337200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c100d50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 --2- 172.21.15.116:0/852337200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101c30 0x7f9f9c102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 -- 172.21.15.116:0/852337200 >> 172.21.15.116:0/852337200 conn(0x7f9f9c0fb340 msgr2=0x7f9f9c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 -- 172.21.15.116:0/852337200 shutdown_connections 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.282+0000 7f9fa2283640 1 -- 172.21.15.116:0/852337200 wait complete. 2024-03-24T09:21:57.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 Processor -- start 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 -- start start 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c1976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 0x7f9f9c197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c101c30 0x7f9f9c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c19f1c0 con 0x7f9f9c101290 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c19f330 con 0x7f9f9c101c30 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa2283640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c19f4f0 con 0x7f9f9c1008d0 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9f9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 0x7f9f9c197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9f9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 0x7f9f9c197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39826/0 (socket says 172.21.15.116:39826) 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/147659376 learned_addr learned my addr 172.21.15.116:0/147659376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c1976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.283+0000 7f9fa0820640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c101c30 0x7f9f9c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/147659376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c1008d0 msgr2=0x7f9f9c1976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c1976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/147659376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c101c30 msgr2=0x7f9f9c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c101c30 0x7f9f9c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/147659376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8c009560 con 0x7f9f9c101290 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c1976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 0x7f9f9c197c10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9f90005940 tx=0x7f9f90032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f997fa640 1 -- 172.21.15.116:0/147659376 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9002c510 con 0x7f9f9c101290 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f90009490 con 0x7f9f9c101290 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f997fa640 1 -- 172.21.15.116:0/147659376 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f9002ce90 con 0x7f9f9c101290 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9f997fa640 1 -- 172.21.15.116:0/147659376 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f90003e20 con 0x7f9f9c101290 2024-03-24T09:21:57.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.284+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9c19fa90 con 0x7f9f9c101290 2024-03-24T09:21:57.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.286+0000 7f9f997fa640 1 -- 172.21.15.116:0/147659376 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f9f9003b040 con 0x7f9f9c101290 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 Processor -- start 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 -- start start 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f997fa640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9f840a20c0 0x7f9f840a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283223980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283228a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f2832290c0 0x55f28322b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f283192180 con 0x55f283223560 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f28318f6b0 con 0x55f2832290c0 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f28318ae90 con 0x55f283223f50 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa0820640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9f840a20c0 0x7f9f840a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283223980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283223980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39834/0 (socket says 172.21.15.116:39834) 2024-03-24T09:21:57.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2129658018 learned_addr learned my addr 172.21.15.116:0/2129658018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f997fa640 1 -- 172.21.15.116:0/147659376 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9f900efba0 con 0x7f9f9c101290 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa0820640 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f2832290c0 0x55f28322b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283228a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2129658018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 msgr2=0x55f283228a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283228a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2129658018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f2832290c0 msgr2=0x55f28322b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f2832290c0 0x55f28322b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2129658018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f283147d80 con 0x55f283223560 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9fa0820640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9f840a20c0 0x7f9f840a4580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005fd0 tx=0x7f9f94009e00 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.287+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283223980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c011680 tx=0x7f9f8c011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65fec1f758e72743 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9f827fc640 1 -- 172.21.15.116:0/2129658018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c005020 con 0x55f283223560 2024-03-24T09:21:57.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9f827fc640 1 -- 172.21.15.116:0/2129658018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f8c005630 con 0x55f283223560 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9f827fc640 1 -- 172.21.15.116:0/2129658018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c00f8c0 con 0x55f283223560 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2129658018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 msgr2=0x55f283223980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283223980 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c011680 tx=0x7f9f8c011a80 comp rx=0 tx=0).stop 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2129658018 shutdown_connections 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f2832290c0 0x55f28322b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283228a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2129658018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283223980 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2129658018 >> 172.21.15.116:0/2129658018 conn(0x55f28321ead0 msgr2=0x55f28321eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2129658018 shutdown_connections 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.288+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2129658018 wait complete. 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 -- start start 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 Processor -- start 2024-03-24T09:21:57.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f28322d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223f50 0x55f28322da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2832290c0 0x55f283234ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f28312fd50 con 0x55f283223560 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f282e5e310 con 0x55f283223f50 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f2831108f0 con 0x55f2832290c0 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa4e5a740 10 client.0 _ll_register_callbacks cb 0x55f282e9bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f28322d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f28322d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39850/0 (socket says 172.21.15.116:39850) 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2791884768 learned_addr learned my addr 172.21.15.116:0/2791884768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223f50 0x55f28322da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9fa0820640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2832290c0 0x55f283234ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2832290c0 msgr2=0x55f283234ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2832290c0 0x55f283234ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2791884768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223f50 msgr2=0x55f28322da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223f50 0x55f28322da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.289+0000 7f9f9bfff640 1 -- 172.21.15.116:0/2791884768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8c009e30 con 0x55f283223560 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f28322d530 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0057c0 tx=0x7f9f8c005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6432a05adf1242 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/2791884768 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c00f6c0 con 0x55f283223560 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f282e8a6c0 con 0x55f283223560 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/2791884768 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9f8c00f820 con 0x55f283223560 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/2791884768 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9f8c00f9e0 con 0x55f283223560 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 10 client.15495 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f283189510 con 0x55f283223560 2024-03-24T09:21:57.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/2791884768 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f282e8a6c0 con 0x55f283223560 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/2791884768 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9f8c024bc0 con 0x55f283223560 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 10 client.15495 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 10 client.15495 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 20 client.15495 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f283239ce0 0x55f28323b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 4 client.15495 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.290+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f282edb270 con 0x55f283239ce0 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.291+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f283239ce0 0x55f28323b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.291+0000 7f9f7b7fe640 10 client.15495 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.291+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f283239ce0 0x55f28323b150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d85339c31ea0e9c0 server_cookie=dd33468f2df1475e in_seq=0 out_seq=0 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.291+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/2791884768 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f282edb270 con 0x55f283239ce0 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.291+0000 7f9f7b7fe640 10 client.15495 handle_command_reply: tid=0 2024-03-24T09:21:57.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.291+0000 7f9fa4e5a740 1 client.15495 shutdown 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 msgr2=0x55f28322d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f28322d530 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0057c0 tx=0x7f9f8c005230 comp rx=0 tx=0).stop 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 shutdown_connections 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f283239ce0 0x55f28323b150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f2832290c0 0x55f283234ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9f7b7fe640 0 client.15495 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223f50 0x55f28322da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/2791884768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f28322d530 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 >> 172.21.15.116:0/2791884768 conn(0x55f28321ead0 msgr2=0x55f28321dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 shutdown_connections 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/2791884768 wait complete. 2024-03-24T09:21:57.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.292+0000 7f9fa4e5a740 20 client.15495 trim_cache size 0 max 16384 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 Processor -- start 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 -- start start 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f28321ead0 0x55f283222440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223560 0x55f283234a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 0x55f28322cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f28321f520 con 0x55f283223f50 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f283218eb0 con 0x55f283223560 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f28323a0c0 con 0x55f28321ead0 2024-03-24T09:21:57.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 0x55f28322cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 0x55f28322cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39852/0 (socket says 172.21.15.116:39852) 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 -- 172.21.15.116:0/3411882791 learned_addr learned my addr 172.21.15.116:0/3411882791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f28321ead0 0x55f283222440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223560 0x55f283234a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 -- 172.21.15.116:0/3411882791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f28321ead0 msgr2=0x55f283222440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f28321ead0 0x55f283222440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 -- 172.21.15.116:0/3411882791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223560 msgr2=0x55f283234a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223560 0x55f283234a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 -- 172.21.15.116:0/3411882791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f283125a80 con 0x55f283223f50 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9fa0820640 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 0x55f28322cb50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9f940138a0 tx=0x7f9f94009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2459f7a48bbcb3f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9f81ffb640 1 -- 172.21.15.116:0/3411882791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9401fe70 con 0x55f283223f50 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9f81ffb640 1 -- 172.21.15.116:0/3411882791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9f94013170 con 0x55f283223f50 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.310+0000 7f9f81ffb640 1 -- 172.21.15.116:0/3411882791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f94004020 con 0x55f283223f50 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/3411882791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 msgr2=0x55f28322cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 0x55f28322cb50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9f940138a0 tx=0x7f9f94009500 comp rx=0 tx=0).stop 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/3411882791 shutdown_connections 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223f50 0x55f28322cb50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f283223560 0x55f283234a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/3411882791 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f28321ead0 0x55f283222440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/3411882791 >> 172.21.15.116:0/3411882791 conn(0x55f283228bc0 msgr2=0x55f283221d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/3411882791 shutdown_connections 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/3411882791 wait complete. 2024-03-24T09:21:57.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.311+0000 7f9fa4e5a740 1 -- start start 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 Processor -- start 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f28321ead0 0x55f283234350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283234890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283230df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f283231360 con 0x55f283223560 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f2832314d0 con 0x55f28321ead0 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f283231690 con 0x55f283223f50 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 10 client.0 _ll_register_callbacks cb 0x7f9f940137c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283234890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283234890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39860/0 (socket says 172.21.15.116:39860) 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/1064204985 learned_addr learned my addr 172.21.15.116:0/1064204985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa0820640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283230df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f28321ead0 0x55f283234350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 msgr2=0x55f283230df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283230df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/1064204985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f28321ead0 msgr2=0x55f283234350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f28321ead0 0x55f283234350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 -- 172.21.15.116:0/1064204985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9401b040 con 0x55f283223560 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa0820640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283230df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f9b7fe640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283234890 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9f900b8f80 tx=0x7f9f900dc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addf1b0b6d57735d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/1064204985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f900c0650 con 0x55f283223560 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/1064204985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f9f900c4ca0 con 0x55f283223560 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.312+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f283219c60 con 0x55f283223560 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/1064204985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9f900ba8b0 con 0x55f283223560 2024-03-24T09:21:57.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 10 client.15501 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f28314b7e0 con 0x55f283223560 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/1064204985 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f283219c60 con 0x55f283223560 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/1064204985 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9f900c4810 con 0x55f283223560 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 10 client.15501 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 10 client.15501 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 20 client.15501 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f2832290c0 0x55f2832a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 4 client.15501 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "status"}) -- 0x7f9f8c005630 con 0x55f2832290c0 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f2832290c0 0x55f2832a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9f7b7fe640 10 client.15501 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:57.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.313+0000 7f9f9bfff640 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f2832290c0 0x55f2832a3680 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=892432f3aa8bad1a server_cookie=3f2df8d2499e6874 in_seq=0 out_seq=0 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9f7b7fe640 1 -- 172.21.15.116:0/1064204985 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x7f9f8c005630 con 0x55f2832290c0 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9f7b7fe640 10 client.15501 handle_command_reply: tid=0 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 client.15501 shutdown 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 msgr2=0x55f283234890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283234890 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9f900b8f80 tx=0x7f9f900dc040 comp rx=0 tx=0).stop 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 shutdown_connections 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f2832290c0 0x55f2832a3680 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f283223f50 0x55f283230df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f283223560 0x55f283234890 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9f7b7fe640 0 client.15501 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 --2- 172.21.15.116:0/1064204985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f28321ead0 0x55f283234350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 >> 172.21.15.116:0/1064204985 conn(0x55f283228bc0 msgr2=0x55f2832225c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 shutdown_connections 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 1 -- 172.21.15.116:0/1064204985 wait complete. 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa4e5a740 20 client.15501 trim_cache size 0 max 16384 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24484, 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:57.323 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:57.324 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 114.29420269000001, 2024-03-24T09:21:57.324 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:57.324 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 93, 2024-03-24T09:21:57.324 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:57.324 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 152.95109158299999 2024-03-24T09:21:57.324 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9f840a20c0 msgr2=0x7f9f840a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa2283640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9f840a20c0 0x7f9f840a4580 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9f94005fd0 tx=0x7f9f94009e00 comp rx=0 tx=0).stop 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 msgr2=0x7f9f9c197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.314+0000 7f9fa2283640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 0x7f9f9c197c10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9f90005940 tx=0x7f9f90032470 comp rx=0 tx=0).stop 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 shutdown_connections 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c101c30 0x7f9f9c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f9c101290 0x7f9f9c197c10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f9f840a20c0 0x7f9f840a4580 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 --2- 172.21.15.116:0/147659376 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9f9c1008d0 0x7f9f9c1976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 >> 172.21.15.116:0/147659376 conn(0x7f9f9c0fb340 msgr2=0x7f9f9c0ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 shutdown_connections 2024-03-24T09:21:57.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.315+0000 7f9fa2283640 1 -- 172.21.15.116:0/147659376 wait complete. 2024-03-24T09:21:57.326 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:57 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1705634225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:57.326 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:57 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3597666901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:57.326 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:57 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/628618932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:57.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T09:21:57.327 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:57 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1705634225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:57.328 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:57 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3597666901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:57.328 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:57 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/628618932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:57.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 -- 172.21.15.116:0/226429982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f41021b0 msgr2=0x7f13f41045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 --2- 172.21.15.116:0/226429982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f41021b0 0x7f13f41045a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f13e4009900 tx=0x7f13e402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:57.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 -- 172.21.15.116:0/226429982 shutdown_connections 2024-03-24T09:21:57.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 --2- 172.21.15.116:0/226429982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4107410 0x7f13f4109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 --2- 172.21.15.116:0/226429982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13f4104ae0 0x7f13f4106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 --2- 172.21.15.116:0/226429982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f41021b0 0x7f13f41045a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 -- 172.21.15.116:0/226429982 >> 172.21.15.116:0/226429982 conn(0x7f13f40fbbc0 msgr2=0x7f13f40fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 -- 172.21.15.116:0/226429982 shutdown_connections 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.429+0000 7f13f9cae640 1 -- 172.21.15.116:0/226429982 wait complete. 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 Processor -- start 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 -- start start 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13f41021b0 0x7f13f4101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4104ae0 0x7f13f4100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 0x7f13f41006a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4100e00 con 0x7f13f4107410 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4100f70 con 0x7f13f41021b0 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f9cae640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41a1790 con 0x7f13f4104ae0 2024-03-24T09:21:57.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 0x7f13f41006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 0x7f13f41006a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39880/0 (socket says 172.21.15.116:39880) 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f3fff640 1 -- 172.21.15.116:0/412460811 learned_addr learned my addr 172.21.15.116:0/412460811 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4104ae0 0x7f13f4100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.430+0000 7f13f37fe640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13f41021b0 0x7f13f4101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f3fff640 1 -- 172.21.15.116:0/412460811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4104ae0 msgr2=0x7f13f4100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f3fff640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4104ae0 0x7f13f4100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f3fff640 1 -- 172.21.15.116:0/412460811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13f41021b0 msgr2=0x7f13f4101b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f3fff640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13f41021b0 0x7f13f4101b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f3fff640 1 -- 172.21.15.116:0/412460811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e8009490 con 0x7f13f4107410 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4104ae0 0x7f13f4100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f3fff640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 0x7f13f41006a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009cf0 tx=0x7f13ec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f0ff9640 1 -- 172.21.15.116:0/412460811 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec009b40 con 0x7f13f4107410 2024-03-24T09:21:57.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f0ff9640 1 -- 172.21.15.116:0/412460811 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13ec003b70 con 0x7f13f4107410 2024-03-24T09:21:57.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec009840 con 0x7f13f4107410 2024-03-24T09:21:57.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f0ff9640 1 -- 172.21.15.116:0/412460811 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0107e0 con 0x7f13f4107410 2024-03-24T09:21:57.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.431+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e4009560 con 0x7f13f4107410 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13f0ff9640 1 -- 172.21.15.116:0/412460811 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f13ec003ce0 con 0x7f13f4107410 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 Processor -- start 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 -- start start 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x56494361e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x564943623a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943626550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56494358d180 con 0x5649436240c0 2024-03-24T09:21:57.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56494358a6b0 con 0x56494361e560 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.433+0000 7f13fc96d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564943585e90 con 0x56494361ef50 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x56494361e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f0ff9640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13dc0a2140 0x7f13dc0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x56494361e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57622/0 (socket says 172.21.15.116:57622) 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 -- 172.21.15.116:0/2543760667 learned_addr learned my addr 172.21.15.116:0/2543760667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f37fe640 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x564943623a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943626550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f3fff640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13dc0a2140 0x7f13dc0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 -- 172.21.15.116:0/2543760667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 msgr2=0x564943623a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x564943623a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 -- 172.21.15.116:0/2543760667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 msgr2=0x564943626550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943626550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 -- 172.21.15.116:0/2543760667 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564943542d80 con 0x56494361e560 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f0ff9640 1 -- 172.21.15.116:0/412460811 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f13ec014070 con 0x7f13f4107410 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x56494361e980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f13e8011d20 tx=0x7f13e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbae25d8ddb7c96e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13f3fff640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13dc0a2140 0x7f13dc0a4600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f13ec003f90 tx=0x7f13ec06a170 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13d9ffb640 1 -- 172.21.15.116:0/2543760667 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e800fc50 con 0x56494361e560 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13d9ffb640 1 -- 172.21.15.116:0/2543760667 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13e80044e0 con 0x56494361e560 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.434+0000 7f13d9ffb640 1 -- 172.21.15.116:0/2543760667 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8003d30 con 0x56494361e560 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 -- 172.21.15.116:0/2543760667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 msgr2=0x56494361e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x56494361e980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f13e8011d20 tx=0x7f13e80046c0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 -- 172.21.15.116:0/2543760667 shutdown_connections 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943626550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x564943623a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 --2- 172.21.15.116:0/2543760667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x56494361e980 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 -- 172.21.15.116:0/2543760667 >> 172.21.15.116:0/2543760667 conn(0x564943619ad0 msgr2=0x564943619ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 -- 172.21.15.116:0/2543760667 shutdown_connections 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 -- 172.21.15.116:0/2543760667 wait complete. 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.435+0000 7f13fc96d740 1 -- start start 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 Processor -- start 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 0x5649436285c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x564943628b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649436240c0 0x56494362fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56494352ad50 con 0x56494361e560 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564943259310 con 0x56494361ef50 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56494350b8f0 con 0x5649436240c0 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13fc96d740 10 client.0 _ll_register_callbacks cb 0x564943296ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 0x5649436285c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 0x5649436285c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39906/0 (socket says 172.21.15.116:39906) 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 -- 172.21.15.116:0/137286660 learned_addr learned my addr 172.21.15.116:0/137286660 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f37fe640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x564943628b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649436240c0 0x56494362fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 -- 172.21.15.116:0/137286660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649436240c0 msgr2=0x56494362fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649436240c0 0x56494362fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 -- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 msgr2=0x564943628b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x564943628b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 -- 172.21.15.116:0/137286660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e8019040 con 0x56494361e560 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f37fe640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x564943628b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 0x5649436285c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f13e8010000 tx=0x7f13e8005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a491ebdf3de40175 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:21:57.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.436+0000 7f13d2ffd640 1 -- 172.21.15.116:0/137286660 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e80119e0 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13d2ffd640 1 -- 172.21.15.116:0/137286660 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f13e8005350 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649432856c0 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13d2ffd640 1 -- 172.21.15.116:0/137286660 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f13e8005510 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 10 client.15513 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564943584510 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13d2ffd640 1 -- 172.21.15.116:0/137286660 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5649432856c0 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13d2ffd640 1 -- 172.21.15.116:0/137286660 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f13e8003e80 con 0x56494361e560 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 10 client.15513 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 10 client.15513 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 20 client.15513 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564943634ca0 0x564943636110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 4 client.15513 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:21:57.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649432d6270 con 0x564943634ca0 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.437+0000 7f13f37fe640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564943634ca0 0x564943636110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13d2ffd640 10 client.15513 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13f37fe640 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564943634ca0 0x564943636110 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2e653d2937cbda3 server_cookie=ac60e51c69428dbf in_seq=0 out_seq=0 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13d2ffd640 1 -- 172.21.15.116:0/137286660 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649432d6270 con 0x564943634ca0 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13d2ffd640 10 client.15513 handle_command_reply: tid=0 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 client.15513 shutdown 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 msgr2=0x5649436285c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 0x5649436285c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f13e8010000 tx=0x7f13e8005850 comp rx=0 tx=0).stop 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 shutdown_connections 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649436240c0 0x56494362fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564943634ca0 0x564943636110 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x564943628b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 --2- 172.21.15.116:0/137286660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56494361e560 0x5649436285c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13d2ffd640 0 client.15513 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:57.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.438+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 >> 172.21.15.116:0/137286660 conn(0x564943619ad0 msgr2=0x564943618ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.439+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 shutdown_connections 2024-03-24T09:21:57.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.439+0000 7f13fc96d740 1 -- 172.21.15.116:0/137286660 wait complete. 2024-03-24T09:21:57.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.439+0000 7f13fc96d740 20 client.15513 trim_cache size 0 max 16384 2024-03-24T09:21:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 Processor -- start 2024-03-24T09:21:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 -- start start 2024-03-24T09:21:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x564943630160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x56494362fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943614810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564943619990 con 0x5649436240c0 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564943634800 con 0x56494361e560 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.456+0000 7f13fc96d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564943635080 con 0x56494361ef50 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943614810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943614810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39910/0 (socket says 172.21.15.116:39910) 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 -- 172.21.15.116:0/3116362425 learned_addr learned my addr 172.21.15.116:0/3116362425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f37fe640 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x56494362fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x564943630160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 -- 172.21.15.116:0/3116362425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 msgr2=0x56494362fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x56494362fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 -- 172.21.15.116:0/3116362425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 msgr2=0x564943630160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x564943630160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 -- 172.21.15.116:0/3116362425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564943520a80 con 0x5649436240c0 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f37fe640 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x56494362fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943614810 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f13e80043d0 tx=0x7f13e800ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c31daa2c979b72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13d97fa640 1 -- 172.21.15.116:0/3116362425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e801f4a0 con 0x5649436240c0 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13d97fa640 1 -- 172.21.15.116:0/3116362425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13e80119e0 con 0x5649436240c0 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.457+0000 7f13d97fa640 1 -- 172.21.15.116:0/3116362425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e801f9f0 con 0x5649436240c0 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- 172.21.15.116:0/3116362425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 msgr2=0x564943614810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943614810 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f13e80043d0 tx=0x7f13e800ff30 comp rx=0 tx=0).stop 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- 172.21.15.116:0/3116362425 shutdown_connections 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943614810 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361ef50 0x56494362fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- 172.21.15.116:0/3116362425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361e560 0x564943630160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- 172.21.15.116:0/3116362425 >> 172.21.15.116:0/3116362425 conn(0x564943619ad0 msgr2=0x564943619eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- 172.21.15.116:0/3116362425 shutdown_connections 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- 172.21.15.116:0/3116362425 wait complete. 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- start start 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 Processor -- start 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361e560 0x56494362aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x56494362af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943698a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56494362b6a0 con 0x5649436240c0 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564943698f70 con 0x56494361ef50 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564943699130 con 0x56494361e560 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13fc96d740 10 client.0 _ll_register_callbacks cb 0x7f13e8011810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:21:57.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943698a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13f2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943698a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39912/0 (socket says 172.21.15.116:39912) 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13f2ffd640 1 -- 172.21.15.116:0/4143318061 learned_addr learned my addr 172.21.15.116:0/4143318061 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13f37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x56494362af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.458+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361e560 0x56494362aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f37fe640 1 -- 172.21.15.116:0/4143318061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361e560 msgr2=0x56494362aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f37fe640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361e560 0x56494362aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f37fe640 1 -- 172.21.15.116:0/4143318061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 msgr2=0x564943698a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f37fe640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943698a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f37fe640 1 -- 172.21.15.116:0/4143318061 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f410a440 con 0x56494361ef50 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943698a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13f37fe640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x56494362af60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000c00 tx=0x7f13e4005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b4d4ce392c9f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13d2ffd640 1 -- 172.21.15.116:0/4143318061 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e404ab10 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13e80044e0 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13d2ffd640 1 -- 172.21.15.116:0/4143318061 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f13e4046040 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13d2ffd640 1 -- 172.21.15.116:0/4143318061 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f13e404ac70 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13fc96d740 10 client.25358 fetch_fsmap learned FSMap version 28 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564943584510 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.459+0000 7f13d2ffd640 1 -- 172.21.15.116:0/4143318061 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13e80044e0 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13d2ffd640 1 -- 172.21.15.116:0/4143318061 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f13e403c040 con 0x56494361ef50 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 10 client.25358 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 10 client.25358 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 20 client.25358 populate_metadata read hostname 'smithi116' 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56494369e980 0x56494369d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 4 client.25358 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "status"}) -- 0x5649432c2610 con 0x56494369e980 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56494369e980 0x56494369d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13f2ffd640 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56494369e980 0x56494369d690 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af4d9ffde551fa6e server_cookie=b48f95aeab1c5657 in_seq=0 out_seq=0 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13d2ffd640 10 client.25358 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13d2ffd640 1 -- 172.21.15.116:0/4143318061 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5649432c2610 con 0x56494369e980 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13d2ffd640 10 client.25358 handle_command_reply: tid=0 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 client.25358 shutdown 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 msgr2=0x56494362af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x56494362af60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f13e4000c00 tx=0x7f13e4005900 comp rx=0 tx=0).stop 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 shutdown_connections 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649436240c0 0x564943698a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56494369e980 0x56494369d690 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56494361ef50 0x56494362af60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 --2- 172.21.15.116:0/4143318061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56494361e560 0x56494362aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13d2ffd640 0 client.25358 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.460+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 >> 172.21.15.116:0/4143318061 conn(0x564943619ad0 msgr2=0x564943619eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 shutdown_connections 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13fc96d740 1 -- 172.21.15.116:0/4143318061 wait complete. 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13fc96d740 20 client.25358 trim_cache size 0 max 16384 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:21:57.470 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "69c76602-e9be-11ee-95cd-87774f69a715", 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24515, 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 138.362042, 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 28, 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 93, 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 89, 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 144.50316602300001 2024-03-24T09:21:57.471 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13dc0a2140 msgr2=0x7f13dc0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13dc0a2140 0x7f13dc0a4600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f13ec003f90 tx=0x7f13ec06a170 comp rx=0 tx=0).stop 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 msgr2=0x7f13f41006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 0x7f13f41006a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f13ec009cf0 tx=0x7f13ec00c6a0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 shutdown_connections 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f4107410 0x7f13f41006a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f13dc0a2140 0x7f13dc0a4600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13f4104ae0 0x7f13f4100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 --2- 172.21.15.116:0/412460811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13f41021b0 0x7f13f4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 >> 172.21.15.116:0/412460811 conn(0x7f13f40fbbc0 msgr2=0x7f13f4102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 shutdown_connections 2024-03-24T09:21:57.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.461+0000 7f13f9cae640 1 -- 172.21.15.116:0/412460811 wait complete. 2024-03-24T09:21:57.474 INFO:teuthology.run_tasks:Running task sequential... 2024-03-24T09:21:57.484 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-24T09:21:57.484 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-24T09:21:57.484 INFO:teuthology.task.print:Enabling mgr modules 2024-03-24T09:21:57.484 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-03-24T09:21:57.485 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T09:21:57.485 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:21:57.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-24T09:21:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:57 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1705634225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:21:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:57 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3597666901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:21:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:57 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/628618932' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 -- 172.21.15.116:0/490330454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0102200 msgr2=0x7f25e0108a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 --2- 172.21.15.116:0/490330454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0102200 0x7f25e0108a90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f25d8009ca0 tx=0x7f25d802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 -- 172.21.15.116:0/490330454 shutdown_connections 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 --2- 172.21.15.116:0/490330454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0102200 0x7f25e0108a90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 --2- 172.21.15.116:0/490330454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0101840 0x7f25e0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 --2- 172.21.15.116:0/490330454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e0100650 0x7f25e0100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 -- 172.21.15.116:0/490330454 >> 172.21.15.116:0/490330454 conn(0x7f25e00fbbc0 msgr2=0x7f25e00fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 -- 172.21.15.116:0/490330454 shutdown_connections 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.585+0000 7f25e8702640 1 -- 172.21.15.116:0/490330454 wait complete. 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 Processor -- start 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 -- start start 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e0100650 0x7f25e01975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 0x7f25e0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0102200 0x7f25e019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25e019f0c0 con 0x7f25e0101840 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f25e019f230 con 0x7f25e0102200 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e8702640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f25e019f3f0 con 0x7f25e0100650 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 0x7f25e0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 0x7f25e0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39930/0 (socket says 172.21.15.116:39930) 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 -- 172.21.15.116:0/1881751928 learned_addr learned my addr 172.21.15.116:0/1881751928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e6c9f640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0102200 0x7f25e019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e649e640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e0100650 0x7f25e01975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 -- 172.21.15.116:0/1881751928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e0100650 msgr2=0x7f25e01975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e0100650 0x7f25e01975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 -- 172.21.15.116:0/1881751928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0102200 msgr2=0x7f25e019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0102200 0x7f25e019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 -- 172.21.15.116:0/1881751928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25dc009490 con 0x7f25e0101840 2024-03-24T09:21:57.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e6c9f640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0102200 0x7f25e019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25e5c9d640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 0x7f25e0197b10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f25d000ea60 tx=0x7f25d000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.586+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d000cde0 con 0x7f25e0101840 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.587+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d00044d0 con 0x7f25e0101840 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.587+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d0009490 con 0x7f25e0101840 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.587+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d0010690 con 0x7f25e0101840 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.587+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d8009900 con 0x7f25e0101840 2024-03-24T09:21:57.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.587+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e010bf10 con 0x7f25e0101840 2024-03-24T09:21:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.589+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 136757+0+0 (secure 0 0 0) 0x7f25d0003e40 con 0x7f25e0101840 2024-03-24T09:21:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.590+0000 7f25d77fe640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25cc0a2160 0x7f25cc0a4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.590+0000 7f25e649e640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25cc0a2160 0x7f25cc0a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.590+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f25d0014070 con 0x7f25e0101840 2024-03-24T09:21:57.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.590+0000 7f25e649e640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25cc0a2160 0x7f25cc0a4620 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f25e01016a0 tx=0x7f25dc005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:57.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.593+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f25d0097220 con 0x7f25e0101840 2024-03-24T09:21:57.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:57.814+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f25e019fde0 con 0x7f25e0101840 2024-03-24T09:21:58.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.048+0000 7f25d77fe640 1 -- 172.21.15.116:0/1881751928 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f25d009a8e0 con 0x7f25e0101840 2024-03-24T09:21:58.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25cc0a2160 msgr2=0x7f25cc0a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25cc0a2160 0x7f25cc0a4620 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f25e01016a0 tx=0x7f25dc005f90 comp rx=0 tx=0).stop 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 msgr2=0x7f25e0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 0x7f25e0197b10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f25d000ea60 tx=0x7f25d000ef20 comp rx=0 tx=0).stop 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 shutdown_connections 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25e0102200 0x7f25e019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25e0101840 0x7f25e0197b10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25cc0a2160 0x7f25cc0a4620 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 --2- 172.21.15.116:0/1881751928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25e0100650 0x7f25e01975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 >> 172.21.15.116:0/1881751928 conn(0x7f25e00fbbc0 msgr2=0x7f25e00fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 shutdown_connections 2024-03-24T09:21:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.050+0000 7f25e8702640 1 -- 172.21.15.116:0/1881751928 wait complete. 2024-03-24T09:21:58.067 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-24T09:21:58.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 -- 172.21.15.116:0/4274776443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b01029c0 msgr2=0x7fe8b0102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 --2- 172.21.15.116:0/4274776443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b01029c0 0x7fe8b0102e40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009900 tx=0x7fe8a402bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 -- 172.21.15.116:0/4274776443 shutdown_connections 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 --2- 172.21.15.116:0/4274776443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8b0103380 0x7fe8b0109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 --2- 172.21.15.116:0/4274776443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b01029c0 0x7fe8b0102e40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 --2- 172.21.15.116:0/4274776443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b00ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 -- 172.21.15.116:0/4274776443 >> 172.21.15.116:0/4274776443 conn(0x7fe8b00fb360 msgr2=0x7fe8b00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 -- 172.21.15.116:0/4274776443 shutdown_connections 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.174+0000 7fe8b584e640 1 -- 172.21.15.116:0/4274776443 wait complete. 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 Processor -- start 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 -- start start 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b01976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8b01029c0 0x7fe8b0197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b0103380 0x7fe8b019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b019f150 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b019f2c0 con 0x7fe8b01029c0 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8b584e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b019f480 con 0x7fe8b0103380 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b01976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8aeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b01976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39962/0 (socket says 172.21.15.116:39962) 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8aeffd640 1 -- 172.21.15.116:0/409467710 learned_addr learned my addr 172.21.15.116:0/409467710 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8ae7fc640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8b01029c0 0x7fe8b0197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.175+0000 7fe8af7fe640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b0103380 0x7fe8b019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8aeffd640 1 -- 172.21.15.116:0/409467710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b0103380 msgr2=0x7fe8b019ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8aeffd640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b0103380 0x7fe8b019ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8aeffd640 1 -- 172.21.15.116:0/409467710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8b01029c0 msgr2=0x7fe8b0197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8aeffd640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8b01029c0 0x7fe8b0197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8aeffd640 1 -- 172.21.15.116:0/409467710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a4009560 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8aeffd640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b01976b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000e9c0 tx=0x7fe8a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:58.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a00096e0 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b019f760 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8a00044e0 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0010430 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.176+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b019fc40 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.177+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b0108be0 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.178+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 136774+0+0 (secure 0 0 0) 0x7fe8a0003f80 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.178+0000 7fe8b484c640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 0x7fe8980a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.179+0000 7fe8ae7fc640 1 -- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 msgr2=0x7fe8980a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:21:58.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.179+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7fe8a0014070 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.179+0000 7fe8ae7fc640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 0x7fe8980a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:21:58.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.187+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe8a0097050 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:58 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1881751928' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T09:21:58.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:58 smithi117 ceph-mon[35788]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T09:21:58.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:21:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: ignoring --setuser ceph since I am not root 2024-03-24T09:21:58.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:21:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: ignoring --setgroup ceph since I am not root 2024-03-24T09:21:58.317 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:58 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: ignoring --setuser ceph since I am not root 2024-03-24T09:21:58.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:58 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: ignoring --setgroup ceph since I am not root 2024-03-24T09:21:58.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:58 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1881751928' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T09:21:58.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:58 smithi116 ceph-mon[32207]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T09:21:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.380+0000 7fe8ae7fc640 1 -- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 msgr2=0x7fe8980a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:21:58.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.380+0000 7fe8ae7fc640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 0x7fe8980a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:21:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.397+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fe8b010cff0 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.400+0000 7fe8b484c640 1 -- 172.21.15.116:0/409467710 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v32)=0 v32) ==== 138+0+0 (secure 0 0 0) 0x7fe8a009a710 con 0x7fe8b00ff9f0 2024-03-24T09:21:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 msgr2=0x7fe8980a4670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:21:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 0x7fe8980a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 msgr2=0x7fe8b01976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b01976b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe8a000e9c0 tx=0x7fe8a000ee80 comp rx=0 tx=0).stop 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 shutdown_connections 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7fe8980a21b0 0x7fe8980a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8b0103380 0x7fe8b019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8b01029c0 0x7fe8b0197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 --2- 172.21.15.116:0/409467710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8b00ff9f0 0x7fe8b01976b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 >> 172.21.15.116:0/409467710 conn(0x7fe8b00fb360 msgr2=0x7fe8b0105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 shutdown_connections 2024-03-24T09:21:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.403+0000 7fe8b584e640 1 -- 172.21.15.116:0/409467710 wait complete. 2024-03-24T09:21:58.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 -- 172.21.15.116:0/1362697829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc102270 msgr2=0x7f25dc104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 --2- 172.21.15.116:0/1362697829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc102270 0x7f25dc104660 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f25cc009900 tx=0x7f25cc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 -- 172.21.15.116:0/1362697829 shutdown_connections 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 --2- 172.21.15.116:0/1362697829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25dc1074d0 0x7f25dc1098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 --2- 172.21.15.116:0/1362697829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc104ba0 0x7f25dc106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 --2- 172.21.15.116:0/1362697829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc102270 0x7f25dc104660 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 -- 172.21.15.116:0/1362697829 >> 172.21.15.116:0/1362697829 conn(0x7f25dc0fbba0 msgr2=0x7f25dc0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 -- 172.21.15.116:0/1362697829 shutdown_connections 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.521+0000 7f25e17e2640 1 -- 172.21.15.116:0/1362697829 wait complete. 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 Processor -- start 2024-03-24T09:21:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 -- start start 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25dc102270 0x7f25dc101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 0x7f25dc100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc1074d0 0x7f25dc100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc100ec0 con 0x7f25dc104ba0 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc101030 con 0x7f25dc1074d0 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25e17e2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f25dc1a1720 con 0x7f25dc102270 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25da7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 0x7f25dc100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25db7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc1074d0 0x7f25dc100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25da7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 0x7f25dc100220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39990/0 (socket says 172.21.15.116:39990) 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25daffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25dc102270 0x7f25dc101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25da7fc640 1 -- 172.21.15.116:0/3489245711 learned_addr learned my addr 172.21.15.116:0/3489245711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25db7fe640 1 -- 172.21.15.116:0/3489245711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25dc102270 msgr2=0x7f25dc101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25db7fe640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25dc102270 0x7f25dc101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25db7fe640 1 -- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 msgr2=0x7f25dc100220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25db7fe640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 0x7f25dc100220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.522+0000 7f25db7fe640 1 -- 172.21.15.116:0/3489245711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25cc009560 con 0x7f25dc1074d0 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25da7fc640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 0x7f25dc100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25db7fe640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc1074d0 0x7f25dc100760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f25d400ea80 tx=0x7f25d400ef40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d400cea0 con 0x7f25dc1074d0 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25dc1a19d0 con 0x7f25dc1074d0 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25d4005020 con 0x7f25dc1074d0 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25dc1a1f30 con 0x7f25dc1074d0 2024-03-24T09:21:58.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.523+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4010690 con 0x7f25dc1074d0 2024-03-24T09:21:58.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1881751928' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-24T09:21:58.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-mon[35965]: pgmap v365: 129 pgs: 129 active+clean; 646 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-03-24T09:21:58.527 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: ignoring --setuser ceph since I am not root 2024-03-24T09:21:58.527 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: ignoring --setgroup ceph since I am not root 2024-03-24T09:21:58.527 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-mgr[37858]: -- 172.21.15.143:0/3875954974 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563135bda1a0 con 0x563135bb0800 2024-03-24T09:21:58.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.524+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25dc0fd690 con 0x7f25dc1074d0 2024-03-24T09:21:58.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.525+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 19) ==== 136774+0+0 (secure 0 0 0) 0x7f25d4005190 con 0x7f25dc1074d0 2024-03-24T09:21:58.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.526+0000 7f25c3fff640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25c40a21b0 0x7f25c40a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:21:58.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.526+0000 7f25daffd640 1 -- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25c40a21b0 msgr2=0x7f25c40a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:21:58.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.526+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f25d4014070 con 0x7f25dc1074d0 2024-03-24T09:21:58.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.526+0000 7f25daffd640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25c40a21b0 0x7f25c40a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:21:58.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.533+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f25d4097300 con 0x7f25dc1074d0 2024-03-24T09:21:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.716+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f25dc02d1d0 con 0x7f25dc1074d0 2024-03-24T09:21:58.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.721+0000 7f25c3fff640 1 -- 172.21.15.116:0/3489245711 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v33)=0 v33) ==== 133+0+0 (secure 0 0 0) 0x7f25d409a9c0 con 0x7f25dc1074d0 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25c40a21b0 msgr2=0x7f25c40a4670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25c40a21b0 0x7f25c40a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc1074d0 msgr2=0x7f25dc100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc1074d0 0x7f25dc100760 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f25d400ea80 tx=0x7f25d400ef40 comp rx=0 tx=0).stop 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 shutdown_connections 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc1074d0 0x7f25dc100760 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25dc104ba0 0x7f25dc100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f25c40a21b0 0x7f25c40a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 --2- 172.21.15.116:0/3489245711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f25dc102270 0x7f25dc101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:21:58.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.723+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 >> 172.21.15.116:0/3489245711 conn(0x7f25dc0fbba0 msgr2=0x7f25dc102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:21:58.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.724+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 shutdown_connections 2024-03-24T09:21:58.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:21:58.724+0000 7f25e17e2640 1 -- 172.21.15.116:0/3489245711 wait complete. 2024-03-24T09:21:58.735 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-24T09:21:58.746 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-24T09:21:58.747 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-03-24T09:21:58.747 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-03-24T09:21:58.747 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T09:21:58.747 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T09:21:58.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:21:58.747 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-24T09:21:58.762 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-24T09:21:58.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:21:58.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-24T09:21:58.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:21:58.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-24T09:21:58.929 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:58 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:58.666+0000 7fc30e88a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:21:58.929 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:58 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:58.770+0000 7fc30e88a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:21:58.939 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-24T09:21:58.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T09:21:59.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:58.680+0000 7fab530bc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:21:59.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:58.778+0000 7fab530bc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:21:59.050 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:21:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:21:58.769+0000 7f238cc9e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T09:21:59.050 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:21:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:21:58.863+0000 7f238cc9e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T09:21:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:21:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:59 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1881751928' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T09:21:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:59 smithi117 ceph-mon[35788]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-24T09:21:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:59 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/409467710' entity='client.admin' 2024-03-24T09:21:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:21:59 smithi117 ceph-mon[35788]: from='client.? ' entity='client.admin' 2024-03-24T09:21:59.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1881751928' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T09:21:59.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-mon[32207]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-24T09:21:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/409467710' entity='client.admin' 2024-03-24T09:21:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-mon[32207]: from='client.? ' entity='client.admin' 2024-03-24T09:21:59.482 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1881751928' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-24T09:21:59.482 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-mon[35965]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-24T09:21:59.482 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/409467710' entity='client.admin' 2024-03-24T09:21:59.482 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-mon[35965]: from='client.? ' entity='client.admin' 2024-03-24T09:21:59.622 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:59.480+0000 7fc30e88a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:21:59.622 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:59.577+0000 7fc30e88a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:21:59.774 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:59.482+0000 7fab530bc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:21:59.774 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:59.606+0000 7fab530bc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:21:59.903 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:59.724+0000 7fc30e88a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:21:59.903 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:59.820+0000 7fc30e88a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:22:00.065 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:21:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:21:59.764+0000 7f238cc9e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T09:22:00.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:21:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:21:59.893+0000 7f238cc9e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T09:22:00.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.065+0000 7f238cc9e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:22:00.172 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:21:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:21:59.903+0000 7fc30e88a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:22:00.172 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:00.172+0000 7fc30e88a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:22:00.241 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:59.778+0000 7fab530bc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T09:22:00.241 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:59.886+0000 7fab530bc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:22:00.241 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:21:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:21:59.971+0000 7fab530bc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:22:00.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.184+0000 7f238cc9e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T09:22:00.316 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.296+0000 7f238cc9e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T09:22:00.490 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:00.283+0000 7fc30e88a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:22:00.490 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:00.371+0000 7fc30e88a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:22:00.524 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:00.240+0000 7fab530bc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:22:00.524 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:00.365+0000 7fab530bc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:22:00.524 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:00.470+0000 7fab530bc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:22:00.743 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:00.490+0000 7fc30e88a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:22:00.744 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:00.667+0000 7fc30e88a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:22:00.922 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.627+0000 7f238cc9e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T09:22:00.922 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.738+0000 7f238cc9e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T09:22:00.922 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.825+0000 7f238cc9e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T09:22:00.922 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:00 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:00.921+0000 7f238cc9e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 -- 172.21.15.116:0/3532706405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 msgr2=0x7f81f010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 --2- 172.21.15.116:0/3532706405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f010df80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009990 tx=0x7f81e0009ca0 comp rx=0 tx=0).stop 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 -- 172.21.15.116:0/3532706405 shutdown_connections 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 --2- 172.21.15.116:0/3532706405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f010df80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 --2- 172.21.15.116:0/3532706405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0107140 0x7f81f0107520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 --2- 172.21.15.116:0/3532706405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81f0101100 0x7f81f0101560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 -- 172.21.15.116:0/3532706405 >> 172.21.15.116:0/3532706405 conn(0x7f81f0078030 msgr2=0x7f81f0078440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:00.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 -- 172.21.15.116:0/3532706405 shutdown_connections 2024-03-24T09:22:00.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 -- 172.21.15.116:0/3532706405 wait complete. 2024-03-24T09:22:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.964+0000 7f81f5f34700 1 Processor -- start 2024-03-24T09:22:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 -- start start 2024-03-24T09:22:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0101100 0x7f81f0199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f01999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81f0107140 0x7f81f01a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:00.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81f019a210 con 0x7f81f0101aa0 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a4a40 con 0x7f81f0101100 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81f5f34700 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a4b80 con 0x7f81f0107140 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81eeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f01999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81eeffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f01999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40002/0 (socket says 172.21.15.116:40002) 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81eeffd700 1 -- 172.21.15.116:0/2321956571 learned_addr learned my addr 172.21.15.116:0/2321956571 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81ef7fe700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0101100 0x7f81f0199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81effff700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81f0107140 0x7f81f01a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81ef7fe700 1 -- 172.21.15.116:0/2321956571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81f0107140 msgr2=0x7f81f01a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81ef7fe700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81f0107140 0x7f81f01a4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81ef7fe700 1 -- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 msgr2=0x7f81f01999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.965+0000 7f81ef7fe700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f01999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81ef7fe700 1 -- 172.21.15.116:0/2321956571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e00095f0 con 0x7f81f0101100 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81eeffd700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:00.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81ef7fe700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0101100 0x7f81f0199480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81e800ef40 tx=0x7f81e800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e800c9c0 con 0x7f81f0101100 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f01a4d80 con 0x7f81f0101100 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f81e800f040 con 0x7f81f0101100 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e8014480 con 0x7f81f0101100 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.966+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f01a5270 con 0x7f81f0101100 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.967+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f0109660 con 0x7f81f0101100 2024-03-24T09:22:00.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.968+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 19) ==== 136774+0+0 (secure 0 0 0) 0x7f81e800cb20 con 0x7f81f0101100 2024-03-24T09:22:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.969+0000 7f81ecff9700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 0x7f81dc0a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.969+0000 7f81eeffd700 1 -- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 msgr2=0x7f81dc0a4660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:22:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.969+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f81e80ce2f0 con 0x7f81f0101100 2024-03-24T09:22:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.969+0000 7f81eeffd700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 0x7f81dc0a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:22:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:00.977+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f81e8096ef0 con 0x7f81f0101100 2024-03-24T09:22:01.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:00.567+0000 7fab530bc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T09:22:01.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:00.753+0000 7fab530bc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:22:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.170+0000 7f81eeffd700 1 -- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 msgr2=0x7f81dc0a4660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:22:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.170+0000 7f81eeffd700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 0x7f81dc0a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:22:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.255+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f81f0110fc0 con 0x7f81f0101100 2024-03-24T09:22:01.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.256+0000 7f81ecff9700 1 -- 172.21.15.116:0/2321956571 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7f81e809a5b0 con 0x7f81f0101100 2024-03-24T09:22:01.256 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-24T09:22:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 msgr2=0x7f81dc0a4660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:22:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 0x7f81dc0a4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:01.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0101100 msgr2=0x7f81f0199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0101100 0x7f81f0199480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f81e800ef40 tx=0x7f81e800c3a0 comp rx=0 tx=0).stop 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 shutdown_connections 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f81f0107140 0x7f81f01a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81f0101100 0x7f81f0199480 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f81dc0a21a0 0x7f81dc0a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 --2- 172.21.15.116:0/2321956571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0101aa0 0x7f81f01999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 >> 172.21.15.116:0/2321956571 conn(0x7f81f0078030 msgr2=0x7f81f00ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 shutdown_connections 2024-03-24T09:22:01.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:01.258+0000 7f81f5f34700 1 -- 172.21.15.116:0/2321956571 wait complete. 2024-03-24T09:22:01.308 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:01 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:01.141+0000 7f238cc9e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T09:22:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:01 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2321956571' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:22:01.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:01 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2321956571' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:22:01.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:01 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2321956571' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T09:22:01.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:01 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:01.657+0000 7fc30e88a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:22:01.924 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:01 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:01.789+0000 7fc30e88a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T09:22:01.953 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T09:22:01.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:01.954 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-24T09:22:01.971 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T09:22:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T09:22:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T09:22:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:22:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T09:22:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-24T09:22:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T09:22:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42154sec preferred_lft 42154sec 2024-03-24T09:22:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-24T09:22:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-24T09:22:01.973 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T09:22:01.973 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:22:01.973 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:22:01.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-24T09:22:01.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-24T09:22:01.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-24T09:22:01.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T09:22:01.974 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:22:02.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:22:02.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:22:02.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:02.129 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T09:22:02.156 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-24T09:22:02.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:02.157 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-24T09:22:02.213 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-24T09:22:02.213 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T09:22:02.213 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T09:22:02.213 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T09:22:02.213 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T09:22:02.213 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:22:02.213 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:22:02.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T09:22:02.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T09:22:02.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T09:22:02.213 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:22:02.236 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:02.141+0000 7fc30e88a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:22:02.236 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:02.236+0000 7fc30e88a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:22:02.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:01.852+0000 7fab530bc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:22:02.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:01.991+0000 7fab530bc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:22:02.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:22:02.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:22:02.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:02.334 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-24T09:22:02.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:02.349 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-24T09:22:02.403 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:22:02.403 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:22:02.403 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:02.403 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T09:22:02.403 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:22:02.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:22:02.502 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:02.445+0000 7fc30e88a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:22:02.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:22:02.527 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T09:22:02.527 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:22:02.566 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:02 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:02.245+0000 7f238cc9e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T09:22:02.566 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:02 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:02.376+0000 7f238cc9e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T09:22:02.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:22:02.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:22:02.654 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:02.365+0000 7fab530bc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:22:02.654 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:02.455+0000 7fab530bc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:22:02.655 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-24T09:22:02.655 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-24T09:22:02.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-24T09:22:02.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T09:22:02.655 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-24T09:22:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-24T09:22:02.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:22:02.739 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-03-24T09:22:02.740 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:22:02.740 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:02.754 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T09:22:02.754 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T09:22:02.754 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:02.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-24T09:22:02.817 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:02.647+0000 7fc30e88a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:22:02.844 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-24T09:22:02.900 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T09:22:02.901 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/35f41a9dbcebef80300000130596a6c83d16e0d20a4f3c64f6fcf65c3551145b/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/c7427c083e1aa70ee5719eaef49d1706a13be3aa707c6f38e2bc752286142654/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3a42c195b5c65f2cf516ac08f6f0b6dd04b6aa345e3c1e24cdb74b8899e2c4b9/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6fb648b372ea483bdf49220fd699c479c8a251e10ca97e038d3933433e391567/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/06636e80d376544c2d95157acfaadee5bf17824d0b5b2de4bc9cff1e3ca7909c/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/4634c5834fbaa5a6ca4762eb78372a6622ed6cb6dad36ef2eccc759d01803633/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/f9cbea23acb91d910aab346dbe893b7881e8e5a4d632350f71d0763c94c3fb8d/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/bdda17d67e3e115dae972830f914e98b56622b46d8eb7d7e9d9bc4b670fb8a97/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d3b738dc89f8fedf5f42454e24f014603e5095215d346fcdcb3ab94444177c21/merged 2024-03-24T09:22:02.902 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/75c4d7635463175883f7e9c2a30a7bb3f3b6fe121a2c13484d27003bc55fbf42/merged 2024-03-24T09:22:02.903 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-24T09:22:02.903 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:02.903 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:02.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:02.903 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-24T09:22:02.955 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T09:22:02.955 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/f1e517f0-a347-4e9a-8ab1-7fed09784049 --client_fs=cephfs) 2024-03-24T09:22:02.997 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-24T09:22:03.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:02.654+0000 7fab530bc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:22:03.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:02.871+0000 7fab530bc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:22:03.025 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T09:22:03.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:02 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:02.720+0000 7f238cc9e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T09:22:03.067 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:02 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:02.831+0000 7f238cc9e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T09:22:03.081 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-24T09:22:03.081 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-24T09:22:03.081 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-24T09:22:03.082 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/35f41a9dbcebef80300000130596a6c83d16e0d20a4f3c64f6fcf65c3551145b/merged 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/c7427c083e1aa70ee5719eaef49d1706a13be3aa707c6f38e2bc752286142654/merged 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3a42c195b5c65f2cf516ac08f6f0b6dd04b6aa345e3c1e24cdb74b8899e2c4b9/merged 2024-03-24T09:22:03.083 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6fb648b372ea483bdf49220fd699c479c8a251e10ca97e038d3933433e391567/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/06636e80d376544c2d95157acfaadee5bf17824d0b5b2de4bc9cff1e3ca7909c/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/4634c5834fbaa5a6ca4762eb78372a6622ed6cb6dad36ef2eccc759d01803633/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/f9cbea23acb91d910aab346dbe893b7881e8e5a4d632350f71d0763c94c3fb8d/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/bdda17d67e3e115dae972830f914e98b56622b46d8eb7d7e9d9bc4b670fb8a97/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d3b738dc89f8fedf5f42454e24f014603e5095215d346fcdcb3ab94444177c21/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/75c4d7635463175883f7e9c2a30a7bb3f3b6fe121a2c13484d27003bc55fbf42/merged 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:03.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:03.084 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-24T09:22:03.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2024-03-24T09:22:03.090+0000 7f0786438480 -1 init, newargv = 0x557eb63607a0 newargc=15 2024-03-24T09:22:03.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[86622]: starting ceph client 2024-03-24T09:22:03.112 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[86622]: starting fuse 2024-03-24T09:22:03.233 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:02.961+0000 7fc30e88a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:22:03.524 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:03.253+0000 7fab530bc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:22:03.566 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:03 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:03.128+0000 7f238cc9e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:22:03.566 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:03 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:03.361+0000 7f238cc9e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T09:22:03.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:03.232+0000 7fc30e88a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:22:03.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:03.474+0000 7fc30e88a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:22:03.795 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:03.544+0000 7fab530bc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:22:04.007 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:03 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:03.705+0000 7f238cc9e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T09:22:04.207 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-03-24T09:22:04.235 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T09:22:04.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:03.794+0000 7fab530bc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-03-24T09:22:04.291 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/selinux 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysctl.service 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/nfs/rpc_pipefs 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-03-24T09:22:04.292 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/35f41a9dbcebef80300000130596a6c83d16e0d20a4f3c64f6fcf65c3551145b/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/c7427c083e1aa70ee5719eaef49d1706a13be3aa707c6f38e2bc752286142654/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/0 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/3a42c195b5c65f2cf516ac08f6f0b6dd04b6aa345e3c1e24cdb74b8899e2c4b9/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/6fb648b372ea483bdf49220fd699c479c8a251e10ca97e038d3933433e391567/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/06636e80d376544c2d95157acfaadee5bf17824d0b5b2de4bc9cff1e3ca7909c/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/4634c5834fbaa5a6ca4762eb78372a6622ed6cb6dad36ef2eccc759d01803633/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/f9cbea23acb91d910aab346dbe893b7881e8e5a4d632350f71d0763c94c3fb8d/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/bdda17d67e3e115dae972830f914e98b56622b46d8eb7d7e9d9bc4b670fb8a97/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/d3b738dc89f8fedf5f42454e24f014603e5095215d346fcdcb3ab94444177c21/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/containers/storage/overlay/75c4d7635463175883f7e9c2a30a7bb3f3b6fe121a2c13484d27003bc55fbf42/merged 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-03-24T09:22:04.293 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:04.294 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:22:04.294 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:04.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:04.294 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-03-24T09:22:04.310 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:04 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:04.007+0000 7f238cc9e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T09:22:04.311 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:04 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:04.310+0000 7f238cc9e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T09:22:04.346 INFO:teuthology.orchestra.run.smithi116.stdout:119 2024-03-24T09:22:04.346 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-24T09:22:04.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-03-24T09:22:04.347 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2024-03-24T09:22:04.348 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-03-24T09:22:04.472 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.86622.asok 2024-03-24T09:22:04.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T09:22:04.481 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:04.297+0000 7fc30e88a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:22:04.482 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.86622.asok 2024-03-24T09:22:04.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:04.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.86622.asok status 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "86622", 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/volumes/qa/sv_0/f1e517f0-a347-4e9a-8ab1-7fed09784049" 2024-03-24T09:22:04.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 15567, 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 15567 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3023577365 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:22:04.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3023577365 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.15567 192.168.0.1:0/3023577365", 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/3023577365", 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 5, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 28, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 93, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs" 2024-03-24T09:22:04.623 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:22:04.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:04.630 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-03-24T09:22:04.630 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:04.645 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-03-24T09:22:04.645 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:04.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:04.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:04.714 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T09:22:04.725 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T09:22:04.725 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:22:04.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- 172.21.15.116:0/2121690762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101820 msgr2=0x7f5404101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- 172.21.15.116:0/2121690762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101820 0x7f5404101ca0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5400009900 tx=0x7f540002bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- 172.21.15.116:0/2121690762 shutdown_connections 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- 172.21.15.116:0/2121690762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54041021e0 0x7f5404108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- 172.21.15.116:0/2121690762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5404101820 0x7f5404101ca0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- 172.21.15.116:0/2121690762 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 0x7f5404100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- 172.21.15.116:0/2121690762 >> 172.21.15.116:0/2121690762 conn(0x7f54040fbba0 msgr2=0x7f54040fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- 172.21.15.116:0/2121690762 shutdown_connections 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- 172.21.15.116:0/2121690762 wait complete. 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 Processor -- start 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- start start 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 0x7f5404197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5404101820 0x7f5404197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 0x7f540419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f540419f140 con 0x7f54041021e0 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f540419f2b0 con 0x7f5404101820 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.859+0000 7f540c615640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f540419f470 con 0x7f5404100630 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 0x7f540419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540a3b1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 0x7f5404197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 0x7f540419ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40034/0 (socket says 172.21.15.116:40034) 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540a3b1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 0x7f5404197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:39944/0 (socket says 172.21.15.116:39944) 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 -- 172.21.15.116:0/565179770 learned_addr learned my addr 172.21.15.116:0/565179770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f5409bb0640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5404101820 0x7f5404197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 -- 172.21.15.116:0/565179770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 msgr2=0x7f5404197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 0x7f5404197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 -- 172.21.15.116:0/565179770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5404101820 msgr2=0x7f5404197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5404101820 0x7f5404197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 -- 172.21.15.116:0/565179770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5400009560 con 0x7f54041021e0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540abb2640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 0x7f540419ec00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00eb10 tx=0x7f53fc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc004b20 con 0x7f54041021e0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53fc004c80 con 0x7f54041021e0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540419f750 con 0x7f54041021e0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc010690 con 0x7f54041021e0 2024-03-24T09:22:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.860+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f540419fbe0 con 0x7f54041021e0 2024-03-24T09:22:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.862+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5404107950 con 0x7f54041021e0 2024-03-24T09:22:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.862+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 19) ==== 136774+0+0 (secure 0 0 0) 0x7f53fc005020 con 0x7f54041021e0 2024-03-24T09:22:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.863+0000 7f53fb7fe640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 0x7f53f00a4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.863+0000 7f540a3b1640 1 -- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 msgr2=0x7f53f00a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:22:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.863+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9208+0+0 (secure 0 0 0) 0x7f53fc014070 con 0x7f54041021e0 2024-03-24T09:22:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.863+0000 7f540a3b1640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 0x7f53f00a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T09:22:04.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:04.867+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f53fc097280 con 0x7f54041021e0 2024-03-24T09:22:05.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:04.713+0000 7fab530bc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:22:05.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.064+0000 7f540a3b1640 1 -- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 msgr2=0x7f53f00a4670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/2363601294 2024-03-24T09:22:05.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.064+0000 7f540a3b1640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 0x7f53f00a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T09:22:05.067 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:04.912+0000 7fc30e88a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:22:05.067 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.016+0000 7fc30e88a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:22:05.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.110+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f540402d130 con 0x7f54041021e0 2024-03-24T09:22:05.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.162+0000 7f53fb7fe640 1 -- 172.21.15.116:0/565179770 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) ==== 117+0+0 (secure 0 0 0) 0x7f53fc09a940 con 0x7f54041021e0 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 msgr2=0x7f53f00a4670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 0x7f53f00a4670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 msgr2=0x7f540419ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 0x7f540419ec00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00eb10 tx=0x7f53fc00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 shutdown_connections 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54041021e0 0x7f540419ec00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5404101820 0x7f5404197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.116:6800/2363601294,v1:172.21.15.116:6801/2363601294] conn(0x7f53f00a21b0 0x7f53f00a4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 --2- 172.21.15.116:0/565179770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5404100630 0x7f5404197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 >> 172.21.15.116:0/565179770 conn(0x7f54040fbba0 msgr2=0x7f5404104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 shutdown_connections 2024-03-24T09:22:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.164+0000 7f540c615640 1 -- 172.21.15.116:0/565179770 wait complete. 2024-03-24T09:22:05.177 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-24T09:22:05.187 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-24T09:22:05.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:22:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.342+0000 7ff0b5047640 1 -- 172.21.15.116:0/1870237645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 msgr2=0x7ff0b0108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.342+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1870237645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 0x7ff0b0108a70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009ca0 tx=0x7ff0a802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 -- 172.21.15.116:0/1870237645 shutdown_connections 2024-03-24T09:22:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1870237645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 0x7ff0b0108a70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1870237645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0b0101820 0x7ff0b0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1870237645 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 -- 172.21.15.116:0/1870237645 >> 172.21.15.116:0/1870237645 conn(0x7ff0b00fbba0 msgr2=0x7ff0b00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 -- 172.21.15.116:0/1870237645 shutdown_connections 2024-03-24T09:22:05.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.343+0000 7ff0b5047640 1 -- 172.21.15.116:0/1870237645 wait complete. 2024-03-24T09:22:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 Processor -- start 2024-03-24T09:22:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 -- start start 2024-03-24T09:22:05.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0b0101820 0x7ff0b0197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 0x7ff0b019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b019f180 con 0x7ff0b01021e0 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b019f2f0 con 0x7ff0b0101820 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0b5047640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b019f4b0 con 0x7ff0b0100630 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0aed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0aed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:39976/0 (socket says 172.21.15.116:39976) 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0aed9d640 1 -- 172.21.15.116:0/1665270399 learned_addr learned my addr 172.21.15.116:0/1665270399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0a659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0b0101820 0x7ff0b0197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.344+0000 7ff0af59e640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 0x7ff0b019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0aed9d640 1 -- 172.21.15.116:0/1665270399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0b0101820 msgr2=0x7ff0b0197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0aed9d640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0b0101820 0x7ff0b0197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0aed9d640 1 -- 172.21.15.116:0/1665270399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 msgr2=0x7ff0b019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0aed9d640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 0x7ff0b019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0aed9d640 1 -- 172.21.15.116:0/1665270399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a8009900 con 0x7ff0b0100630 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0aed9d640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0197690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000e9c0 tx=0x7ff0a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a00096e0 con 0x7ff0b0100630 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0a00044e0 con 0x7ff0b0100630 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b019f790 con 0x7ff0b0100630 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0010430 con 0x7ff0b0100630 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.345+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b019fbf0 con 0x7ff0b0100630 2024-03-24T09:22:05.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.346+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b010bef0 con 0x7ff0b0100630 2024-03-24T09:22:05.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.348+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 20) ==== 136441+0+0 (secure 0 0 0) 0x7ff0a0003f80 con 0x7ff0b0100630 2024-03-24T09:22:05.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.349+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff0a0014070 con 0x7ff0b0100630 2024-03-24T09:22:05.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.356+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ff0a0096dc0 con 0x7ff0b0100630 2024-03-24T09:22:05.448 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:05 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/565179770' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T09:22:05.448 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:05.332+0000 7fab530bc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:22:05.494 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.241+0000 7fc30e88a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:22:05.495 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/565179770' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T09:22:05.513 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:05 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/565179770' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-24T09:22:05.513 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:05 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:05.347+0000 7f238cc9e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T09:22:05.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.600+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0b010c080 con 0x7ff0b0100630 2024-03-24T09:22:05.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.602+0000 7ff0acd99640 1 -- 172.21.15.116:0/1665270399 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25293 (secure 0 0 0) 0x7ff0a009a480 con 0x7ff0b0100630 2024-03-24T09:22:05.603 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:05.603 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:22:05.255869+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 msgr2=0x7ff0b0197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0197690 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000e9c0 tx=0x7ff0a000ee80 comp rx=0 tx=0).stop 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 shutdown_connections 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0b01021e0 0x7ff0b019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0b0101820 0x7ff0b0197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 --2- 172.21.15.116:0/1665270399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0b0100630 0x7ff0b0197690 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 >> 172.21.15.116:0/1665270399 conn(0x7ff0b00fbba0 msgr2=0x7ff0b00fc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 shutdown_connections 2024-03-24T09:22:05.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.604+0000 7ff0b5047640 1 -- 172.21.15.116:0/1665270399 wait complete. 2024-03-24T09:22:05.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:22:05.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:22:05.717 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:05.447+0000 7fab530bc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 -- 172.21.15.116:0/1590536238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac107450 msgr2=0x7f0dac109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 --2- 172.21.15.116:0/1590536238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac107450 0x7f0dac109840 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009ca0 tx=0x7f0da402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 -- 172.21.15.116:0/1590536238 shutdown_connections 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 --2- 172.21.15.116:0/1590536238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac107450 0x7f0dac109840 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 --2- 172.21.15.116:0/1590536238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dac104b20 0x7f0dac106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 --2- 172.21.15.116:0/1590536238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 0x7f0dac1045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 -- 172.21.15.116:0/1590536238 >> 172.21.15.116:0/1590536238 conn(0x7f0dac0fbb80 msgr2=0x7f0dac0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:05.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 -- 172.21.15.116:0/1590536238 shutdown_connections 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 -- 172.21.15.116:0/1590536238 wait complete. 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.723+0000 7f0db435e640 1 Processor -- start 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 -- start start 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 0x7f0dac1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 0x7f0dac197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dac107450 0x7f0dac19ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac19f1a0 con 0x7f0dac104b20 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac19f310 con 0x7f0dac107450 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0dac19f4d0 con 0x7f0dac1021f0 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 0x7f0dac197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db20fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 0x7f0dac1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 0x7f0dac197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40192/0 (socket says 172.21.15.116:40192) 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db20fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 0x7f0dac1976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40068/0 (socket says 172.21.15.116:40068) 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 -- 172.21.15.116:0/3766679595 learned_addr learned my addr 172.21.15.116:0/3766679595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:05.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db28fb640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dac107450 0x7f0dac19ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 -- 172.21.15.116:0/3766679595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 msgr2=0x7f0dac1976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 0x7f0dac1976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 -- 172.21.15.116:0/3766679595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dac107450 msgr2=0x7f0dac19ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dac107450 0x7f0dac19ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 -- 172.21.15.116:0/3766679595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0da4009900 con 0x7f0dac104b20 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db18f9640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 0x7f0dac197bf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00e9c0 tx=0x7f0d9c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c009700 con 0x7f0dac104b20 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0d9c00cd80 con 0x7f0dac104b20 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dac19f7b0 con 0x7f0dac104b20 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c010430 con 0x7f0dac104b20 2024-03-24T09:22:05.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.724+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dac19fcc0 con 0x7f0dac104b20 2024-03-24T09:22:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.726+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dac0fd670 con 0x7f0dac104b20 2024-03-24T09:22:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.727+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 20) ==== 136441+0+0 (secure 0 0 0) 0x7f0d9c004f50 con 0x7f0dac104b20 2024-03-24T09:22:05.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.727+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0d9c014070 con 0x7f0dac104b20 2024-03-24T09:22:05.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.732+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0d9c0d1030 con 0x7f0dac104b20 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc26ba1c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc26ba1c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc26ba1c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc26ba1c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc26ba1c700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc268a16700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc268a16700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc268a16700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc268a16700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.818 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:22:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:22:05.556+0000 7fc268a16700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:22:05.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.913+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0dac02d1d0 con 0x7f0dac104b20 2024-03-24T09:22:05.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.913+0000 7f0da37fe640 1 -- 172.21.15.116:0/3766679595 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f0d9c096e50 con 0x7f0dac104b20 2024-03-24T09:22:05.914 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 msgr2=0x7f0dac197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 0x7f0dac197bf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c00e9c0 tx=0x7f0d9c00ee80 comp rx=0 tx=0).stop 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 shutdown_connections 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dac107450 0x7f0dac19ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dac104b20 0x7f0dac197bf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 --2- 172.21.15.116:0/3766679595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0dac1021f0 0x7f0dac1976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 >> 172.21.15.116:0/3766679595 conn(0x7f0dac0fbb80 msgr2=0x7f0dac102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 shutdown_connections 2024-03-24T09:22:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:05.915+0000 7f0db435e640 1 -- 172.21.15.116:0/3766679595 wait complete. 2024-03-24T09:22:05.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:22:06.024 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:22:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z[37854]: 2024-03-24T09:22:05.716+0000 7fab530bc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 -- 172.21.15.116:0/1556141970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41032e0 msgr2=0x7ffbf4109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/1556141970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41032e0 0x7ffbf4109b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ffbec009ca0 tx=0x7ffbec02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 -- 172.21.15.116:0/1556141970 shutdown_connections 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/1556141970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf41032e0 0x7ffbf4109b70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/1556141970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf4102920 0x7ffbf4102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/1556141970 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 -- 172.21.15.116:0/1556141970 >> 172.21.15.116:0/1556141970 conn(0x7ffbf40fccc0 msgr2=0x7ffbf40ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 -- 172.21.15.116:0/1556141970 shutdown_connections 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.034+0000 7ffbf94e8640 1 -- 172.21.15.116:0/1556141970 wait complete. 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 Processor -- start 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 -- start start 2024-03-24T09:22:06.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 0x7ffbf4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf41032e0 0x7ffbf419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf419f170 con 0x7ffbf4102920 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf419f2e0 con 0x7ffbf41032e0 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf94e8640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf419f4a0 con 0x7ffbf4101730 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 0x7ffbf4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf2ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf2ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40082/0 (socket says 172.21.15.116:40082) 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 0x7ffbf4197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40218/0 (socket says 172.21.15.116:40218) 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf2ffd640 1 -- 172.21.15.116:0/2768071853 learned_addr learned my addr 172.21.15.116:0/2768071853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf37fe640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf41032e0 0x7ffbf419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf37fe640 1 -- 172.21.15.116:0/2768071853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 msgr2=0x7ffbf4197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf37fe640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf37fe640 1 -- 172.21.15.116:0/2768071853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 msgr2=0x7ffbf4197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf37fe640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 0x7ffbf4197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf37fe640 1 -- 172.21.15.116:0/2768071853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe8009490 con 0x7ffbf41032e0 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf2ffd640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.035+0000 7ffbf27fc640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 0x7ffbf4197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.036+0000 7ffbf37fe640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf41032e0 0x7ffbf419ec30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffbec02c630 tx=0x7ffbec032980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.036+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbec034e70 con 0x7ffbf41032e0 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.036+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbec009900 con 0x7ffbf41032e0 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.036+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffbec003e20 con 0x7ffbf41032e0 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.036+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbec004810 con 0x7ffbf41032e0 2024-03-24T09:22:06.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.036+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbf419fa60 con 0x7ffbf41032e0 2024-03-24T09:22:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.037+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbf4108ad0 con 0x7ffbf41032e0 2024-03-24T09:22:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.038+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 20) ==== 136441+0+0 (secure 0 0 0) 0x7ffbec03b040 con 0x7ffbf41032e0 2024-03-24T09:22:06.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.038+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ffbec0f0450 con 0x7ffbf41032e0 2024-03-24T09:22:06.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.043+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffbec0b93a0 con 0x7ffbf41032e0 2024-03-24T09:22:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.258+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffbf410d160 con 0x7ffbf41032e0 2024-03-24T09:22:06.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.259+0000 7ffbdbfff640 1 -- 172.21.15.116:0/2768071853 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7ffbec039020 con 0x7ffbf41032e0 2024-03-24T09:22:06.259 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:22:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf41032e0 msgr2=0x7ffbf419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf41032e0 0x7ffbf419ec30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffbec02c630 tx=0x7ffbec032980 comp rx=0 tx=0).stop 2024-03-24T09:22:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 shutdown_connections 2024-03-24T09:22:06.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbf41032e0 0x7ffbf419ec30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbf4102920 0x7ffbf4197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 --2- 172.21.15.116:0/2768071853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffbf4101730 0x7ffbf4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 >> 172.21.15.116:0/2768071853 conn(0x7ffbf40fccc0 msgr2=0x7ffbf4105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 shutdown_connections 2024-03-24T09:22:06.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.261+0000 7ffbf94e8640 1 -- 172.21.15.116:0/2768071853 wait complete. 2024-03-24T09:22:06.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/565179770' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T09:22:06.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:22:06.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: Active manager daemon x restarted 2024-03-24T09:22:06.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: Activating manager daemon x 2024-03-24T09:22:06.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T09:22:06.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: mgrmap e20: x(active, starting, since 0.012317s), standbys: z, y 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:22:06.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:22:06.284 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:22:06.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:22:06.286 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:22:06.286 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:22:06.286 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:22:06.286 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:22:06.286 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: Manager daemon x is now available 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:22:06.287 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: [24/Mar/2024:09:22:05] ENGINE Bus STARTING 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1665270399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: [24/Mar/2024:09:22:05] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: Standby manager daemon z restarted 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: Standby manager daemon z started 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: [24/Mar/2024:09:22:05] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: [24/Mar/2024:09:22:05] ENGINE Bus STARTED 2024-03-24T09:22:06.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:06 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3766679595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 -- 172.21.15.116:0/3985733815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c1021e0 msgr2=0x7fb32c108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 --2- 172.21.15.116:0/3985733815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c1021e0 0x7fb32c108a70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb324009ca0 tx=0x7fb32402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 -- 172.21.15.116:0/3985733815 shutdown_connections 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 --2- 172.21.15.116:0/3985733815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c1021e0 0x7fb32c108a70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 --2- 172.21.15.116:0/3985733815 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32c101820 0x7fb32c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 --2- 172.21.15.116:0/3985733815 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 0x7fb32c100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 -- 172.21.15.116:0/3985733815 >> 172.21.15.116:0/3985733815 conn(0x7fb32c0fbbc0 msgr2=0x7fb32c0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 -- 172.21.15.116:0/3985733815 shutdown_connections 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.404+0000 7fb333756640 1 -- 172.21.15.116:0/3985733815 wait complete. 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 Processor -- start 2024-03-24T09:22:06.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 -- start start 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 0x7fb32c06a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 0x7fb32c19c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32c1021e0 0x7fb32c19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c19f130 con 0x7fb32c101820 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c19f2a0 con 0x7fb32c1021e0 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c19f460 con 0x7fb32c100630 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 0x7fb32c19c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb3314f2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 0x7fb32c06a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 0x7fb32c19c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40244/0 (socket says 172.21.15.116:40244) 2024-03-24T09:22:06.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb3314f2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 0x7fb32c06a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40106/0 (socket says 172.21.15.116:40106) 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 -- 172.21.15.116:0/381128319 learned_addr learned my addr 172.21.15.116:0/381128319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb331cf3640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32c1021e0 0x7fb32c19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 -- 172.21.15.116:0/381128319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 msgr2=0x7fb32c06a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 0x7fb32c06a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 -- 172.21.15.116:0/381128319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32c1021e0 msgr2=0x7fb32c19e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32c1021e0 0x7fb32c19e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 -- 172.21.15.116:0/381128319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb324009900 con 0x7fb32c101820 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb330cf1640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 0x7fb32c19c360 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00d950 tx=0x7fb31c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:06.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c009a30 con 0x7fb32c101820 2024-03-24T09:22:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.405+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32c196190 con 0x7fb32c101820 2024-03-24T09:22:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.406+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb31c0044d0 con 0x7fb32c101820 2024-03-24T09:22:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.406+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0047a0 con 0x7fb32c101820 2024-03-24T09:22:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.406+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32c1966f0 con 0x7fb32c101820 2024-03-24T09:22:06.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.406+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb32c107950 con 0x7fb32c101820 2024-03-24T09:22:06.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.408+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7fb31c009b90 con 0x7fb32c101820 2024-03-24T09:22:06.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.408+0000 7fb3227fc640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3180a1ec0 0x7fb3180a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.408+0000 7fb3314f2640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3180a1ec0 0x7fb3180a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.408+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb31c0cd6d0 con 0x7fb32c101820 2024-03-24T09:22:06.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.409+0000 7fb3314f2640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3180a1ec0 0x7fb3180a4380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb32c101680 tx=0x7fb32800f7a0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:06.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.413+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb31c0964c0 con 0x7fb32c101820 2024-03-24T09:22:06.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/565179770' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T09:22:06.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:22:06.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: Active manager daemon x restarted 2024-03-24T09:22:06.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: Activating manager daemon x 2024-03-24T09:22:06.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T09:22:06.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: mgrmap e20: x(active, starting, since 0.012317s), standbys: z, y 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:22:06.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:22:06.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:22:06.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:22:06.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:22:06.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:22:06.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:22:06.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:22:06.528 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: Manager daemon x is now available 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:22:06.529 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: [24/Mar/2024:09:22:05] ENGINE Bus STARTING 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1665270399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: [24/Mar/2024:09:22:05] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: Standby manager daemon z restarted 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: Standby manager daemon z started 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: [24/Mar/2024:09:22:05] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T09:22:06.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: [24/Mar/2024:09:22:05] ENGINE Bus STARTED 2024-03-24T09:22:06.531 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:06 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3766679595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:06.536 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:06.099+0000 7f238cc9e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T09:22:06.536 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:06.234+0000 7f238cc9e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T09:22:06.537 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/565179770' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-24T09:22:06.537 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:22:06.537 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: Active manager daemon x restarted 2024-03-24T09:22:06.537 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: Activating manager daemon x 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: mgrmap e20: x(active, starting, since 0.012317s), standbys: z, y 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:22:06.538 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T09:22:06.539 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T09:22:06.540 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: Manager daemon x is now available 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: [24/Mar/2024:09:22:05] ENGINE Bus STARTING 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1665270399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: [24/Mar/2024:09:22:05] ENGINE Serving on https://172.21.15.116:7150 2024-03-24T09:22:06.541 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: Standby manager daemon z restarted 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: Standby manager daemon z started 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.143:0/1355176199' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: [24/Mar/2024:09:22:05] ENGINE Serving on http://172.21.15.116:8765 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: [24/Mar/2024:09:22:05] ENGINE Bus STARTED 2024-03-24T09:22:06.542 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3766679595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.588+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb32c10bef0 con 0x7fb32c101820 2024-03-24T09:22:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.588+0000 7fb3227fc640 1 -- 172.21.15.116:0/381128319 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fb32c10bef0 con 0x7fb32c101820 2024-03-24T09:22:06.589 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:22:06.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3180a1ec0 msgr2=0x7fb3180a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3180a1ec0 0x7fb3180a4380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb32c101680 tx=0x7fb32800f7a0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 msgr2=0x7fb32c19c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 0x7fb32c19c360 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00d950 tx=0x7fb31c00de10 comp rx=0 tx=0).stop 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 shutdown_connections 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32c1021e0 0x7fb32c19e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32c101820 0x7fb32c19c360 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3180a1ec0 0x7fb3180a4380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 --2- 172.21.15.116:0/381128319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb32c100630 0x7fb32c06a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.590+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 >> 172.21.15.116:0/381128319 conn(0x7fb32c0fbbc0 msgr2=0x7fb32c104b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.591+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 shutdown_connections 2024-03-24T09:22:06.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.591+0000 7fb333756640 1 -- 172.21.15.116:0/381128319 wait complete. 2024-03-24T09:22:06.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 -- 172.21.15.116:0/3893187480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 msgr2=0x7f87d4102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 --2- 172.21.15.116:0/3893187480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4102f10 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009900 tx=0x7f87c802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 -- 172.21.15.116:0/3893187480 shutdown_connections 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 --2- 172.21.15.116:0/3893187480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87d4103450 0x7f87d4109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 --2- 172.21.15.116:0/3893187480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4102f10 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 --2- 172.21.15.116:0/3893187480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87d40ffa20 0x7f87d40ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 -- 172.21.15.116:0/3893187480 >> 172.21.15.116:0/3893187480 conn(0x7f87d40fb310 msgr2=0x7f87d40fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 -- 172.21.15.116:0/3893187480 shutdown_connections 2024-03-24T09:22:06.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.718+0000 7f87d9023640 1 -- 172.21.15.116:0/3893187480 wait complete. 2024-03-24T09:22:06.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 Processor -- start 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 -- start start 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87d40ffa20 0x7f87d4102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87d4103450 0x7f87d4100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41016e0 con 0x7f87d4102a90 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4101850 con 0x7f87d4103450 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d9023640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f87d41a1770 con 0x7f87d40ffa20 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55672/0 (socket says 172.21.15.116:55672) 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 -- 172.21.15.116:0/4132302953 learned_addr learned my addr 172.21.15.116:0/4132302953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:06.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d2d9d640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87d40ffa20 0x7f87d4102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d359e640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87d4103450 0x7f87d4100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 -- 172.21.15.116:0/4132302953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87d40ffa20 msgr2=0x7f87d4102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87d40ffa20 0x7f87d4102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 -- 172.21.15.116:0/4132302953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87d4103450 msgr2=0x7f87d4100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87d4103450 0x7f87d4100f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.719+0000 7f87d259c640 1 -- 172.21.15.116:0/4132302953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c8009560 con 0x7f87d4102a90 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.720+0000 7f87d259c640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4100a40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009430 tx=0x7f87c8004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.720+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8004dc0 con 0x7f87d4102a90 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.720+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f87c802de90 con 0x7f87d4102a90 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.720+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8003d10 con 0x7f87d4102a90 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.720+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d41a19c0 con 0x7f87d4102a90 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.720+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d41a1ed0 con 0x7f87d4102a90 2024-03-24T09:22:06.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.721+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d4108be0 con 0x7f87d4102a90 2024-03-24T09:22:06.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.722+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f87c8004f20 con 0x7f87d4102a90 2024-03-24T09:22:06.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.722+0000 7f87bbfff640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f87bc0a1ec0 0x7f87bc0a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:06.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.723+0000 7f87d2d9d640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f87bc0a1ec0 0x7f87bc0a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:06.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.723+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f87c80f0150 con 0x7f87d4102a90 2024-03-24T09:22:06.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.723+0000 7f87d2d9d640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f87bc0a1ec0 0x7f87bc0a4380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f87d4069a60 tx=0x7f87c4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:06.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.727+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f87c80b8f40 con 0x7f87d4102a90 2024-03-24T09:22:06.816 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:22:06 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y[37213]: 2024-03-24T09:22:06.536+0000 7f238cc9e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T09:22:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.907+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f87d402d160 con 0x7f87d4102a90 2024-03-24T09:22:06.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.908+0000 7f87bbfff640 1 -- 172.21.15.116:0/4132302953 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f87c80bc600 con 0x7f87d4102a90 2024-03-24T09:22:06.908 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:22:06.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f87bc0a1ec0 msgr2=0x7f87bc0a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f87bc0a1ec0 0x7f87bc0a4380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f87d4069a60 tx=0x7f87c4005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 msgr2=0x7f87d4100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4100a40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f87c8009430 tx=0x7f87c8004140 comp rx=0 tx=0).stop 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 shutdown_connections 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f87d4103450 0x7f87d4100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87d4102a90 0x7f87d4100a40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f87bc0a1ec0 0x7f87bc0a4380 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 --2- 172.21.15.116:0/4132302953 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f87d40ffa20 0x7f87d4102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 >> 172.21.15.116:0/4132302953 conn(0x7f87d40fb310 msgr2=0x7f87d4105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 shutdown_connections 2024-03-24T09:22:06.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:06.910+0000 7f87d9023640 1 -- 172.21.15.116:0/4132302953 wait complete. 2024-03-24T09:22:06.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.023+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/1891690989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 msgr2=0x7f1b78100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.023+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/1891690989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b78100a50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1b68009900 tx=0x7f1b6802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/1891690989 shutdown_connections 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/1891690989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b781021e0 0x7f1b78108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/1891690989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b78101820 0x7f1b78101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/1891690989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b78100a50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/1891690989 >> 172.21.15.116:0/1891690989 conn(0x7f1b780fbbc0 msgr2=0x7f1b780fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/1891690989 shutdown_connections 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/1891690989 wait complete. 2024-03-24T09:22:07.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 Processor -- start 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 -- start start 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b7806a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.024+0000 7f1b7d5b5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b78101820 0x7f1b7819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b7d5b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b781021e0 0x7f1b7819ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b7d5b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7819f230 con 0x7f1b78100630 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b7d5b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7819f3a0 con 0x7f1b781021e0 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b7d5b5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7819f560 con 0x7f1b78101820 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b7806a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b7806a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55692/0 (socket says 172.21.15.116:55692) 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 -- 172.21.15.116:0/525262438 learned_addr learned my addr 172.21.15.116:0/525262438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b777fe640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b781021e0 0x7f1b7819ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b767fc640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b78101820 0x7f1b7819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 -- 172.21.15.116:0/525262438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b78101820 msgr2=0x7f1b7819c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b78101820 0x7f1b7819c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 -- 172.21.15.116:0/525262438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b781021e0 msgr2=0x7f1b7819ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b781021e0 0x7f1b7819ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 -- 172.21.15.116:0/525262438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b68009560 con 0x7f1b78100630 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b777fe640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b781021e0 0x7f1b7819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b767fc640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b78101820 0x7f1b7819c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b76ffd640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b7806a850 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b6802c7d0 tx=0x7f1b68004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68004500 con 0x7f1b78100630 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b7819f780 con 0x7f1b78100630 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b6802de90 con 0x7f1b78100630 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68005350 con 0x7f1b78100630 2024-03-24T09:22:07.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.025+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b7819fce0 con 0x7f1b78100630 2024-03-24T09:22:07.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.026+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b781079c0 con 0x7f1b78100630 2024-03-24T09:22:07.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.028+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 21) ==== 136568+0+0 (secure 0 0 0) 0x7f1b680054b0 con 0x7f1b78100630 2024-03-24T09:22:07.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.028+0000 7f1b6ffff640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b640a1ec0 0x7f1b640a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.029+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1b680f03c0 con 0x7f1b78100630 2024-03-24T09:22:07.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.029+0000 7f1b767fc640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b640a1ec0 0x7f1b640a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.029+0000 7f1b767fc640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b640a1ec0 0x7f1b640a4380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b7806a5c0 tx=0x7f1b60005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:07.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.033+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1b680b91b0 con 0x7f1b78100630 2024-03-24T09:22:07.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.233+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b7810c0f0 con 0x7f1b78100630 2024-03-24T09:22:07.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.235+0000 7f1b6ffff640 1 -- 172.21.15.116:0/525262438 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25293 (secure 0 0 0) 0x7f1b68043090 con 0x7f1b78100630 2024-03-24T09:22:07.235 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:07.236 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:22:05.255869+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:22:07.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b640a1ec0 msgr2=0x7f1b640a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b640a1ec0 0x7f1b640a4380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1b7806a5c0 tx=0x7f1b60005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:07.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 msgr2=0x7f1b7806a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b7806a850 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f1b6802c7d0 tx=0x7f1b68004040 comp rx=0 tx=0).stop 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 shutdown_connections 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b781021e0 0x7f1b7819ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b640a1ec0 0x7f1b640a4380 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b78101820 0x7f1b7819c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 --2- 172.21.15.116:0/525262438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b78100630 0x7f1b7806a850 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 >> 172.21.15.116:0/525262438 conn(0x7f1b780fbbc0 msgr2=0x7f1b78104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 shutdown_connections 2024-03-24T09:22:07.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.237+0000 7f1b7d5b5640 1 -- 172.21.15.116:0/525262438 wait complete. 2024-03-24T09:22:07.250 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:22:07.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 -- 172.21.15.116:0/205856380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10040ff9e0 msgr2=0x7f10040ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 --2- 172.21.15.116:0/205856380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10040ff9e0 0x7f10040ffe00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1000009900 tx=0x7f100002bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 -- 172.21.15.116:0/205856380 shutdown_connections 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 --2- 172.21.15.116:0/205856380 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1004103410 0x7f1004109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 --2- 172.21.15.116:0/205856380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004102a50 0x7f1004102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 --2- 172.21.15.116:0/205856380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10040ff9e0 0x7f10040ffe00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.367+0000 7f100c64e640 1 -- 172.21.15.116:0/205856380 >> 172.21.15.116:0/205856380 conn(0x7f10040fb2f0 msgr2=0x7f10040fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:07.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.368+0000 7f100c64e640 1 -- 172.21.15.116:0/205856380 shutdown_connections 2024-03-24T09:22:07.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.368+0000 7f100c64e640 1 -- 172.21.15.116:0/205856380 wait complete. 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.368+0000 7f100c64e640 1 Processor -- start 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.368+0000 7f100c64e640 1 -- start start 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.368+0000 7f100c64e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10040ff9e0 0x7f10041976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.368+0000 7f100c64e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 0x7f1004197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100c64e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004103410 0x7f100419eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100c64e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f100419f1e0 con 0x7f1004102a50 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100c64e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f100419f350 con 0x7f1004103410 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100c64e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f100419f510 con 0x7f10040ff9e0 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f1009be9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 0x7f1004197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f1009be9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 0x7f1004197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55716/0 (socket says 172.21.15.116:55716) 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f1009be9640 1 -- 172.21.15.116:0/1674500320 learned_addr learned my addr 172.21.15.116:0/1674500320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:07.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10040ff9e0 0x7f10041976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100abeb640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004103410 0x7f100419eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 -- 172.21.15.116:0/1674500320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004103410 msgr2=0x7f100419eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004103410 0x7f100419eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 -- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 msgr2=0x7f1004197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 0x7f1004197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 -- 172.21.15.116:0/1674500320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1000009560 con 0x7f10040ff9e0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f1009be9640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 0x7f1004197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100abeb640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004103410 0x7f100419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f100a3ea640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10040ff9e0 0x7f10041976f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1000036040 tx=0x7f1000004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.369+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000004500 con 0x7f10040ff9e0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.370+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100419f790 con 0x7f10040ff9e0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.370+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f100002de90 con 0x7f10040ff9e0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.370+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10000052b0 con 0x7f10040ff9e0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.370+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f100419fcc0 con 0x7f10040ff9e0 2024-03-24T09:22:07.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.371+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1004108c70 con 0x7f10040ff9e0 2024-03-24T09:22:07.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.372+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7f1000003cc0 con 0x7f10040ff9e0 2024-03-24T09:22:07.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.372+0000 7f0ffb7fe640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ff00a1ec0 0x7f0ff00a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.373+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f10000f02f0 con 0x7f10040ff9e0 2024-03-24T09:22:07.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.373+0000 7f1009be9640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ff00a1ec0 0x7f0ff00a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.373+0000 7f1009be9640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ff00a1ec0 0x7f0ff00a4380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1004198c00 tx=0x7f0ff4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:07.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.378+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f10000b90e0 con 0x7f10040ff9e0 2024-03-24T09:22:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2768071853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: mgrmap e21: x(active, since 1.01837s), standbys: y, z 2024-03-24T09:22:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: Standby manager daemon y restarted 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: Standby manager daemon y started 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/381128319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:07 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4132302953' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2768071853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: mgrmap e21: x(active, since 1.01837s), standbys: y, z 2024-03-24T09:22:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: Standby manager daemon y restarted 2024-03-24T09:22:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: Standby manager daemon y started 2024-03-24T09:22:07.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T09:22:07.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:22:07.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T09:22:07.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:22:07.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/381128319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:07 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4132302953' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2768071853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: mgrmap e21: x(active, since 1.01837s), standbys: y, z 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: Standby manager daemon y restarted 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: Standby manager daemon y started 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='mgr.? 172.21.15.117:0/513599091' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/381128319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:07 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4132302953' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:07.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.573+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f100402d160 con 0x7f10040ff9e0 2024-03-24T09:22:07.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.574+0000 7f0ffb7fe640 1 -- 172.21.15.116:0/1674500320 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f10000bc7a0 con 0x7f10040ff9e0 2024-03-24T09:22:07.574 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ff00a1ec0 msgr2=0x7f0ff00a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ff00a1ec0 0x7f0ff00a4380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1004198c00 tx=0x7f0ff4005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10040ff9e0 msgr2=0x7f10041976f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10040ff9e0 0x7f10041976f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1000036040 tx=0x7f1000004040 comp rx=0 tx=0).stop 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 shutdown_connections 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1004103410 0x7f100419eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1004102a50 0x7f1004197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ff00a1ec0 0x7f0ff00a4380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 --2- 172.21.15.116:0/1674500320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f10040ff9e0 0x7f10041976f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 >> 172.21.15.116:0/1674500320 conn(0x7f10040fb2f0 msgr2=0x7f1004105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:07.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 shutdown_connections 2024-03-24T09:22:07.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.576+0000 7f100c64e640 1 -- 172.21.15.116:0/1674500320 wait complete. 2024-03-24T09:22:07.587 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:22:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 -- 172.21.15.116:0/2278071682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc1021c0 msgr2=0x7f38fc108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 --2- 172.21.15.116:0/2278071682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc1021c0 0x7f38fc108a50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f38f4009ca0 tx=0x7f38f402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 -- 172.21.15.116:0/2278071682 shutdown_connections 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 --2- 172.21.15.116:0/2278071682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc1021c0 0x7f38fc108a50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 --2- 172.21.15.116:0/2278071682 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38fc101800 0x7f38fc101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 --2- 172.21.15.116:0/2278071682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc100610 0x7f38fc100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 -- 172.21.15.116:0/2278071682 >> 172.21.15.116:0/2278071682 conn(0x7f38fc0fbba0 msgr2=0x7f38fc0fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 -- 172.21.15.116:0/2278071682 shutdown_connections 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.700+0000 7f3900a15640 1 -- 172.21.15.116:0/2278071682 wait complete. 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 Processor -- start 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 -- start start 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 0x7f38fc197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38fc101800 0x7f38fc197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc1021c0 0x7f38fc19ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc19f1f0 con 0x7f38fc100610 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc19f360 con 0x7f38fc1021c0 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f3900a15640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f38fc19f520 con 0x7f38fc101800 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 0x7f38fc197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 0x7f38fc197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55770/0 (socket says 172.21.15.116:55770) 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 -- 172.21.15.116:0/4138010829 learned_addr learned my addr 172.21.15.116:0/4138010829 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38f9d9b640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38fc101800 0x7f38fc197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fad9d640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc1021c0 0x7f38fc19ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 -- 172.21.15.116:0/4138010829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38fc101800 msgr2=0x7f38fc197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38fc101800 0x7f38fc197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 -- 172.21.15.116:0/4138010829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc1021c0 msgr2=0x7f38fc19ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.701+0000 7f38fa59c640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc1021c0 0x7f38fc19ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f38fa59c640 1 -- 172.21.15.116:0/4138010829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f4009900 con 0x7f38fc100610 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f38fa59c640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 0x7f38fc197700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00e9c0 tx=0x7f38ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec009700 con 0x7f38fc100610 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38fc19f800 con 0x7f38fc100610 2024-03-24T09:22:07.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38ec0044d0 con 0x7f38fc100610 2024-03-24T09:22:07.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec010430 con 0x7f38fc100610 2024-03-24T09:22:07.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.702+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38fc19fd60 con 0x7f38fc100610 2024-03-24T09:22:07.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.703+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38fc10bea0 con 0x7f38fc100610 2024-03-24T09:22:07.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.704+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7f38ec003f10 con 0x7f38fc100610 2024-03-24T09:22:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.704+0000 7f38eb7fe640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38e40a1ec0 0x7f38e40a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.705+0000 7f38f9d9b640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38e40a1ec0 0x7f38e40a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.705+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f38ec014070 con 0x7f38fc100610 2024-03-24T09:22:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.705+0000 7f38f9d9b640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38e40a1ec0 0x7f38e40a4380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f38fc198c10 tx=0x7f38f000b410 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:07.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.713+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f38ec096f20 con 0x7f38fc100610 2024-03-24T09:22:07.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.894+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38fc19ffe0 con 0x7f38fc100610 2024-03-24T09:22:07.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.895+0000 7f38eb7fe640 1 -- 172.21.15.116:0/4138010829 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f38ec09a5e0 con 0x7f38fc100610 2024-03-24T09:22:07.895 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:22:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38e40a1ec0 msgr2=0x7f38e40a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38e40a1ec0 0x7f38e40a4380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f38fc198c10 tx=0x7f38f000b410 comp rx=0 tx=0).stop 2024-03-24T09:22:07.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 msgr2=0x7f38fc197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 0x7f38fc197700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f38ec00e9c0 tx=0x7f38ec00ee80 comp rx=0 tx=0).stop 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 shutdown_connections 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38fc1021c0 0x7f38fc19ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38e40a1ec0 0x7f38e40a4380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f38fc101800 0x7f38fc197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 --2- 172.21.15.116:0/4138010829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38fc100610 0x7f38fc197700 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 >> 172.21.15.116:0/4138010829 conn(0x7f38fc0fbba0 msgr2=0x7f38fc0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 shutdown_connections 2024-03-24T09:22:07.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:07.897+0000 7f3900a15640 1 -- 172.21.15.116:0/4138010829 wait complete. 2024-03-24T09:22:07.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 -- 172.21.15.116:0/312890408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb41021c0 msgr2=0x7fdbb4108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 --2- 172.21.15.116:0/312890408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb41021c0 0x7fdbb4108a50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009ca0 tx=0x7fdbac02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 -- 172.21.15.116:0/312890408 shutdown_connections 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 --2- 172.21.15.116:0/312890408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb41021c0 0x7fdbb4108a50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 --2- 172.21.15.116:0/312890408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 0x7fdbb4101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 --2- 172.21.15.116:0/312890408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb4100610 0x7fdbb4100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.016+0000 7fdbb913e640 1 -- 172.21.15.116:0/312890408 >> 172.21.15.116:0/312890408 conn(0x7fdbb40fbb80 msgr2=0x7fdbb40fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.017+0000 7fdbb913e640 1 -- 172.21.15.116:0/312890408 shutdown_connections 2024-03-24T09:22:08.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.017+0000 7fdbb913e640 1 -- 172.21.15.116:0/312890408 wait complete. 2024-03-24T09:22:08.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.017+0000 7fdbb913e640 1 Processor -- start 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.017+0000 7fdbb913e640 1 -- start start 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 0x7fdbb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 0x7fdbb419c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb41021c0 0x7fdbb419ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb419f1e0 con 0x7fdbb4100610 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb419f350 con 0x7fdbb4101800 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb419f510 con 0x7fdbb41021c0 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 0x7fdbb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 0x7fdbb40693e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55792/0 (socket says 172.21.15.116:55792) 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 -- 172.21.15.116:0/2131425162 learned_addr learned my addr 172.21.15.116:0/2131425162 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb259c640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 0x7fdbb419c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb359e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb41021c0 0x7fdbb419ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 -- 172.21.15.116:0/2131425162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb41021c0 msgr2=0x7fdbb419ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb41021c0 0x7fdbb419ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 -- 172.21.15.116:0/2131425162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 msgr2=0x7fdbb419c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 0x7fdbb419c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 -- 172.21.15.116:0/2131425162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac009900 con 0x7fdbb4100610 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb259c640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 0x7fdbb419c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb2d9d640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 0x7fdbb40693e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdba800e9c0 tx=0x7fdba800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba80096e0 con 0x7fdbb4100610 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdba80044e0 con 0x7fdbb4100610 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba8010430 con 0x7fdbb4100610 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb419f790 con 0x7fdbb4100610 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.018+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb419fc50 con 0x7fdbb4100610 2024-03-24T09:22:08.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.019+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbb4107a00 con 0x7fdbb4100610 2024-03-24T09:22:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.021+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 22) ==== 136568+0+0 (secure 0 0 0) 0x7fdba8003f60 con 0x7fdbb4100610 2024-03-24T09:22:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.021+0000 7fdb9bfff640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdb9c0a1ec0 0x7fdb9c0a4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.021+0000 7fdbb259c640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdb9c0a1ec0 0x7fdb9c0a4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.021+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdba8014070 con 0x7fdbb4100610 2024-03-24T09:22:08.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.021+0000 7fdbb259c640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdb9c0a1ec0 0x7fdb9c0a4380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4069150 tx=0x7fdba4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.025+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fdba8096f90 con 0x7fdbb4100610 2024-03-24T09:22:08.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.211+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdbb402d160 con 0x7fdbb4100610 2024-03-24T09:22:08.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.211+0000 7fdb9bfff640 1 -- 172.21.15.116:0/2131425162 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fdba809a650 con 0x7fdbb4100610 2024-03-24T09:22:08.212 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdb9c0a1ec0 msgr2=0x7fdb9c0a4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdb9c0a1ec0 0x7fdb9c0a4380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4069150 tx=0x7fdba4005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 msgr2=0x7fdbb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 0x7fdbb40693e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fdba800e9c0 tx=0x7fdba800ee80 comp rx=0 tx=0).stop 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 shutdown_connections 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdb9c0a1ec0 0x7fdb9c0a4380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbb41021c0 0x7fdbb419ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdbb4101800 0x7fdbb419c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 --2- 172.21.15.116:0/2131425162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdbb4100610 0x7fdbb40693e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.213+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 >> 172.21.15.116:0/2131425162 conn(0x7fdbb40fbb80 msgr2=0x7fdbb4104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.214+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 shutdown_connections 2024-03-24T09:22:08.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.214+0000 7fdbb913e640 1 -- 172.21.15.116:0/2131425162 wait complete. 2024-03-24T09:22:08.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 -- 172.21.15.116:0/1137553949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3081021c0 msgr2=0x7fa308108a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 --2- 172.21.15.116:0/1137553949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3081021c0 0x7fa308108a50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa2f8009900 tx=0x7fa2f802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 -- 172.21.15.116:0/1137553949 shutdown_connections 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 --2- 172.21.15.116:0/1137553949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3081021c0 0x7fa308108a50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 --2- 172.21.15.116:0/1137553949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308101800 0x7fa308101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 --2- 172.21.15.116:0/1137553949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa308100610 0x7fa308100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 -- 172.21.15.116:0/1137553949 >> 172.21.15.116:0/1137553949 conn(0x7fa3080fbba0 msgr2=0x7fa3080fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 -- 172.21.15.116:0/1137553949 shutdown_connections 2024-03-24T09:22:08.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.335+0000 7fa30de57640 1 -- 172.21.15.116:0/1137553949 wait complete. 2024-03-24T09:22:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 Processor -- start 2024-03-24T09:22:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 -- start start 2024-03-24T09:22:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308100610 0x7fa3081976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 0x7fa308197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3081021c0 0x7fa30819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa30819f1d0 con 0x7fa308101800 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa30819f340 con 0x7fa3081021c0 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa30de57640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa30819f500 con 0x7fa308100610 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa306ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 0x7fa308197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa306ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 0x7fa308197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55804/0 (socket says 172.21.15.116:55804) 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa306ffd640 1 -- 172.21.15.116:0/3196231482 learned_addr learned my addr 172.21.15.116:0/3196231482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa307fff640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3081021c0 0x7fa30819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.336+0000 7fa3077fe640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308100610 0x7fa3081976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa306ffd640 1 -- 172.21.15.116:0/3196231482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308100610 msgr2=0x7fa3081976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa306ffd640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308100610 0x7fa3081976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa306ffd640 1 -- 172.21.15.116:0/3196231482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3081021c0 msgr2=0x7fa30819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa306ffd640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3081021c0 0x7fa30819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa306ffd640 1 -- 172.21.15.116:0/3196231482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f8009560 con 0x7fa308101800 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa306ffd640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 0x7fa308197c20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000e9f0 tx=0x7fa2f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0009720 con 0x7fa308101800 2024-03-24T09:22:08.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30819f7e0 con 0x7fa308101800 2024-03-24T09:22:08.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa2f000cd80 con 0x7fa308101800 2024-03-24T09:22:08.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0010430 con 0x7fa308101800 2024-03-24T09:22:08.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.337+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30819fca0 con 0x7fa308101800 2024-03-24T09:22:08.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.338+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa30810bed0 con 0x7fa308101800 2024-03-24T09:22:08.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.339+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136877+0+0 (secure 0 0 0) 0x7fa2f0005020 con 0x7fa308101800 2024-03-24T09:22:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.340+0000 7fa304ff9640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa2f40a2230 0x7fa2f40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.340+0000 7fa3077fe640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa2f40a2230 0x7fa2f40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.340+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa2f0014070 con 0x7fa308101800 2024-03-24T09:22:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.340+0000 7fa3077fe640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa2f40a2230 0x7fa2f40a46f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa308101660 tx=0x7fa300009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.344+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fa2f0097070 con 0x7fa308101800 2024-03-24T09:22:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/525262438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-24T09:22:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1674500320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:08 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4138010829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.524+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa30819ff20 con 0x7fa308101800 2024-03-24T09:22:08.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.525+0000 7fa304ff9640 1 -- 172.21.15.116:0/3196231482 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fa2f009a730 con 0x7fa308101800 2024-03-24T09:22:08.526 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:22:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa2f40a2230 msgr2=0x7fa2f40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa2f40a2230 0x7fa2f40a46f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa308101660 tx=0x7fa300009500 comp rx=0 tx=0).stop 2024-03-24T09:22:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 msgr2=0x7fa308197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 0x7fa308197c20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa2f000e9f0 tx=0x7fa2f000edf0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 shutdown_connections 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3081021c0 0x7fa30819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa308101800 0x7fa308197c20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa2f40a2230 0x7fa2f40a46f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 --2- 172.21.15.116:0/3196231482 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa308100610 0x7fa3081976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 >> 172.21.15.116:0/3196231482 conn(0x7fa3080fbba0 msgr2=0x7fa3080fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 shutdown_connections 2024-03-24T09:22:08.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.527+0000 7fa30de57640 1 -- 172.21.15.116:0/3196231482 wait complete. 2024-03-24T09:22:08.537 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/525262438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:08.537 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:08.537 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-24T09:22:08.537 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1674500320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.538 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:08 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4138010829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:08.539 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-24T09:22:08.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:22:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/525262438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: mgrmap e22: x(active, since 2s), standbys: z, y 2024-03-24T09:22:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1674500320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:08.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:08.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:08 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4138010829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- 172.21.15.116:0/225449961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1021e0 msgr2=0x7f7aec108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- 172.21.15.116:0/225449961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1021e0 0x7f7aec108a70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009ca0 tx=0x7f7ae402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- 172.21.15.116:0/225449961 shutdown_connections 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- 172.21.15.116:0/225449961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1021e0 0x7f7aec108a70 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- 172.21.15.116:0/225449961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec101820 0x7f7aec101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- 172.21.15.116:0/225449961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aec100630 0x7f7aec100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- 172.21.15.116:0/225449961 >> 172.21.15.116:0/225449961 conn(0x7f7aec0fbba0 msgr2=0x7f7aec0fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- 172.21.15.116:0/225449961 shutdown_connections 2024-03-24T09:22:08.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- 172.21.15.116:0/225449961 wait complete. 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 Processor -- start 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- start start 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aec100630 0x7f7aec069400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 0x7f7aec19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec1021e0 0x7f7aec1a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec1a13c0 con 0x7f7aec101820 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec1a1530 con 0x7f7aec1021e0 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.644+0000 7f7af382e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec1a16f0 con 0x7f7aec100630 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 0x7f7aec19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 0x7f7aec19e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55820/0 (socket says 172.21.15.116:55820) 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af1dcb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec1021e0 0x7f7aec1a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 -- 172.21.15.116:0/4099658033 learned_addr learned my addr 172.21.15.116:0/4099658033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:08.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af15ca640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aec100630 0x7f7aec069400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 -- 172.21.15.116:0/4099658033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aec100630 msgr2=0x7f7aec069400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aec100630 0x7f7aec069400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 -- 172.21.15.116:0/4099658033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec1021e0 msgr2=0x7f7aec1a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec1021e0 0x7f7aec1a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 -- 172.21.15.116:0/4099658033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae4009900 con 0x7f7aec101820 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af1dcb640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec1021e0 0x7f7aec1a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af0dc9640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 0x7f7aec19e7d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00e9f0 tx=0x7f7adc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc009700 con 0x7f7aec101820 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7adc00cd80 con 0x7f7aec101820 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aec1a19d0 con 0x7f7aec101820 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc018840 con 0x7f7aec101820 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.645+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aec1a1ee0 con 0x7f7aec101820 2024-03-24T09:22:08.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.646+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aec1079d0 con 0x7f7aec101820 2024-03-24T09:22:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.647+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136877+0+0 (secure 0 0 0) 0x7f7adc0189a0 con 0x7f7aec101820 2024-03-24T09:22:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.648+0000 7f7ae27fc640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7ad80a2230 0x7f7ad80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.648+0000 7f7af15ca640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7ad80a2230 0x7f7ad80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.648+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7adc014070 con 0x7f7aec101820 2024-03-24T09:22:08.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.648+0000 7f7af15ca640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7ad80a2230 0x7f7ad80a46f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005fd0 tx=0x7f7ae800b410 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.651+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7adc0974c0 con 0x7f7aec101820 2024-03-24T09:22:08.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.867+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7aec10c110 con 0x7f7aec101820 2024-03-24T09:22:08.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.867+0000 7f7ae27fc640 1 -- 172.21.15.116:0/4099658033 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f7adc09ab80 con 0x7f7aec101820 2024-03-24T09:22:08.868 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:08.868 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:22:05.111617+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24422,24443,24451,24484,24515]},"id":1}]} 2024-03-24T09:22:08.868 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-24T09:22:08.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7ad80a2230 msgr2=0x7f7ad80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7ad80a2230 0x7f7ad80a46f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8005fd0 tx=0x7f7ae800b410 comp rx=0 tx=0).stop 2024-03-24T09:22:08.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 msgr2=0x7f7aec19e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 0x7f7aec19e7d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7adc00e9f0 tx=0x7f7adc00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 shutdown_connections 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7aec1021e0 0x7f7aec1a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101820 0x7f7aec19e7d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7ad80a2230 0x7f7ad80a46f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 --2- 172.21.15.116:0/4099658033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7aec100630 0x7f7aec069400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.869+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 >> 172.21.15.116:0/4099658033 conn(0x7f7aec0fbba0 msgr2=0x7f7aec104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.870+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 shutdown_connections 2024-03-24T09:22:08.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.870+0000 7f7af382e640 1 -- 172.21.15.116:0/4099658033 wait complete. 2024-03-24T09:22:08.881 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-24T09:22:08.881 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-24T09:22:08.881 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T09:22:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 -- 172.21.15.116:0/3876849506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c102930 msgr2=0x7fe93c102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 --2- 172.21.15.116:0/3876849506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c102930 0x7fe93c102db0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fe930009ca0 tx=0x7fe93002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 -- 172.21.15.116:0/3876849506 shutdown_connections 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 --2- 172.21.15.116:0/3876849506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe93c1032f0 0x7fe93c109b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 --2- 172.21.15.116:0/3876849506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c102930 0x7fe93c102db0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 --2- 172.21.15.116:0/3876849506 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe93c101740 0x7fe93c101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 -- 172.21.15.116:0/3876849506 >> 172.21.15.116:0/3876849506 conn(0x7fe93c0fccb0 msgr2=0x7fe93c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 -- 172.21.15.116:0/3876849506 shutdown_connections 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 -- 172.21.15.116:0/3876849506 wait complete. 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 Processor -- start 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.984+0000 7fe944381640 1 -- start start 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 0x7fe93c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe93c102930 0x7fe93c197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe93c1032f0 0x7fe93c19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c19f200 con 0x7fe93c101740 2024-03-24T09:22:08.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c19f370 con 0x7fe93c1032f0 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c19f530 con 0x7fe93c102930 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 0x7fe93c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 0x7fe93c197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55848/0 (socket says 172.21.15.116:55848) 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 -- 172.21.15.116:0/33332246 learned_addr learned my addr 172.21.15.116:0/33332246 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94291e640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe93c1032f0 0x7fe93c19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94191c640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe93c102930 0x7fe93c197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 -- 172.21.15.116:0/33332246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe93c102930 msgr2=0x7fe93c197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe93c102930 0x7fe93c197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 -- 172.21.15.116:0/33332246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe93c1032f0 msgr2=0x7fe93c19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe93c1032f0 0x7fe93c19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 -- 172.21.15.116:0/33332246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe930009900 con 0x7fe93c101740 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe94211d640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 0x7fe93c197710 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe93400e9c0 tx=0x7fe93400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9340096e0 con 0x7fe93c101740 2024-03-24T09:22:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9340044e0 con 0x7fe93c101740 2024-03-24T09:22:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93c19f810 con 0x7fe93c101740 2024-03-24T09:22:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe934010430 con 0x7fe93c101740 2024-03-24T09:22:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.985+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe93c19fca0 con 0x7fe93c101740 2024-03-24T09:22:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.986+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe93c108b80 con 0x7fe93c101740 2024-03-24T09:22:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.987+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136877+0+0 (secure 0 0 0) 0x7fe934003f80 con 0x7fe93c101740 2024-03-24T09:22:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.988+0000 7fe92f7fe640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9280a2230 0x7fe9280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.988+0000 7fe94191c640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9280a2230 0x7fe9280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.988+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe934014070 con 0x7fe93c101740 2024-03-24T09:22:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.988+0000 7fe94191c640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9280a2230 0x7fe9280a46f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe930006010 tx=0x7fe9300325d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:08.992+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe9340970b0 con 0x7fe93c101740 2024-03-24T09:22:09.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.226+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe93c02d240 con 0x7fe9280a2230 2024-03-24T09:22:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.237+0000 7fe92f7fe640 1 -- 172.21.15.116:0/33332246 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339740 (secure 0 0 0) 0x7fe93c02d240 con 0x7fe9280a2230 2024-03-24T09:22:09.241 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:09.243 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:22:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9280a2230 msgr2=0x7fe9280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9280a2230 0x7fe9280a46f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe930006010 tx=0x7fe9300325d0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 msgr2=0x7fe93c197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 0x7fe93c197710 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe93400e9c0 tx=0x7fe93400ee80 comp rx=0 tx=0).stop 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 shutdown_connections 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe93c1032f0 0x7fe93c19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9280a2230 0x7fe9280a46f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe93c102930 0x7fe93c197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 --2- 172.21.15.116:0/33332246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe93c101740 0x7fe93c197710 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 >> 172.21.15.116:0/33332246 conn(0x7fe93c0fccb0 msgr2=0x7fe93c105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 shutdown_connections 2024-03-24T09:22:09.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.241+0000 7fe944381640 1 -- 172.21.15.116:0/33332246 wait complete. 2024-03-24T09:22:09.253 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-03-24T09:22:07.274174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195243,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1488,"num_read_kb":914,"num_write":563,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":948,"ondisk_log_size":948,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378676,"kb_used_data":12836,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563148,"statfs":{"total":1151940427776,"available":1151552663552,"internally_reserved":0,"allocated":13144064,"data_stored":9769911,"data_compressed":534294,"data_compressed_allocated":724992,"data_compressed_original":4194304,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.987430"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430660+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:22:05.430660+0000","last_peered":"2024-03-24T09:22:05.430660+0000","last_clean":"2024-03-24T09:22:05.430660+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:22:05.430660+0000","last_undegraded":"2024-03-24T09:22:05.430660+0000","last_fullsized":"2024-03-24T09:22:05.430660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417006+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:22:05.417006+0000","last_peered":"2024-03-24T09:22:05.417006+0000","last_clean":"2024-03-24T09:22:05.417006+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:22:05.417006+0000","last_undegraded":"2024-03-24T09:22:05.417006+0000","last_fullsized":"2024-03-24T09:22:05.417006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416543+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:22:05.416543+0000","last_peered":"2024-03-24T09:22:05.416543+0000","last_clean":"2024-03-24T09:22:05.416543+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:22:05.416543+0000","last_undegraded":"2024-03-24T09:22:05.416543+0000","last_fullsized":"2024-03-24T09:22:05.416543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.264004+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:22:05.264004+0000","last_peered":"2024-03-24T09:22:05.264004+0000","last_clean":"2024-03-24T09:22:05.264004+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:22:05.264004+0000","last_undegraded":"2024-03-24T09:22:05.264004+0000","last_fullsized":"2024-03-24T09:22:05.264004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383474+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:22:05.383474+0000","last_peered":"2024-03-24T09:22:05.383474+0000","last_clean":"2024-03-24T09:22:05.383474+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:22:05.383474+0000","last_undegraded":"2024-03-24T09:22:05.383474+0000","last_fullsized":"2024-03-24T09:22:05.383474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395243+0000","last_change":"2024-03-24T09:20:35.029608+0000","last_active":"2024-03-24T09:22:05.395243+0000","last_peered":"2024-03-24T09:22:05.395243+0000","last_clean":"2024-03-24T09:22:05.395243+0000","last_became_active":"2024-03-24T09:20:35.029268+0000","last_became_peered":"2024-03-24T09:20:35.029268+0000","last_unstale":"2024-03-24T09:22:05.395243+0000","last_undegraded":"2024-03-24T09:22:05.395243+0000","last_fullsized":"2024-03-24T09:22:05.395243+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:47.087125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430275+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:22:05.430275+0000","last_peered":"2024-03-24T09:22:05.430275+0000","last_clean":"2024-03-24T09:22:05.430275+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:22:05.430275+0000","last_undegraded":"2024-03-24T09:22:05.430275+0000","last_fullsized":"2024-03-24T09:22:05.430275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261421+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:22:05.261421+0000","last_peered":"2024-03-24T09:22:05.261421+0000","last_clean":"2024-03-24T09:22:05.261421+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:22:05.261421+0000","last_undegraded":"2024-03-24T09:22:05.261421+0000","last_fullsized":"2024-03-24T09:22:05.261421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431980+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:22:05.431980+0000","last_peered":"2024-03-24T09:22:05.431980+0000","last_clean":"2024-03-24T09:22:05.431980+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:22:05.431980+0000","last_undegraded":"2024-03-24T09:22:05.431980+0000","last_fullsized":"2024-03-24T09:22:05.431980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395656+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:22:05.395656+0000","last_peered":"2024-03-24T09:22:05.395656+0000","last_clean":"2024-03-24T09:22:05.395656+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:22:05.395656+0000","last_undegraded":"2024-03-24T09:22:05.395656+0000","last_fullsized":"2024-03-24T09:22:05.395656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431628+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:22:05.431628+0000","last_peered":"2024-03-24T09:22:05.431628+0000","last_clean":"2024-03-24T09:22:05.431628+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:22:05.431628+0000","last_undegraded":"2024-03-24T09:22:05.431628+0000","last_fullsized":"2024-03-24T09:22:05.431628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263913+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:22:05.263913+0000","last_peered":"2024-03-24T09:22:05.263913+0000","last_clean":"2024-03-24T09:22:05.263913+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:22:05.263913+0000","last_undegraded":"2024-03-24T09:22:05.263913+0000","last_fullsized":"2024-03-24T09:22:05.263913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430584+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:22:05.430584+0000","last_peered":"2024-03-24T09:22:05.430584+0000","last_clean":"2024-03-24T09:22:05.430584+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:22:05.430584+0000","last_undegraded":"2024-03-24T09:22:05.430584+0000","last_fullsized":"2024-03-24T09:22:05.430584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416250+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:22:05.416250+0000","last_peered":"2024-03-24T09:22:05.416250+0000","last_clean":"2024-03-24T09:22:05.416250+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:22:05.416250+0000","last_undegraded":"2024-03-24T09:22:05.416250+0000","last_fullsized":"2024-03-24T09:22:05.416250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416249+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:22:05.416249+0000","last_peered":"2024-03-24T09:22:05.416249+0000","last_clean":"2024-03-24T09:22:05.416249+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:22:05.416249+0000","last_undegraded":"2024-03-24T09:22:05.416249+0000","last_fullsized":"2024-03-24T09:22:05.416249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429842+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:22:05.429842+0000","last_peered":"2024-03-24T09:22:05.429842+0000","last_clean":"2024-03-24T09:22:05.429842+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:22:05.429842+0000","last_undegraded":"2024-03-24T09:22:05.429842+0000","last_fullsized":"2024-03-24T09:22:05.429842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430550+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:22:05.430550+0000","last_peered":"2024-03-24T09:22:05.430550+0000","last_clean":"2024-03-24T09:22:05.430550+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:22:05.430550+0000","last_undegraded":"2024-03-24T09:22:05.430550+0000","last_fullsized":"2024-03-24T09:22:05.430550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381311+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:22:05.381311+0000","last_peered":"2024-03-24T09:22:05.381311+0000","last_clean":"2024-03-24T09:22:05.381311+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:22:05.381311+0000","last_undegraded":"2024-03-24T09:22:05.381311+0000","last_fullsized":"2024-03-24T09:22:05.381311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265648+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:22:05.265648+0000","last_peered":"2024-03-24T09:22:05.265648+0000","last_clean":"2024-03-24T09:22:05.265648+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:22:05.265648+0000","last_undegraded":"2024-03-24T09:22:05.265648+0000","last_fullsized":"2024-03-24T09:22:05.265648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265189+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:22:05.265189+0000","last_peered":"2024-03-24T09:22:05.265189+0000","last_clean":"2024-03-24T09:22:05.265189+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:22:05.265189+0000","last_undegraded":"2024-03-24T09:22:05.265189+0000","last_fullsized":"2024-03-24T09:22:05.265189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381586+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:22:05.381586+0000","last_peered":"2024-03-24T09:22:05.381586+0000","last_clean":"2024-03-24T09:22:05.381586+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:22:05.381586+0000","last_undegraded":"2024-03-24T09:22:05.381586+0000","last_fullsized":"2024-03-24T09:22:05.381586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262126+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:22:05.262126+0000","last_peered":"2024-03-24T09:22:05.262126+0000","last_clean":"2024-03-24T09:22:05.262126+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:22:05.262126+0000","last_undegraded":"2024-03-24T09:22:05.262126+0000","last_fullsized":"2024-03-24T09:22:05.262126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262833+0000","last_change":"2024-03-24T09:20:35.302025+0000","last_active":"2024-03-24T09:22:05.262833+0000","last_peered":"2024-03-24T09:22:05.262833+0000","last_clean":"2024-03-24T09:22:05.262833+0000","last_became_active":"2024-03-24T09:20:35.301575+0000","last_became_peered":"2024-03-24T09:20:35.301575+0000","last_unstale":"2024-03-24T09:22:05.262833+0000","last_undegraded":"2024-03-24T09:22:05.262833+0000","last_fullsized":"2024-03-24T09:22:05.262833+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:20.305175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262343+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:22:05.262343+0000","last_peered":"2024-03-24T09:22:05.262343+0000","last_clean":"2024-03-24T09:22:05.262343+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:22:05.262343+0000","last_undegraded":"2024-03-24T09:22:05.262343+0000","last_fullsized":"2024-03-24T09:22:05.262343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417089+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:22:05.417089+0000","last_peered":"2024-03-24T09:22:05.417089+0000","last_clean":"2024-03-24T09:22:05.417089+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:22:05.417089+0000","last_undegraded":"2024-03-24T09:22:05.417089+0000","last_fullsized":"2024-03-24T09:22:05.417089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263181+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:22:05.263181+0000","last_peered":"2024-03-24T09:22:05.263181+0000","last_clean":"2024-03-24T09:22:05.263181+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:22:05.263181+0000","last_undegraded":"2024-03-24T09:22:05.263181+0000","last_fullsized":"2024-03-24T09:22:05.263181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384209+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:22:05.384209+0000","last_peered":"2024-03-24T09:22:05.384209+0000","last_clean":"2024-03-24T09:22:05.384209+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:22:05.384209+0000","last_undegraded":"2024-03-24T09:22:05.384209+0000","last_fullsized":"2024-03-24T09:22:05.384209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430840+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:22:05.430840+0000","last_peered":"2024-03-24T09:22:05.430840+0000","last_clean":"2024-03-24T09:22:05.430840+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:22:05.430840+0000","last_undegraded":"2024-03-24T09:22:05.430840+0000","last_fullsized":"2024-03-24T09:22:05.430840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263015+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:22:05.263015+0000","last_peered":"2024-03-24T09:22:05.263015+0000","last_clean":"2024-03-24T09:22:05.263015+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:22:05.263015+0000","last_undegraded":"2024-03-24T09:22:05.263015+0000","last_fullsized":"2024-03-24T09:22:05.263015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383756+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:22:05.383756+0000","last_peered":"2024-03-24T09:22:05.383756+0000","last_clean":"2024-03-24T09:22:05.383756+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:22:05.383756+0000","last_undegraded":"2024-03-24T09:22:05.383756+0000","last_fullsized":"2024-03-24T09:22:05.383756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391949+0000","last_change":"2024-03-24T09:20:35.032271+0000","last_active":"2024-03-24T09:22:05.391949+0000","last_peered":"2024-03-24T09:22:05.391949+0000","last_clean":"2024-03-24T09:22:05.391949+0000","last_became_active":"2024-03-24T09:20:35.031913+0000","last_became_peered":"2024-03-24T09:20:35.031913+0000","last_unstale":"2024-03-24T09:22:05.391949+0000","last_undegraded":"2024-03-24T09:22:05.391949+0000","last_fullsized":"2024-03-24T09:22:05.391949+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:40.322197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265519+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:22:05.265519+0000","last_peered":"2024-03-24T09:22:05.265519+0000","last_clean":"2024-03-24T09:22:05.265519+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:22:05.265519+0000","last_undegraded":"2024-03-24T09:22:05.265519+0000","last_fullsized":"2024-03-24T09:22:05.265519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392028+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:22:05.392028+0000","last_peered":"2024-03-24T09:22:05.392028+0000","last_clean":"2024-03-24T09:22:05.392028+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:22:05.392028+0000","last_undegraded":"2024-03-24T09:22:05.392028+0000","last_fullsized":"2024-03-24T09:22:05.392028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383732+0000","last_change":"2024-03-24T09:20:35.031176+0000","last_active":"2024-03-24T09:22:05.383732+0000","last_peered":"2024-03-24T09:22:05.383732+0000","last_clean":"2024-03-24T09:22:05.383732+0000","last_became_active":"2024-03-24T09:20:35.030888+0000","last_became_peered":"2024-03-24T09:20:35.030888+0000","last_unstale":"2024-03-24T09:22:05.383732+0000","last_undegraded":"2024-03-24T09:22:05.383732+0000","last_fullsized":"2024-03-24T09:22:05.383732+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:43.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265449+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:22:05.265449+0000","last_peered":"2024-03-24T09:22:05.265449+0000","last_clean":"2024-03-24T09:22:05.265449+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:22:05.265449+0000","last_undegraded":"2024-03-24T09:22:05.265449+0000","last_fullsized":"2024-03-24T09:22:05.265449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395214+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:22:05.395214+0000","last_peered":"2024-03-24T09:22:05.395214+0000","last_clean":"2024-03-24T09:22:05.395214+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:22:05.395214+0000","last_undegraded":"2024-03-24T09:22:05.395214+0000","last_fullsized":"2024-03-24T09:22:05.395214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391824+0000","last_change":"2024-03-24T09:20:35.301944+0000","last_active":"2024-03-24T09:22:05.391824+0000","last_peered":"2024-03-24T09:22:05.391824+0000","last_clean":"2024-03-24T09:22:05.391824+0000","last_became_active":"2024-03-24T09:20:35.301588+0000","last_became_peered":"2024-03-24T09:20:35.301588+0000","last_unstale":"2024-03-24T09:22:05.391824+0000","last_undegraded":"2024-03-24T09:22:05.391824+0000","last_fullsized":"2024-03-24T09:22:05.391824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:33.185748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430406+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:22:05.430406+0000","last_peered":"2024-03-24T09:22:05.430406+0000","last_clean":"2024-03-24T09:22:05.430406+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:22:05.430406+0000","last_undegraded":"2024-03-24T09:22:05.430406+0000","last_fullsized":"2024-03-24T09:22:05.430406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430255+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:22:05.430255+0000","last_peered":"2024-03-24T09:22:05.430255+0000","last_clean":"2024-03-24T09:22:05.430255+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:22:05.430255+0000","last_undegraded":"2024-03-24T09:22:05.430255+0000","last_fullsized":"2024-03-24T09:22:05.430255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261576+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:22:05.261576+0000","last_peered":"2024-03-24T09:22:05.261576+0000","last_clean":"2024-03-24T09:22:05.261576+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:22:05.261576+0000","last_undegraded":"2024-03-24T09:22:05.261576+0000","last_fullsized":"2024-03-24T09:22:05.261576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416766+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:22:05.416766+0000","last_peered":"2024-03-24T09:22:05.416766+0000","last_clean":"2024-03-24T09:22:05.416766+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:22:05.416766+0000","last_undegraded":"2024-03-24T09:22:05.416766+0000","last_fullsized":"2024-03-24T09:22:05.416766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392042+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:22:05.392042+0000","last_peered":"2024-03-24T09:22:05.392042+0000","last_clean":"2024-03-24T09:22:05.392042+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:22:05.392042+0000","last_undegraded":"2024-03-24T09:22:05.392042+0000","last_fullsized":"2024-03-24T09:22:05.392042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380573+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:22:05.380573+0000","last_peered":"2024-03-24T09:22:05.380573+0000","last_clean":"2024-03-24T09:22:05.380573+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:22:05.380573+0000","last_undegraded":"2024-03-24T09:22:05.380573+0000","last_fullsized":"2024-03-24T09:22:05.380573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261827+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:22:05.261827+0000","last_peered":"2024-03-24T09:22:05.261827+0000","last_clean":"2024-03-24T09:22:05.261827+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:22:05.261827+0000","last_undegraded":"2024-03-24T09:22:05.261827+0000","last_fullsized":"2024-03-24T09:22:05.261827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":284,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.075787+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:22:06.075787+0000","last_peered":"2024-03-24T09:22:06.075787+0000","last_clean":"2024-03-24T09:22:06.075787+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:22:06.075787+0000","last_undegraded":"2024-03-24T09:22:06.075787+0000","last_fullsized":"2024-03-24T09:22:06.075787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431671+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:22:05.431671+0000","last_peered":"2024-03-24T09:22:05.431671+0000","last_clean":"2024-03-24T09:22:05.431671+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:22:05.431671+0000","last_undegraded":"2024-03-24T09:22:05.431671+0000","last_fullsized":"2024-03-24T09:22:05.431671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380431+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:22:05.380431+0000","last_peered":"2024-03-24T09:22:05.380431+0000","last_clean":"2024-03-24T09:22:05.380431+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:22:05.380431+0000","last_undegraded":"2024-03-24T09:22:05.380431+0000","last_fullsized":"2024-03-24T09:22:05.380431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431572+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:22:05.431572+0000","last_peered":"2024-03-24T09:22:05.431572+0000","last_clean":"2024-03-24T09:22:05.431572+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:22:05.431572+0000","last_undegraded":"2024-03-24T09:22:05.431572+0000","last_fullsized":"2024-03-24T09:22:05.431572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417192+0000","last_change":"2024-03-24T09:20:35.301994+0000","last_active":"2024-03-24T09:22:05.417192+0000","last_peered":"2024-03-24T09:22:05.417192+0000","last_clean":"2024-03-24T09:22:05.417192+0000","last_became_active":"2024-03-24T09:20:35.301570+0000","last_became_peered":"2024-03-24T09:20:35.301570+0000","last_unstale":"2024-03-24T09:22:05.417192+0000","last_undegraded":"2024-03-24T09:22:05.417192+0000","last_fullsized":"2024-03-24T09:22:05.417192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:34.896090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263428+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:22:05.263428+0000","last_peered":"2024-03-24T09:22:05.263428+0000","last_clean":"2024-03-24T09:22:05.263428+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:22:05.263428+0000","last_undegraded":"2024-03-24T09:22:05.263428+0000","last_fullsized":"2024-03-24T09:22:05.263428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391847+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:22:05.391847+0000","last_peered":"2024-03-24T09:22:05.391847+0000","last_clean":"2024-03-24T09:22:05.391847+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:22:05.391847+0000","last_undegraded":"2024-03-24T09:22:05.391847+0000","last_fullsized":"2024-03-24T09:22:05.391847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380857+0000","last_change":"2024-03-24T09:20:35.027718+0000","last_active":"2024-03-24T09:22:05.380857+0000","last_peered":"2024-03-24T09:22:05.380857+0000","last_clean":"2024-03-24T09:22:05.380857+0000","last_became_active":"2024-03-24T09:20:35.027347+0000","last_became_peered":"2024-03-24T09:20:35.027347+0000","last_unstale":"2024-03-24T09:22:05.380857+0000","last_undegraded":"2024-03-24T09:22:05.380857+0000","last_fullsized":"2024-03-24T09:22:05.380857+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:20.097882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395060+0000","last_change":"2024-03-24T09:20:35.301625+0000","last_active":"2024-03-24T09:22:05.395060+0000","last_peered":"2024-03-24T09:22:05.395060+0000","last_clean":"2024-03-24T09:22:05.395060+0000","last_became_active":"2024-03-24T09:20:35.301035+0000","last_became_peered":"2024-03-24T09:20:35.301035+0000","last_unstale":"2024-03-24T09:22:05.395060+0000","last_undegraded":"2024-03-24T09:22:05.395060+0000","last_fullsized":"2024-03-24T09:22:05.395060+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:09.448556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":333,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.021394+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:22:06.021394+0000","last_peered":"2024-03-24T09:22:06.021394+0000","last_clean":"2024-03-24T09:22:06.021394+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:22:06.021394+0000","last_undegraded":"2024-03-24T09:22:06.021394+0000","last_fullsized":"2024-03-24T09:22:06.021394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69194,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":257,"num_read_kb":196,"num_write":44,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384057+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:22:05.384057+0000","last_peered":"2024-03-24T09:22:05.384057+0000","last_clean":"2024-03-24T09:22:05.384057+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:22:05.384057+0000","last_undegraded":"2024-03-24T09:22:05.384057+0000","last_fullsized":"2024-03-24T09:22:05.384057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392229+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:22:05.392229+0000","last_peered":"2024-03-24T09:22:05.392229+0000","last_clean":"2024-03-24T09:22:05.392229+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:22:05.392229+0000","last_undegraded":"2024-03-24T09:22:05.392229+0000","last_fullsized":"2024-03-24T09:22:05.392229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265368+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:22:05.265368+0000","last_peered":"2024-03-24T09:22:05.265368+0000","last_clean":"2024-03-24T09:22:05.265368+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:22:05.265368+0000","last_undegraded":"2024-03-24T09:22:05.265368+0000","last_fullsized":"2024-03-24T09:22:05.265368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430401+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:22:05.430401+0000","last_peered":"2024-03-24T09:22:05.430401+0000","last_clean":"2024-03-24T09:22:05.430401+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:22:05.430401+0000","last_undegraded":"2024-03-24T09:22:05.430401+0000","last_fullsized":"2024-03-24T09:22:05.430401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429872+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:22:05.429872+0000","last_peered":"2024-03-24T09:22:05.429872+0000","last_clean":"2024-03-24T09:22:05.429872+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:22:05.429872+0000","last_undegraded":"2024-03-24T09:22:05.429872+0000","last_fullsized":"2024-03-24T09:22:05.429872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391701+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:22:05.391701+0000","last_peered":"2024-03-24T09:22:05.391701+0000","last_clean":"2024-03-24T09:22:05.391701+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:22:05.391701+0000","last_undegraded":"2024-03-24T09:22:05.391701+0000","last_fullsized":"2024-03-24T09:22:05.391701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430945+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:22:05.430945+0000","last_peered":"2024-03-24T09:22:05.430945+0000","last_clean":"2024-03-24T09:22:05.430945+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:22:05.430945+0000","last_undegraded":"2024-03-24T09:22:05.430945+0000","last_fullsized":"2024-03-24T09:22:05.430945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381184+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:22:05.381184+0000","last_peered":"2024-03-24T09:22:05.381184+0000","last_clean":"2024-03-24T09:22:05.381184+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:22:05.381184+0000","last_undegraded":"2024-03-24T09:22:05.381184+0000","last_fullsized":"2024-03-24T09:22:05.381184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'10","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.079573+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:22:06.079573+0000","last_peered":"2024-03-24T09:22:06.079573+0000","last_clean":"2024-03-24T09:22:06.079573+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:22:06.079573+0000","last_undegraded":"2024-03-24T09:22:06.079573+0000","last_fullsized":"2024-03-24T09:22:06.079573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430294+0000","last_change":"2024-03-24T09:20:35.031305+0000","last_active":"2024-03-24T09:22:05.430294+0000","last_peered":"2024-03-24T09:22:05.430294+0000","last_clean":"2024-03-24T09:22:05.430294+0000","last_became_active":"2024-03-24T09:20:35.030952+0000","last_became_peered":"2024-03-24T09:20:35.030952+0000","last_unstale":"2024-03-24T09:22:05.430294+0000","last_undegraded":"2024-03-24T09:22:05.430294+0000","last_fullsized":"2024-03-24T09:22:05.430294+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:40.987185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430873+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:22:05.430873+0000","last_peered":"2024-03-24T09:22:05.430873+0000","last_clean":"2024-03-24T09:22:05.430873+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:22:05.430873+0000","last_undegraded":"2024-03-24T09:22:05.430873+0000","last_fullsized":"2024-03-24T09:22:05.430873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383466+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:22:05.383466+0000","last_peered":"2024-03-24T09:22:05.383466+0000","last_clean":"2024-03-24T09:22:05.383466+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:22:05.383466+0000","last_undegraded":"2024-03-24T09:22:05.383466+0000","last_fullsized":"2024-03-24T09:22:05.383466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391686+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:22:05.391686+0000","last_peered":"2024-03-24T09:22:05.391686+0000","last_clean":"2024-03-24T09:22:05.391686+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:22:05.391686+0000","last_undegraded":"2024-03-24T09:22:05.391686+0000","last_fullsized":"2024-03-24T09:22:05.391686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391546+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:22:05.391546+0000","last_peered":"2024-03-24T09:22:05.391546+0000","last_clean":"2024-03-24T09:22:05.391546+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:22:05.391546+0000","last_undegraded":"2024-03-24T09:22:05.391546+0000","last_fullsized":"2024-03-24T09:22:05.391546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384256+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:22:05.384256+0000","last_peered":"2024-03-24T09:22:05.384256+0000","last_clean":"2024-03-24T09:22:05.384256+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:22:05.384256+0000","last_undegraded":"2024-03-24T09:22:05.384256+0000","last_fullsized":"2024-03-24T09:22:05.384256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430172+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:22:05.430172+0000","last_peered":"2024-03-24T09:22:05.430172+0000","last_clean":"2024-03-24T09:22:05.430172+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:22:05.430172+0000","last_undegraded":"2024-03-24T09:22:05.430172+0000","last_fullsized":"2024-03-24T09:22:05.430172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383731+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:22:05.383731+0000","last_peered":"2024-03-24T09:22:05.383731+0000","last_clean":"2024-03-24T09:22:05.383731+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:22:05.383731+0000","last_undegraded":"2024-03-24T09:22:05.383731+0000","last_fullsized":"2024-03-24T09:22:05.383731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380914+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:22:05.380914+0000","last_peered":"2024-03-24T09:22:05.380914+0000","last_clean":"2024-03-24T09:22:05.380914+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:22:05.380914+0000","last_undegraded":"2024-03-24T09:22:05.380914+0000","last_fullsized":"2024-03-24T09:22:05.380914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395507+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:22:05.395507+0000","last_peered":"2024-03-24T09:22:05.395507+0000","last_clean":"2024-03-24T09:22:05.395507+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:22:05.395507+0000","last_undegraded":"2024-03-24T09:22:05.395507+0000","last_fullsized":"2024-03-24T09:22:05.395507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430587+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:22:05.430587+0000","last_peered":"2024-03-24T09:22:05.430587+0000","last_clean":"2024-03-24T09:22:05.430587+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:22:05.430587+0000","last_undegraded":"2024-03-24T09:22:05.430587+0000","last_fullsized":"2024-03-24T09:22:05.430587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430650+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:22:05.430650+0000","last_peered":"2024-03-24T09:22:05.430650+0000","last_clean":"2024-03-24T09:22:05.430650+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:22:05.430650+0000","last_undegraded":"2024-03-24T09:22:05.430650+0000","last_fullsized":"2024-03-24T09:22:05.430650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395163+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:22:05.395163+0000","last_peered":"2024-03-24T09:22:05.395163+0000","last_clean":"2024-03-24T09:22:05.395163+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:22:05.395163+0000","last_undegraded":"2024-03-24T09:22:05.395163+0000","last_fullsized":"2024-03-24T09:22:05.395163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430946+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:22:05.430946+0000","last_peered":"2024-03-24T09:22:05.430946+0000","last_clean":"2024-03-24T09:22:05.430946+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:22:05.430946+0000","last_undegraded":"2024-03-24T09:22:05.430946+0000","last_fullsized":"2024-03-24T09:22:05.430946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392124+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:22:05.392124+0000","last_peered":"2024-03-24T09:22:05.392124+0000","last_clean":"2024-03-24T09:22:05.392124+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:22:05.392124+0000","last_undegraded":"2024-03-24T09:22:05.392124+0000","last_fullsized":"2024-03-24T09:22:05.392124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430122+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:22:05.430122+0000","last_peered":"2024-03-24T09:22:05.430122+0000","last_clean":"2024-03-24T09:22:05.430122+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:22:05.430122+0000","last_undegraded":"2024-03-24T09:22:05.430122+0000","last_fullsized":"2024-03-24T09:22:05.430122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431778+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:22:05.431778+0000","last_peered":"2024-03-24T09:22:05.431778+0000","last_clean":"2024-03-24T09:22:05.431778+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:22:05.431778+0000","last_undegraded":"2024-03-24T09:22:05.431778+0000","last_fullsized":"2024-03-24T09:22:05.431778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261595+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:22:05.261595+0000","last_peered":"2024-03-24T09:22:05.261595+0000","last_clean":"2024-03-24T09:22:05.261595+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:22:05.261595+0000","last_undegraded":"2024-03-24T09:22:05.261595+0000","last_fullsized":"2024-03-24T09:22:05.261595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263319+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:22:05.263319+0000","last_peered":"2024-03-24T09:22:05.263319+0000","last_clean":"2024-03-24T09:22:05.263319+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:22:05.263319+0000","last_undegraded":"2024-03-24T09:22:05.263319+0000","last_fullsized":"2024-03-24T09:22:05.263319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429912+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:22:05.429912+0000","last_peered":"2024-03-24T09:22:05.429912+0000","last_clean":"2024-03-24T09:22:05.429912+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:22:05.429912+0000","last_undegraded":"2024-03-24T09:22:05.429912+0000","last_fullsized":"2024-03-24T09:22:05.429912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392506+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:22:05.392506+0000","last_peered":"2024-03-24T09:22:05.392506+0000","last_clean":"2024-03-24T09:22:05.392506+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:22:05.392506+0000","last_undegraded":"2024-03-24T09:22:05.392506+0000","last_fullsized":"2024-03-24T09:22:05.392506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429921+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:22:05.429921+0000","last_peered":"2024-03-24T09:22:05.429921+0000","last_clean":"2024-03-24T09:22:05.429921+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:22:05.429921+0000","last_undegraded":"2024-03-24T09:22:05.429921+0000","last_fullsized":"2024-03-24T09:22:05.429921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430097+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:22:05.430097+0000","last_peered":"2024-03-24T09:22:05.430097+0000","last_clean":"2024-03-24T09:22:05.430097+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:22:05.430097+0000","last_undegraded":"2024-03-24T09:22:05.430097+0000","last_fullsized":"2024-03-24T09:22:05.430097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429932+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:22:05.429932+0000","last_peered":"2024-03-24T09:22:05.429932+0000","last_clean":"2024-03-24T09:22:05.429932+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:22:05.429932+0000","last_undegraded":"2024-03-24T09:22:05.429932+0000","last_fullsized":"2024-03-24T09:22:05.429932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384084+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:22:05.384084+0000","last_peered":"2024-03-24T09:22:05.384084+0000","last_clean":"2024-03-24T09:22:05.384084+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:22:05.384084+0000","last_undegraded":"2024-03-24T09:22:05.384084+0000","last_fullsized":"2024-03-24T09:22:05.384084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430884+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:22:05.430884+0000","last_peered":"2024-03-24T09:22:05.430884+0000","last_clean":"2024-03-24T09:22:05.430884+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:22:05.430884+0000","last_undegraded":"2024-03-24T09:22:05.430884+0000","last_fullsized":"2024-03-24T09:22:05.430884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261523+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:22:05.261523+0000","last_peered":"2024-03-24T09:22:05.261523+0000","last_clean":"2024-03-24T09:22:05.261523+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:22:05.261523+0000","last_undegraded":"2024-03-24T09:22:05.261523+0000","last_fullsized":"2024-03-24T09:22:05.261523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.514742+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:22:05.514742+0000","last_peered":"2024-03-24T09:22:05.514742+0000","last_clean":"2024-03-24T09:22:05.514742+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:22:05.514742+0000","last_undegraded":"2024-03-24T09:22:05.514742+0000","last_fullsized":"2024-03-24T09:22:05.514742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262000+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:22:05.262000+0000","last_peered":"2024-03-24T09:22:05.262000+0000","last_clean":"2024-03-24T09:22:05.262000+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:22:05.262000+0000","last_undegraded":"2024-03-24T09:22:05.262000+0000","last_fullsized":"2024-03-24T09:22:05.262000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265022+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:22:05.265022+0000","last_peered":"2024-03-24T09:22:05.265022+0000","last_clean":"2024-03-24T09:22:05.265022+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:22:05.265022+0000","last_undegraded":"2024-03-24T09:22:05.265022+0000","last_fullsized":"2024-03-24T09:22:05.265022+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.264376+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:22:05.264376+0000","last_peered":"2024-03-24T09:22:05.264376+0000","last_clean":"2024-03-24T09:22:05.264376+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:22:05.264376+0000","last_undegraded":"2024-03-24T09:22:05.264376+0000","last_fullsized":"2024-03-24T09:22:05.264376+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416864+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:22:05.416864+0000","last_peered":"2024-03-24T09:22:05.416864+0000","last_clean":"2024-03-24T09:22:05.416864+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:22:05.416864+0000","last_undegraded":"2024-03-24T09:22:05.416864+0000","last_fullsized":"2024-03-24T09:22:05.416864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263725+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:22:05.263725+0000","last_peered":"2024-03-24T09:22:05.263725+0000","last_clean":"2024-03-24T09:22:05.263725+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:22:05.263725+0000","last_undegraded":"2024-03-24T09:22:05.263725+0000","last_fullsized":"2024-03-24T09:22:05.263725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429825+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:22:05.429825+0000","last_peered":"2024-03-24T09:22:05.429825+0000","last_clean":"2024-03-24T09:22:05.429825+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:22:05.429825+0000","last_undegraded":"2024-03-24T09:22:05.429825+0000","last_fullsized":"2024-03-24T09:22:05.429825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383425+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:22:05.383425+0000","last_peered":"2024-03-24T09:22:05.383425+0000","last_clean":"2024-03-24T09:22:05.383425+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:22:05.383425+0000","last_undegraded":"2024-03-24T09:22:05.383425+0000","last_fullsized":"2024-03-24T09:22:05.383425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384159+0000","last_change":"2024-03-24T09:21:36.042199+0000","last_active":"2024-03-24T09:22:05.384159+0000","last_peered":"2024-03-24T09:22:05.384159+0000","last_clean":"2024-03-24T09:22:05.384159+0000","last_became_active":"2024-03-24T09:21:36.041615+0000","last_became_peered":"2024-03-24T09:21:36.041615+0000","last_unstale":"2024-03-24T09:22:05.384159+0000","last_undegraded":"2024-03-24T09:22:05.384159+0000","last_fullsized":"2024-03-24T09:22:05.384159+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:40:34.301227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261666+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:22:05.261666+0000","last_peered":"2024-03-24T09:22:05.261666+0000","last_clean":"2024-03-24T09:22:05.261666+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:22:05.261666+0000","last_undegraded":"2024-03-24T09:22:05.261666+0000","last_fullsized":"2024-03-24T09:22:05.261666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429552+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:22:05.429552+0000","last_peered":"2024-03-24T09:22:05.429552+0000","last_clean":"2024-03-24T09:22:05.429552+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:22:05.429552+0000","last_undegraded":"2024-03-24T09:22:05.429552+0000","last_fullsized":"2024-03-24T09:22:05.429552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263733+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:22:05.263733+0000","last_peered":"2024-03-24T09:22:05.263733+0000","last_clean":"2024-03-24T09:22:05.263733+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:22:05.263733+0000","last_undegraded":"2024-03-24T09:22:05.263733+0000","last_fullsized":"2024-03-24T09:22:05.263733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429612+0000","last_change":"2024-03-24T09:21:36.048092+0000","last_active":"2024-03-24T09:22:05.429612+0000","last_peered":"2024-03-24T09:22:05.429612+0000","last_clean":"2024-03-24T09:22:05.429612+0000","last_became_active":"2024-03-24T09:21:36.041520+0000","last_became_peered":"2024-03-24T09:21:36.041520+0000","last_unstale":"2024-03-24T09:22:05.429612+0000","last_undegraded":"2024-03-24T09:22:05.429612+0000","last_fullsized":"2024-03-24T09:22:05.429612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:53.315658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262181+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:22:05.262181+0000","last_peered":"2024-03-24T09:22:05.262181+0000","last_clean":"2024-03-24T09:22:05.262181+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:22:05.262181+0000","last_undegraded":"2024-03-24T09:22:05.262181+0000","last_fullsized":"2024-03-24T09:22:05.262181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263715+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:22:05.263715+0000","last_peered":"2024-03-24T09:22:05.263715+0000","last_clean":"2024-03-24T09:22:05.263715+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:22:05.263715+0000","last_undegraded":"2024-03-24T09:22:05.263715+0000","last_fullsized":"2024-03-24T09:22:05.263715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429820+0000","last_change":"2024-03-24T09:21:36.044659+0000","last_active":"2024-03-24T09:22:05.429820+0000","last_peered":"2024-03-24T09:22:05.429820+0000","last_clean":"2024-03-24T09:22:05.429820+0000","last_became_active":"2024-03-24T09:21:36.039364+0000","last_became_peered":"2024-03-24T09:21:36.039364+0000","last_unstale":"2024-03-24T09:22:05.429820+0000","last_undegraded":"2024-03-24T09:22:05.429820+0000","last_fullsized":"2024-03-24T09:22:05.429820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:51.993956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430424+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:22:05.430424+0000","last_peered":"2024-03-24T09:22:05.430424+0000","last_clean":"2024-03-24T09:22:05.430424+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:22:05.430424+0000","last_undegraded":"2024-03-24T09:22:05.430424+0000","last_fullsized":"2024-03-24T09:22:05.430424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430230+0000","last_change":"2024-03-24T09:21:36.039252+0000","last_active":"2024-03-24T09:22:05.430230+0000","last_peered":"2024-03-24T09:22:05.430230+0000","last_clean":"2024-03-24T09:22:05.430230+0000","last_became_active":"2024-03-24T09:21:36.038637+0000","last_became_peered":"2024-03-24T09:21:36.038637+0000","last_unstale":"2024-03-24T09:22:05.430230+0000","last_undegraded":"2024-03-24T09:22:05.430230+0000","last_fullsized":"2024-03-24T09:22:05.430230+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:49.512786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.076395+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:22:06.076395+0000","last_peered":"2024-03-24T09:22:06.076395+0000","last_clean":"2024-03-24T09:22:06.076395+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:22:06.076395+0000","last_undegraded":"2024-03-24T09:22:06.076395+0000","last_fullsized":"2024-03-24T09:22:06.076395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2054,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431370+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:22:05.431370+0000","last_peered":"2024-03-24T09:22:05.431370+0000","last_clean":"2024-03-24T09:22:05.431370+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:22:05.431370+0000","last_undegraded":"2024-03-24T09:22:05.431370+0000","last_fullsized":"2024-03-24T09:22:05.431370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416208+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:22:05.416208+0000","last_peered":"2024-03-24T09:22:05.416208+0000","last_clean":"2024-03-24T09:22:05.416208+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:22:05.416208+0000","last_undegraded":"2024-03-24T09:22:05.416208+0000","last_fullsized":"2024-03-24T09:22:05.416208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.512009+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:22:05.512009+0000","last_peered":"2024-03-24T09:22:05.512009+0000","last_clean":"2024-03-24T09:22:05.512009+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:22:05.512009+0000","last_undegraded":"2024-03-24T09:22:05.512009+0000","last_fullsized":"2024-03-24T09:22:05.512009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431252+0000","last_change":"2024-03-24T09:20:35.031333+0000","last_active":"2024-03-24T09:22:05.431252+0000","last_peered":"2024-03-24T09:22:05.431252+0000","last_clean":"2024-03-24T09:22:05.431252+0000","last_became_active":"2024-03-24T09:20:35.030995+0000","last_became_peered":"2024-03-24T09:20:35.030995+0000","last_unstale":"2024-03-24T09:22:05.431252+0000","last_undegraded":"2024-03-24T09:22:05.431252+0000","last_fullsized":"2024-03-24T09:22:05.431252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:05.746740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391405+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:22:05.391405+0000","last_peered":"2024-03-24T09:22:05.391405+0000","last_clean":"2024-03-24T09:22:05.391405+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:22:05.391405+0000","last_undegraded":"2024-03-24T09:22:05.391405+0000","last_fullsized":"2024-03-24T09:22:05.391405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430787+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:22:05.430787+0000","last_peered":"2024-03-24T09:22:05.430787+0000","last_clean":"2024-03-24T09:22:05.430787+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:22:05.430787+0000","last_undegraded":"2024-03-24T09:22:05.430787+0000","last_fullsized":"2024-03-24T09:22:05.430787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431653+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:22:05.431653+0000","last_peered":"2024-03-24T09:22:05.431653+0000","last_clean":"2024-03-24T09:22:05.431653+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:22:05.431653+0000","last_undegraded":"2024-03-24T09:22:05.431653+0000","last_fullsized":"2024-03-24T09:22:05.431653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429626+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:22:05.429626+0000","last_peered":"2024-03-24T09:22:05.429626+0000","last_clean":"2024-03-24T09:22:05.429626+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:22:05.429626+0000","last_undegraded":"2024-03-24T09:22:05.429626+0000","last_fullsized":"2024-03-24T09:22:05.429626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265298+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:22:05.265298+0000","last_peered":"2024-03-24T09:22:05.265298+0000","last_clean":"2024-03-24T09:22:05.265298+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:22:05.265298+0000","last_undegraded":"2024-03-24T09:22:05.265298+0000","last_fullsized":"2024-03-24T09:22:05.265298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265210+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:22:05.265210+0000","last_peered":"2024-03-24T09:22:05.265210+0000","last_clean":"2024-03-24T09:22:05.265210+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:22:05.265210+0000","last_undegraded":"2024-03-24T09:22:05.265210+0000","last_fullsized":"2024-03-24T09:22:05.265210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416529+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:22:05.416529+0000","last_peered":"2024-03-24T09:22:05.416529+0000","last_clean":"2024-03-24T09:22:05.416529+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:22:05.416529+0000","last_undegraded":"2024-03-24T09:22:05.416529+0000","last_fullsized":"2024-03-24T09:22:05.416529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430754+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:22:05.430754+0000","last_peered":"2024-03-24T09:22:05.430754+0000","last_clean":"2024-03-24T09:22:05.430754+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:22:05.430754+0000","last_undegraded":"2024-03-24T09:22:05.430754+0000","last_fullsized":"2024-03-24T09:22:05.430754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430399+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:22:05.430399+0000","last_peered":"2024-03-24T09:22:05.430399+0000","last_clean":"2024-03-24T09:22:05.430399+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:22:05.430399+0000","last_undegraded":"2024-03-24T09:22:05.430399+0000","last_fullsized":"2024-03-24T09:22:05.430399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265749+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:22:05.265749+0000","last_peered":"2024-03-24T09:22:05.265749+0000","last_clean":"2024-03-24T09:22:05.265749+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:22:05.265749+0000","last_undegraded":"2024-03-24T09:22:05.265749+0000","last_fullsized":"2024-03-24T09:22:05.265749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262063+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:22:05.262063+0000","last_peered":"2024-03-24T09:22:05.262063+0000","last_clean":"2024-03-24T09:22:05.262063+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:22:05.262063+0000","last_undegraded":"2024-03-24T09:22:05.262063+0000","last_fullsized":"2024-03-24T09:22:05.262063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391993+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:22:05.391993+0000","last_peered":"2024-03-24T09:22:05.391993+0000","last_clean":"2024-03-24T09:22:05.391993+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:22:05.391993+0000","last_undegraded":"2024-03-24T09:22:05.391993+0000","last_fullsized":"2024-03-24T09:22:05.391993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261494+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:22:05.261494+0000","last_peered":"2024-03-24T09:22:05.261494+0000","last_clean":"2024-03-24T09:22:05.261494+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:22:05.261494+0000","last_undegraded":"2024-03-24T09:22:05.261494+0000","last_fullsized":"2024-03-24T09:22:05.261494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430502+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:22:05.430502+0000","last_peered":"2024-03-24T09:22:05.430502+0000","last_clean":"2024-03-24T09:22:05.430502+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:22:05.430502+0000","last_undegraded":"2024-03-24T09:22:05.430502+0000","last_fullsized":"2024-03-24T09:22:05.430502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430370+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:22:05.430370+0000","last_peered":"2024-03-24T09:22:05.430370+0000","last_clean":"2024-03-24T09:22:05.430370+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:22:05.430370+0000","last_undegraded":"2024-03-24T09:22:05.430370+0000","last_fullsized":"2024-03-24T09:22:05.430370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430156+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:22:05.430156+0000","last_peered":"2024-03-24T09:22:05.430156+0000","last_clean":"2024-03-24T09:22:05.430156+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:22:05.430156+0000","last_undegraded":"2024-03-24T09:22:05.430156+0000","last_fullsized":"2024-03-24T09:22:05.430156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107004,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604641,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1352,"num_read_kb":795,"num_write":380,"num_write_kb":1476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1813923,"data_compressed":80715,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":531197,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1172328,"data_compressed":63929,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1093632,"data_stored":927555,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1085440,"data_stored":923837,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":1129270,"data_compressed":65311,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526240,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":596464,"data_compressed":31246,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":528087,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":598100,"data_compressed":31246,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1050271,"data_compressed":54945,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":531628,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1142784,"data_stored":1254934,"data_compressed":71467,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71439,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":73075,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135445,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":204337,"data_compressed":7759,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:22:09.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- 172.21.15.116:0/2742628883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 msgr2=0x7f49a8101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- 172.21.15.116:0/2742628883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8101430 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f4998009900 tx=0x7f499802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- 172.21.15.116:0/2742628883 shutdown_connections 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- 172.21.15.116:0/2742628883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a8101970 0x7f49a8108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- 172.21.15.116:0/2742628883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8101430 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- 172.21.15.116:0/2742628883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49a80ffdc0 0x7f49a81001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- 172.21.15.116:0/2742628883 >> 172.21.15.116:0/2742628883 conn(0x7f49a80fb330 msgr2=0x7f49a80fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- 172.21.15.116:0/2742628883 shutdown_connections 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- 172.21.15.116:0/2742628883 wait complete. 2024-03-24T09:22:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 Processor -- start 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- start start 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49a80ffdc0 0x7f49a81954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a8101970 0x7f49a819ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49a819cfb0 con 0x7f49a8100fb0 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f49a819d120 con 0x7f49a80ffdc0 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.400+0000 7f49afdf4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f49a819d2e0 con 0x7f49a8101970 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8195a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55880/0 (socket says 172.21.15.116:55880) 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 -- 172.21.15.116:0/3293817132 learned_addr learned my addr 172.21.15.116:0/3293817132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49adb90640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49a80ffdc0 0x7f49a81954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ae391640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a8101970 0x7f49a819ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 -- 172.21.15.116:0/3293817132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a8101970 msgr2=0x7f49a819ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a8101970 0x7f49a819ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 -- 172.21.15.116:0/3293817132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49a80ffdc0 msgr2=0x7f49a81954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49a80ffdc0 0x7f49a81954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 -- 172.21.15.116:0/3293817132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4998009560 con 0x7f49a8100fb0 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49ad38f640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8195a00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4998036040 tx=0x7f4998004140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49980035f0 con 0x7f49a8100fb0 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f499802de90 con 0x7f49a8100fb0 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4998005490 con 0x7f49a8100fb0 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a819d560 con 0x7f49a8100fb0 2024-03-24T09:22:09.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.401+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a819da70 con 0x7f49a8100fb0 2024-03-24T09:22:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.402+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a8107150 con 0x7f49a8100fb0 2024-03-24T09:22:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.404+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136877+0+0 (secure 0 0 0) 0x7f4998003750 con 0x7f49a8100fb0 2024-03-24T09:22:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.405+0000 7f499effd640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f49940a2230 0x7f49940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.405+0000 7f49adb90640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f49940a2230 0x7f49940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.405+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f49980f0500 con 0x7f49a8100fb0 2024-03-24T09:22:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.405+0000 7f49adb90640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f49940a2230 0x7f49940a46f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f49a8100e10 tx=0x7f49a4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:09.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.409+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f49980b8ef0 con 0x7f49a8100fb0 2024-03-24T09:22:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2131425162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-03-24T09:22:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3196231482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-03-24T09:22:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Unable to set osd_memory_target on smithi143 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4099658033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Unable to set osd_memory_target on smithi117 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:22:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:09 smithi143 ceph-mon[35965]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2131425162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3196231482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Unable to set osd_memory_target on smithi143 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4099658033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Unable to set osd_memory_target on smithi117 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:22:09.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:09 smithi117 ceph-mon[35788]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2131425162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3196231482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Unable to set osd_memory_target on smithi143 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4099658033' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Unable to set osd_memory_target on smithi117 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-24T09:22:09.572 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:09 smithi116 ceph-mon[32207]: Updating smithi143:/etc/ceph/ceph.conf 2024-03-24T09:22:09.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.603+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49a8061b50 con 0x7f49940a2230 2024-03-24T09:22:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.613+0000 7f499effd640 1 -- 172.21.15.116:0/3293817132 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339740 (secure 0 0 0) 0x7f49a8061b50 con 0x7f49940a2230 2024-03-24T09:22:09.617 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f49940a2230 msgr2=0x7f49940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f49940a2230 0x7f49940a46f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f49a8100e10 tx=0x7f49a4005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 msgr2=0x7f49a8195a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8195a00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4998036040 tx=0x7f4998004140 comp rx=0 tx=0).stop 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 shutdown_connections 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f49940a2230 0x7f49940a46f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f49a8101970 0x7f49a819ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49a8100fb0 0x7f49a8195a00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 --2- 172.21.15.116:0/3293817132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49a80ffdc0 0x7f49a81954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.616+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 >> 172.21.15.116:0/3293817132 conn(0x7f49a80fb330 msgr2=0x7f49a8104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:09.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.617+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 shutdown_connections 2024-03-24T09:22:09.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.617+0000 7f49afdf4640 1 -- 172.21.15.116:0/3293817132 wait complete. 2024-03-24T09:22:09.630 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T09:22:09.274608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195243,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1488,"num_read_kb":914,"num_write":563,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":948,"ondisk_log_size":948,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378676,"kb_used_data":12836,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563148,"statfs":{"total":1151940427776,"available":1151552663552,"internally_reserved":0,"allocated":13144064,"data_stored":9769911,"data_compressed":534294,"data_compressed_allocated":724992,"data_compressed_original":4194304,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987864"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430660+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:22:05.430660+0000","last_peered":"2024-03-24T09:22:05.430660+0000","last_clean":"2024-03-24T09:22:05.430660+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:22:05.430660+0000","last_undegraded":"2024-03-24T09:22:05.430660+0000","last_fullsized":"2024-03-24T09:22:05.430660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417006+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:22:05.417006+0000","last_peered":"2024-03-24T09:22:05.417006+0000","last_clean":"2024-03-24T09:22:05.417006+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:22:05.417006+0000","last_undegraded":"2024-03-24T09:22:05.417006+0000","last_fullsized":"2024-03-24T09:22:05.417006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416543+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:22:05.416543+0000","last_peered":"2024-03-24T09:22:05.416543+0000","last_clean":"2024-03-24T09:22:05.416543+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:22:05.416543+0000","last_undegraded":"2024-03-24T09:22:05.416543+0000","last_fullsized":"2024-03-24T09:22:05.416543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.264004+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:22:05.264004+0000","last_peered":"2024-03-24T09:22:05.264004+0000","last_clean":"2024-03-24T09:22:05.264004+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:22:05.264004+0000","last_undegraded":"2024-03-24T09:22:05.264004+0000","last_fullsized":"2024-03-24T09:22:05.264004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383474+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:22:05.383474+0000","last_peered":"2024-03-24T09:22:05.383474+0000","last_clean":"2024-03-24T09:22:05.383474+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:22:05.383474+0000","last_undegraded":"2024-03-24T09:22:05.383474+0000","last_fullsized":"2024-03-24T09:22:05.383474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395243+0000","last_change":"2024-03-24T09:20:35.029608+0000","last_active":"2024-03-24T09:22:05.395243+0000","last_peered":"2024-03-24T09:22:05.395243+0000","last_clean":"2024-03-24T09:22:05.395243+0000","last_became_active":"2024-03-24T09:20:35.029268+0000","last_became_peered":"2024-03-24T09:20:35.029268+0000","last_unstale":"2024-03-24T09:22:05.395243+0000","last_undegraded":"2024-03-24T09:22:05.395243+0000","last_fullsized":"2024-03-24T09:22:05.395243+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:47.087125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430275+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:22:05.430275+0000","last_peered":"2024-03-24T09:22:05.430275+0000","last_clean":"2024-03-24T09:22:05.430275+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:22:05.430275+0000","last_undegraded":"2024-03-24T09:22:05.430275+0000","last_fullsized":"2024-03-24T09:22:05.430275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261421+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:22:05.261421+0000","last_peered":"2024-03-24T09:22:05.261421+0000","last_clean":"2024-03-24T09:22:05.261421+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:22:05.261421+0000","last_undegraded":"2024-03-24T09:22:05.261421+0000","last_fullsized":"2024-03-24T09:22:05.261421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431980+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:22:05.431980+0000","last_peered":"2024-03-24T09:22:05.431980+0000","last_clean":"2024-03-24T09:22:05.431980+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:22:05.431980+0000","last_undegraded":"2024-03-24T09:22:05.431980+0000","last_fullsized":"2024-03-24T09:22:05.431980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395656+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:22:05.395656+0000","last_peered":"2024-03-24T09:22:05.395656+0000","last_clean":"2024-03-24T09:22:05.395656+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:22:05.395656+0000","last_undegraded":"2024-03-24T09:22:05.395656+0000","last_fullsized":"2024-03-24T09:22:05.395656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431628+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:22:05.431628+0000","last_peered":"2024-03-24T09:22:05.431628+0000","last_clean":"2024-03-24T09:22:05.431628+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:22:05.431628+0000","last_undegraded":"2024-03-24T09:22:05.431628+0000","last_fullsized":"2024-03-24T09:22:05.431628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263913+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:22:05.263913+0000","last_peered":"2024-03-24T09:22:05.263913+0000","last_clean":"2024-03-24T09:22:05.263913+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:22:05.263913+0000","last_undegraded":"2024-03-24T09:22:05.263913+0000","last_fullsized":"2024-03-24T09:22:05.263913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430584+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:22:05.430584+0000","last_peered":"2024-03-24T09:22:05.430584+0000","last_clean":"2024-03-24T09:22:05.430584+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:22:05.430584+0000","last_undegraded":"2024-03-24T09:22:05.430584+0000","last_fullsized":"2024-03-24T09:22:05.430584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416250+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:22:05.416250+0000","last_peered":"2024-03-24T09:22:05.416250+0000","last_clean":"2024-03-24T09:22:05.416250+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:22:05.416250+0000","last_undegraded":"2024-03-24T09:22:05.416250+0000","last_fullsized":"2024-03-24T09:22:05.416250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416249+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:22:05.416249+0000","last_peered":"2024-03-24T09:22:05.416249+0000","last_clean":"2024-03-24T09:22:05.416249+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:22:05.416249+0000","last_undegraded":"2024-03-24T09:22:05.416249+0000","last_fullsized":"2024-03-24T09:22:05.416249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429842+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:22:05.429842+0000","last_peered":"2024-03-24T09:22:05.429842+0000","last_clean":"2024-03-24T09:22:05.429842+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:22:05.429842+0000","last_undegraded":"2024-03-24T09:22:05.429842+0000","last_fullsized":"2024-03-24T09:22:05.429842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430550+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:22:05.430550+0000","last_peered":"2024-03-24T09:22:05.430550+0000","last_clean":"2024-03-24T09:22:05.430550+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:22:05.430550+0000","last_undegraded":"2024-03-24T09:22:05.430550+0000","last_fullsized":"2024-03-24T09:22:05.430550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381311+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:22:05.381311+0000","last_peered":"2024-03-24T09:22:05.381311+0000","last_clean":"2024-03-24T09:22:05.381311+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:22:05.381311+0000","last_undegraded":"2024-03-24T09:22:05.381311+0000","last_fullsized":"2024-03-24T09:22:05.381311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265648+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:22:05.265648+0000","last_peered":"2024-03-24T09:22:05.265648+0000","last_clean":"2024-03-24T09:22:05.265648+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:22:05.265648+0000","last_undegraded":"2024-03-24T09:22:05.265648+0000","last_fullsized":"2024-03-24T09:22:05.265648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265189+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:22:05.265189+0000","last_peered":"2024-03-24T09:22:05.265189+0000","last_clean":"2024-03-24T09:22:05.265189+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:22:05.265189+0000","last_undegraded":"2024-03-24T09:22:05.265189+0000","last_fullsized":"2024-03-24T09:22:05.265189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381586+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:22:05.381586+0000","last_peered":"2024-03-24T09:22:05.381586+0000","last_clean":"2024-03-24T09:22:05.381586+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:22:05.381586+0000","last_undegraded":"2024-03-24T09:22:05.381586+0000","last_fullsized":"2024-03-24T09:22:05.381586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262126+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:22:05.262126+0000","last_peered":"2024-03-24T09:22:05.262126+0000","last_clean":"2024-03-24T09:22:05.262126+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:22:05.262126+0000","last_undegraded":"2024-03-24T09:22:05.262126+0000","last_fullsized":"2024-03-24T09:22:05.262126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262833+0000","last_change":"2024-03-24T09:20:35.302025+0000","last_active":"2024-03-24T09:22:05.262833+0000","last_peered":"2024-03-24T09:22:05.262833+0000","last_clean":"2024-03-24T09:22:05.262833+0000","last_became_active":"2024-03-24T09:20:35.301575+0000","last_became_peered":"2024-03-24T09:20:35.301575+0000","last_unstale":"2024-03-24T09:22:05.262833+0000","last_undegraded":"2024-03-24T09:22:05.262833+0000","last_fullsized":"2024-03-24T09:22:05.262833+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:20.305175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262343+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:22:05.262343+0000","last_peered":"2024-03-24T09:22:05.262343+0000","last_clean":"2024-03-24T09:22:05.262343+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:22:05.262343+0000","last_undegraded":"2024-03-24T09:22:05.262343+0000","last_fullsized":"2024-03-24T09:22:05.262343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417089+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:22:05.417089+0000","last_peered":"2024-03-24T09:22:05.417089+0000","last_clean":"2024-03-24T09:22:05.417089+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:22:05.417089+0000","last_undegraded":"2024-03-24T09:22:05.417089+0000","last_fullsized":"2024-03-24T09:22:05.417089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263181+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:22:05.263181+0000","last_peered":"2024-03-24T09:22:05.263181+0000","last_clean":"2024-03-24T09:22:05.263181+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:22:05.263181+0000","last_undegraded":"2024-03-24T09:22:05.263181+0000","last_fullsized":"2024-03-24T09:22:05.263181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384209+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:22:05.384209+0000","last_peered":"2024-03-24T09:22:05.384209+0000","last_clean":"2024-03-24T09:22:05.384209+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:22:05.384209+0000","last_undegraded":"2024-03-24T09:22:05.384209+0000","last_fullsized":"2024-03-24T09:22:05.384209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430840+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:22:05.430840+0000","last_peered":"2024-03-24T09:22:05.430840+0000","last_clean":"2024-03-24T09:22:05.430840+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:22:05.430840+0000","last_undegraded":"2024-03-24T09:22:05.430840+0000","last_fullsized":"2024-03-24T09:22:05.430840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263015+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:22:05.263015+0000","last_peered":"2024-03-24T09:22:05.263015+0000","last_clean":"2024-03-24T09:22:05.263015+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:22:05.263015+0000","last_undegraded":"2024-03-24T09:22:05.263015+0000","last_fullsized":"2024-03-24T09:22:05.263015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383756+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:22:05.383756+0000","last_peered":"2024-03-24T09:22:05.383756+0000","last_clean":"2024-03-24T09:22:05.383756+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:22:05.383756+0000","last_undegraded":"2024-03-24T09:22:05.383756+0000","last_fullsized":"2024-03-24T09:22:05.383756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391949+0000","last_change":"2024-03-24T09:20:35.032271+0000","last_active":"2024-03-24T09:22:05.391949+0000","last_peered":"2024-03-24T09:22:05.391949+0000","last_clean":"2024-03-24T09:22:05.391949+0000","last_became_active":"2024-03-24T09:20:35.031913+0000","last_became_peered":"2024-03-24T09:20:35.031913+0000","last_unstale":"2024-03-24T09:22:05.391949+0000","last_undegraded":"2024-03-24T09:22:05.391949+0000","last_fullsized":"2024-03-24T09:22:05.391949+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:40.322197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265519+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:22:05.265519+0000","last_peered":"2024-03-24T09:22:05.265519+0000","last_clean":"2024-03-24T09:22:05.265519+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:22:05.265519+0000","last_undegraded":"2024-03-24T09:22:05.265519+0000","last_fullsized":"2024-03-24T09:22:05.265519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392028+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:22:05.392028+0000","last_peered":"2024-03-24T09:22:05.392028+0000","last_clean":"2024-03-24T09:22:05.392028+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:22:05.392028+0000","last_undegraded":"2024-03-24T09:22:05.392028+0000","last_fullsized":"2024-03-24T09:22:05.392028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383732+0000","last_change":"2024-03-24T09:20:35.031176+0000","last_active":"2024-03-24T09:22:05.383732+0000","last_peered":"2024-03-24T09:22:05.383732+0000","last_clean":"2024-03-24T09:22:05.383732+0000","last_became_active":"2024-03-24T09:20:35.030888+0000","last_became_peered":"2024-03-24T09:20:35.030888+0000","last_unstale":"2024-03-24T09:22:05.383732+0000","last_undegraded":"2024-03-24T09:22:05.383732+0000","last_fullsized":"2024-03-24T09:22:05.383732+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:43.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265449+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:22:05.265449+0000","last_peered":"2024-03-24T09:22:05.265449+0000","last_clean":"2024-03-24T09:22:05.265449+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:22:05.265449+0000","last_undegraded":"2024-03-24T09:22:05.265449+0000","last_fullsized":"2024-03-24T09:22:05.265449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395214+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:22:05.395214+0000","last_peered":"2024-03-24T09:22:05.395214+0000","last_clean":"2024-03-24T09:22:05.395214+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:22:05.395214+0000","last_undegraded":"2024-03-24T09:22:05.395214+0000","last_fullsized":"2024-03-24T09:22:05.395214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391824+0000","last_change":"2024-03-24T09:20:35.301944+0000","last_active":"2024-03-24T09:22:05.391824+0000","last_peered":"2024-03-24T09:22:05.391824+0000","last_clean":"2024-03-24T09:22:05.391824+0000","last_became_active":"2024-03-24T09:20:35.301588+0000","last_became_peered":"2024-03-24T09:20:35.301588+0000","last_unstale":"2024-03-24T09:22:05.391824+0000","last_undegraded":"2024-03-24T09:22:05.391824+0000","last_fullsized":"2024-03-24T09:22:05.391824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:33.185748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430406+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:22:05.430406+0000","last_peered":"2024-03-24T09:22:05.430406+0000","last_clean":"2024-03-24T09:22:05.430406+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:22:05.430406+0000","last_undegraded":"2024-03-24T09:22:05.430406+0000","last_fullsized":"2024-03-24T09:22:05.430406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430255+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:22:05.430255+0000","last_peered":"2024-03-24T09:22:05.430255+0000","last_clean":"2024-03-24T09:22:05.430255+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:22:05.430255+0000","last_undegraded":"2024-03-24T09:22:05.430255+0000","last_fullsized":"2024-03-24T09:22:05.430255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261576+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:22:05.261576+0000","last_peered":"2024-03-24T09:22:05.261576+0000","last_clean":"2024-03-24T09:22:05.261576+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:22:05.261576+0000","last_undegraded":"2024-03-24T09:22:05.261576+0000","last_fullsized":"2024-03-24T09:22:05.261576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416766+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:22:05.416766+0000","last_peered":"2024-03-24T09:22:05.416766+0000","last_clean":"2024-03-24T09:22:05.416766+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:22:05.416766+0000","last_undegraded":"2024-03-24T09:22:05.416766+0000","last_fullsized":"2024-03-24T09:22:05.416766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392042+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:22:05.392042+0000","last_peered":"2024-03-24T09:22:05.392042+0000","last_clean":"2024-03-24T09:22:05.392042+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:22:05.392042+0000","last_undegraded":"2024-03-24T09:22:05.392042+0000","last_fullsized":"2024-03-24T09:22:05.392042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380573+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:22:05.380573+0000","last_peered":"2024-03-24T09:22:05.380573+0000","last_clean":"2024-03-24T09:22:05.380573+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:22:05.380573+0000","last_undegraded":"2024-03-24T09:22:05.380573+0000","last_fullsized":"2024-03-24T09:22:05.380573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261827+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:22:05.261827+0000","last_peered":"2024-03-24T09:22:05.261827+0000","last_clean":"2024-03-24T09:22:05.261827+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:22:05.261827+0000","last_undegraded":"2024-03-24T09:22:05.261827+0000","last_fullsized":"2024-03-24T09:22:05.261827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":284,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.075787+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:22:06.075787+0000","last_peered":"2024-03-24T09:22:06.075787+0000","last_clean":"2024-03-24T09:22:06.075787+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:22:06.075787+0000","last_undegraded":"2024-03-24T09:22:06.075787+0000","last_fullsized":"2024-03-24T09:22:06.075787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431671+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:22:05.431671+0000","last_peered":"2024-03-24T09:22:05.431671+0000","last_clean":"2024-03-24T09:22:05.431671+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:22:05.431671+0000","last_undegraded":"2024-03-24T09:22:05.431671+0000","last_fullsized":"2024-03-24T09:22:05.431671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380431+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:22:05.380431+0000","last_peered":"2024-03-24T09:22:05.380431+0000","last_clean":"2024-03-24T09:22:05.380431+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:22:05.380431+0000","last_undegraded":"2024-03-24T09:22:05.380431+0000","last_fullsized":"2024-03-24T09:22:05.380431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431572+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:22:05.431572+0000","last_peered":"2024-03-24T09:22:05.431572+0000","last_clean":"2024-03-24T09:22:05.431572+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:22:05.431572+0000","last_undegraded":"2024-03-24T09:22:05.431572+0000","last_fullsized":"2024-03-24T09:22:05.431572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417192+0000","last_change":"2024-03-24T09:20:35.301994+0000","last_active":"2024-03-24T09:22:05.417192+0000","last_peered":"2024-03-24T09:22:05.417192+0000","last_clean":"2024-03-24T09:22:05.417192+0000","last_became_active":"2024-03-24T09:20:35.301570+0000","last_became_peered":"2024-03-24T09:20:35.301570+0000","last_unstale":"2024-03-24T09:22:05.417192+0000","last_undegraded":"2024-03-24T09:22:05.417192+0000","last_fullsized":"2024-03-24T09:22:05.417192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:34.896090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263428+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:22:05.263428+0000","last_peered":"2024-03-24T09:22:05.263428+0000","last_clean":"2024-03-24T09:22:05.263428+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:22:05.263428+0000","last_undegraded":"2024-03-24T09:22:05.263428+0000","last_fullsized":"2024-03-24T09:22:05.263428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391847+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:22:05.391847+0000","last_peered":"2024-03-24T09:22:05.391847+0000","last_clean":"2024-03-24T09:22:05.391847+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:22:05.391847+0000","last_undegraded":"2024-03-24T09:22:05.391847+0000","last_fullsized":"2024-03-24T09:22:05.391847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380857+0000","last_change":"2024-03-24T09:20:35.027718+0000","last_active":"2024-03-24T09:22:05.380857+0000","last_peered":"2024-03-24T09:22:05.380857+0000","last_clean":"2024-03-24T09:22:05.380857+0000","last_became_active":"2024-03-24T09:20:35.027347+0000","last_became_peered":"2024-03-24T09:20:35.027347+0000","last_unstale":"2024-03-24T09:22:05.380857+0000","last_undegraded":"2024-03-24T09:22:05.380857+0000","last_fullsized":"2024-03-24T09:22:05.380857+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:20.097882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395060+0000","last_change":"2024-03-24T09:20:35.301625+0000","last_active":"2024-03-24T09:22:05.395060+0000","last_peered":"2024-03-24T09:22:05.395060+0000","last_clean":"2024-03-24T09:22:05.395060+0000","last_became_active":"2024-03-24T09:20:35.301035+0000","last_became_peered":"2024-03-24T09:20:35.301035+0000","last_unstale":"2024-03-24T09:22:05.395060+0000","last_undegraded":"2024-03-24T09:22:05.395060+0000","last_fullsized":"2024-03-24T09:22:05.395060+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:09.448556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":333,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.021394+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:22:06.021394+0000","last_peered":"2024-03-24T09:22:06.021394+0000","last_clean":"2024-03-24T09:22:06.021394+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:22:06.021394+0000","last_undegraded":"2024-03-24T09:22:06.021394+0000","last_fullsized":"2024-03-24T09:22:06.021394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69194,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":257,"num_read_kb":196,"num_write":44,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384057+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:22:05.384057+0000","last_peered":"2024-03-24T09:22:05.384057+0000","last_clean":"2024-03-24T09:22:05.384057+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:22:05.384057+0000","last_undegraded":"2024-03-24T09:22:05.384057+0000","last_fullsized":"2024-03-24T09:22:05.384057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392229+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:22:05.392229+0000","last_peered":"2024-03-24T09:22:05.392229+0000","last_clean":"2024-03-24T09:22:05.392229+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:22:05.392229+0000","last_undegraded":"2024-03-24T09:22:05.392229+0000","last_fullsized":"2024-03-24T09:22:05.392229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265368+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:22:05.265368+0000","last_peered":"2024-03-24T09:22:05.265368+0000","last_clean":"2024-03-24T09:22:05.265368+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:22:05.265368+0000","last_undegraded":"2024-03-24T09:22:05.265368+0000","last_fullsized":"2024-03-24T09:22:05.265368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430401+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:22:05.430401+0000","last_peered":"2024-03-24T09:22:05.430401+0000","last_clean":"2024-03-24T09:22:05.430401+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:22:05.430401+0000","last_undegraded":"2024-03-24T09:22:05.430401+0000","last_fullsized":"2024-03-24T09:22:05.430401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429872+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:22:05.429872+0000","last_peered":"2024-03-24T09:22:05.429872+0000","last_clean":"2024-03-24T09:22:05.429872+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:22:05.429872+0000","last_undegraded":"2024-03-24T09:22:05.429872+0000","last_fullsized":"2024-03-24T09:22:05.429872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391701+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:22:05.391701+0000","last_peered":"2024-03-24T09:22:05.391701+0000","last_clean":"2024-03-24T09:22:05.391701+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:22:05.391701+0000","last_undegraded":"2024-03-24T09:22:05.391701+0000","last_fullsized":"2024-03-24T09:22:05.391701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430945+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:22:05.430945+0000","last_peered":"2024-03-24T09:22:05.430945+0000","last_clean":"2024-03-24T09:22:05.430945+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:22:05.430945+0000","last_undegraded":"2024-03-24T09:22:05.430945+0000","last_fullsized":"2024-03-24T09:22:05.430945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381184+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:22:05.381184+0000","last_peered":"2024-03-24T09:22:05.381184+0000","last_clean":"2024-03-24T09:22:05.381184+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:22:05.381184+0000","last_undegraded":"2024-03-24T09:22:05.381184+0000","last_fullsized":"2024-03-24T09:22:05.381184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'10","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.079573+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:22:06.079573+0000","last_peered":"2024-03-24T09:22:06.079573+0000","last_clean":"2024-03-24T09:22:06.079573+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:22:06.079573+0000","last_undegraded":"2024-03-24T09:22:06.079573+0000","last_fullsized":"2024-03-24T09:22:06.079573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430294+0000","last_change":"2024-03-24T09:20:35.031305+0000","last_active":"2024-03-24T09:22:05.430294+0000","last_peered":"2024-03-24T09:22:05.430294+0000","last_clean":"2024-03-24T09:22:05.430294+0000","last_became_active":"2024-03-24T09:20:35.030952+0000","last_became_peered":"2024-03-24T09:20:35.030952+0000","last_unstale":"2024-03-24T09:22:05.430294+0000","last_undegraded":"2024-03-24T09:22:05.430294+0000","last_fullsized":"2024-03-24T09:22:05.430294+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:40.987185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430873+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:22:05.430873+0000","last_peered":"2024-03-24T09:22:05.430873+0000","last_clean":"2024-03-24T09:22:05.430873+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:22:05.430873+0000","last_undegraded":"2024-03-24T09:22:05.430873+0000","last_fullsized":"2024-03-24T09:22:05.430873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383466+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:22:05.383466+0000","last_peered":"2024-03-24T09:22:05.383466+0000","last_clean":"2024-03-24T09:22:05.383466+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:22:05.383466+0000","last_undegraded":"2024-03-24T09:22:05.383466+0000","last_fullsized":"2024-03-24T09:22:05.383466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391686+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:22:05.391686+0000","last_peered":"2024-03-24T09:22:05.391686+0000","last_clean":"2024-03-24T09:22:05.391686+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:22:05.391686+0000","last_undegraded":"2024-03-24T09:22:05.391686+0000","last_fullsized":"2024-03-24T09:22:05.391686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391546+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:22:05.391546+0000","last_peered":"2024-03-24T09:22:05.391546+0000","last_clean":"2024-03-24T09:22:05.391546+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:22:05.391546+0000","last_undegraded":"2024-03-24T09:22:05.391546+0000","last_fullsized":"2024-03-24T09:22:05.391546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384256+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:22:05.384256+0000","last_peered":"2024-03-24T09:22:05.384256+0000","last_clean":"2024-03-24T09:22:05.384256+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:22:05.384256+0000","last_undegraded":"2024-03-24T09:22:05.384256+0000","last_fullsized":"2024-03-24T09:22:05.384256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430172+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:22:05.430172+0000","last_peered":"2024-03-24T09:22:05.430172+0000","last_clean":"2024-03-24T09:22:05.430172+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:22:05.430172+0000","last_undegraded":"2024-03-24T09:22:05.430172+0000","last_fullsized":"2024-03-24T09:22:05.430172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383731+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:22:05.383731+0000","last_peered":"2024-03-24T09:22:05.383731+0000","last_clean":"2024-03-24T09:22:05.383731+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:22:05.383731+0000","last_undegraded":"2024-03-24T09:22:05.383731+0000","last_fullsized":"2024-03-24T09:22:05.383731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380914+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:22:05.380914+0000","last_peered":"2024-03-24T09:22:05.380914+0000","last_clean":"2024-03-24T09:22:05.380914+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:22:05.380914+0000","last_undegraded":"2024-03-24T09:22:05.380914+0000","last_fullsized":"2024-03-24T09:22:05.380914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395507+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:22:05.395507+0000","last_peered":"2024-03-24T09:22:05.395507+0000","last_clean":"2024-03-24T09:22:05.395507+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:22:05.395507+0000","last_undegraded":"2024-03-24T09:22:05.395507+0000","last_fullsized":"2024-03-24T09:22:05.395507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430587+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:22:05.430587+0000","last_peered":"2024-03-24T09:22:05.430587+0000","last_clean":"2024-03-24T09:22:05.430587+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:22:05.430587+0000","last_undegraded":"2024-03-24T09:22:05.430587+0000","last_fullsized":"2024-03-24T09:22:05.430587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430650+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:22:05.430650+0000","last_peered":"2024-03-24T09:22:05.430650+0000","last_clean":"2024-03-24T09:22:05.430650+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:22:05.430650+0000","last_undegraded":"2024-03-24T09:22:05.430650+0000","last_fullsized":"2024-03-24T09:22:05.430650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395163+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:22:05.395163+0000","last_peered":"2024-03-24T09:22:05.395163+0000","last_clean":"2024-03-24T09:22:05.395163+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:22:05.395163+0000","last_undegraded":"2024-03-24T09:22:05.395163+0000","last_fullsized":"2024-03-24T09:22:05.395163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430946+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:22:05.430946+0000","last_peered":"2024-03-24T09:22:05.430946+0000","last_clean":"2024-03-24T09:22:05.430946+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:22:05.430946+0000","last_undegraded":"2024-03-24T09:22:05.430946+0000","last_fullsized":"2024-03-24T09:22:05.430946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392124+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:22:05.392124+0000","last_peered":"2024-03-24T09:22:05.392124+0000","last_clean":"2024-03-24T09:22:05.392124+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:22:05.392124+0000","last_undegraded":"2024-03-24T09:22:05.392124+0000","last_fullsized":"2024-03-24T09:22:05.392124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430122+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:22:05.430122+0000","last_peered":"2024-03-24T09:22:05.430122+0000","last_clean":"2024-03-24T09:22:05.430122+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:22:05.430122+0000","last_undegraded":"2024-03-24T09:22:05.430122+0000","last_fullsized":"2024-03-24T09:22:05.430122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431778+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:22:05.431778+0000","last_peered":"2024-03-24T09:22:05.431778+0000","last_clean":"2024-03-24T09:22:05.431778+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:22:05.431778+0000","last_undegraded":"2024-03-24T09:22:05.431778+0000","last_fullsized":"2024-03-24T09:22:05.431778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261595+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:22:05.261595+0000","last_peered":"2024-03-24T09:22:05.261595+0000","last_clean":"2024-03-24T09:22:05.261595+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:22:05.261595+0000","last_undegraded":"2024-03-24T09:22:05.261595+0000","last_fullsized":"2024-03-24T09:22:05.261595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263319+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:22:05.263319+0000","last_peered":"2024-03-24T09:22:05.263319+0000","last_clean":"2024-03-24T09:22:05.263319+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:22:05.263319+0000","last_undegraded":"2024-03-24T09:22:05.263319+0000","last_fullsized":"2024-03-24T09:22:05.263319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429912+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:22:05.429912+0000","last_peered":"2024-03-24T09:22:05.429912+0000","last_clean":"2024-03-24T09:22:05.429912+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:22:05.429912+0000","last_undegraded":"2024-03-24T09:22:05.429912+0000","last_fullsized":"2024-03-24T09:22:05.429912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392506+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:22:05.392506+0000","last_peered":"2024-03-24T09:22:05.392506+0000","last_clean":"2024-03-24T09:22:05.392506+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:22:05.392506+0000","last_undegraded":"2024-03-24T09:22:05.392506+0000","last_fullsized":"2024-03-24T09:22:05.392506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429921+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:22:05.429921+0000","last_peered":"2024-03-24T09:22:05.429921+0000","last_clean":"2024-03-24T09:22:05.429921+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:22:05.429921+0000","last_undegraded":"2024-03-24T09:22:05.429921+0000","last_fullsized":"2024-03-24T09:22:05.429921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430097+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:22:05.430097+0000","last_peered":"2024-03-24T09:22:05.430097+0000","last_clean":"2024-03-24T09:22:05.430097+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:22:05.430097+0000","last_undegraded":"2024-03-24T09:22:05.430097+0000","last_fullsized":"2024-03-24T09:22:05.430097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429932+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:22:05.429932+0000","last_peered":"2024-03-24T09:22:05.429932+0000","last_clean":"2024-03-24T09:22:05.429932+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:22:05.429932+0000","last_undegraded":"2024-03-24T09:22:05.429932+0000","last_fullsized":"2024-03-24T09:22:05.429932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384084+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:22:05.384084+0000","last_peered":"2024-03-24T09:22:05.384084+0000","last_clean":"2024-03-24T09:22:05.384084+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:22:05.384084+0000","last_undegraded":"2024-03-24T09:22:05.384084+0000","last_fullsized":"2024-03-24T09:22:05.384084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430884+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:22:05.430884+0000","last_peered":"2024-03-24T09:22:05.430884+0000","last_clean":"2024-03-24T09:22:05.430884+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:22:05.430884+0000","last_undegraded":"2024-03-24T09:22:05.430884+0000","last_fullsized":"2024-03-24T09:22:05.430884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261523+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:22:05.261523+0000","last_peered":"2024-03-24T09:22:05.261523+0000","last_clean":"2024-03-24T09:22:05.261523+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:22:05.261523+0000","last_undegraded":"2024-03-24T09:22:05.261523+0000","last_fullsized":"2024-03-24T09:22:05.261523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.514742+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:22:05.514742+0000","last_peered":"2024-03-24T09:22:05.514742+0000","last_clean":"2024-03-24T09:22:05.514742+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:22:05.514742+0000","last_undegraded":"2024-03-24T09:22:05.514742+0000","last_fullsized":"2024-03-24T09:22:05.514742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262000+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:22:05.262000+0000","last_peered":"2024-03-24T09:22:05.262000+0000","last_clean":"2024-03-24T09:22:05.262000+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:22:05.262000+0000","last_undegraded":"2024-03-24T09:22:05.262000+0000","last_fullsized":"2024-03-24T09:22:05.262000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265022+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:22:05.265022+0000","last_peered":"2024-03-24T09:22:05.265022+0000","last_clean":"2024-03-24T09:22:05.265022+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:22:05.265022+0000","last_undegraded":"2024-03-24T09:22:05.265022+0000","last_fullsized":"2024-03-24T09:22:05.265022+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.264376+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:22:05.264376+0000","last_peered":"2024-03-24T09:22:05.264376+0000","last_clean":"2024-03-24T09:22:05.264376+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:22:05.264376+0000","last_undegraded":"2024-03-24T09:22:05.264376+0000","last_fullsized":"2024-03-24T09:22:05.264376+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416864+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:22:05.416864+0000","last_peered":"2024-03-24T09:22:05.416864+0000","last_clean":"2024-03-24T09:22:05.416864+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:22:05.416864+0000","last_undegraded":"2024-03-24T09:22:05.416864+0000","last_fullsized":"2024-03-24T09:22:05.416864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263725+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:22:05.263725+0000","last_peered":"2024-03-24T09:22:05.263725+0000","last_clean":"2024-03-24T09:22:05.263725+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:22:05.263725+0000","last_undegraded":"2024-03-24T09:22:05.263725+0000","last_fullsized":"2024-03-24T09:22:05.263725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429825+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:22:05.429825+0000","last_peered":"2024-03-24T09:22:05.429825+0000","last_clean":"2024-03-24T09:22:05.429825+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:22:05.429825+0000","last_undegraded":"2024-03-24T09:22:05.429825+0000","last_fullsized":"2024-03-24T09:22:05.429825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383425+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:22:05.383425+0000","last_peered":"2024-03-24T09:22:05.383425+0000","last_clean":"2024-03-24T09:22:05.383425+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:22:05.383425+0000","last_undegraded":"2024-03-24T09:22:05.383425+0000","last_fullsized":"2024-03-24T09:22:05.383425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384159+0000","last_change":"2024-03-24T09:21:36.042199+0000","last_active":"2024-03-24T09:22:05.384159+0000","last_peered":"2024-03-24T09:22:05.384159+0000","last_clean":"2024-03-24T09:22:05.384159+0000","last_became_active":"2024-03-24T09:21:36.041615+0000","last_became_peered":"2024-03-24T09:21:36.041615+0000","last_unstale":"2024-03-24T09:22:05.384159+0000","last_undegraded":"2024-03-24T09:22:05.384159+0000","last_fullsized":"2024-03-24T09:22:05.384159+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:40:34.301227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261666+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:22:05.261666+0000","last_peered":"2024-03-24T09:22:05.261666+0000","last_clean":"2024-03-24T09:22:05.261666+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:22:05.261666+0000","last_undegraded":"2024-03-24T09:22:05.261666+0000","last_fullsized":"2024-03-24T09:22:05.261666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429552+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:22:05.429552+0000","last_peered":"2024-03-24T09:22:05.429552+0000","last_clean":"2024-03-24T09:22:05.429552+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:22:05.429552+0000","last_undegraded":"2024-03-24T09:22:05.429552+0000","last_fullsized":"2024-03-24T09:22:05.429552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263733+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:22:05.263733+0000","last_peered":"2024-03-24T09:22:05.263733+0000","last_clean":"2024-03-24T09:22:05.263733+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:22:05.263733+0000","last_undegraded":"2024-03-24T09:22:05.263733+0000","last_fullsized":"2024-03-24T09:22:05.263733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429612+0000","last_change":"2024-03-24T09:21:36.048092+0000","last_active":"2024-03-24T09:22:05.429612+0000","last_peered":"2024-03-24T09:22:05.429612+0000","last_clean":"2024-03-24T09:22:05.429612+0000","last_became_active":"2024-03-24T09:21:36.041520+0000","last_became_peered":"2024-03-24T09:21:36.041520+0000","last_unstale":"2024-03-24T09:22:05.429612+0000","last_undegraded":"2024-03-24T09:22:05.429612+0000","last_fullsized":"2024-03-24T09:22:05.429612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:53.315658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262181+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:22:05.262181+0000","last_peered":"2024-03-24T09:22:05.262181+0000","last_clean":"2024-03-24T09:22:05.262181+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:22:05.262181+0000","last_undegraded":"2024-03-24T09:22:05.262181+0000","last_fullsized":"2024-03-24T09:22:05.262181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263715+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:22:05.263715+0000","last_peered":"2024-03-24T09:22:05.263715+0000","last_clean":"2024-03-24T09:22:05.263715+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:22:05.263715+0000","last_undegraded":"2024-03-24T09:22:05.263715+0000","last_fullsized":"2024-03-24T09:22:05.263715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429820+0000","last_change":"2024-03-24T09:21:36.044659+0000","last_active":"2024-03-24T09:22:05.429820+0000","last_peered":"2024-03-24T09:22:05.429820+0000","last_clean":"2024-03-24T09:22:05.429820+0000","last_became_active":"2024-03-24T09:21:36.039364+0000","last_became_peered":"2024-03-24T09:21:36.039364+0000","last_unstale":"2024-03-24T09:22:05.429820+0000","last_undegraded":"2024-03-24T09:22:05.429820+0000","last_fullsized":"2024-03-24T09:22:05.429820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:51.993956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430424+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:22:05.430424+0000","last_peered":"2024-03-24T09:22:05.430424+0000","last_clean":"2024-03-24T09:22:05.430424+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:22:05.430424+0000","last_undegraded":"2024-03-24T09:22:05.430424+0000","last_fullsized":"2024-03-24T09:22:05.430424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430230+0000","last_change":"2024-03-24T09:21:36.039252+0000","last_active":"2024-03-24T09:22:05.430230+0000","last_peered":"2024-03-24T09:22:05.430230+0000","last_clean":"2024-03-24T09:22:05.430230+0000","last_became_active":"2024-03-24T09:21:36.038637+0000","last_became_peered":"2024-03-24T09:21:36.038637+0000","last_unstale":"2024-03-24T09:22:05.430230+0000","last_undegraded":"2024-03-24T09:22:05.430230+0000","last_fullsized":"2024-03-24T09:22:05.430230+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:49.512786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.076395+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:22:06.076395+0000","last_peered":"2024-03-24T09:22:06.076395+0000","last_clean":"2024-03-24T09:22:06.076395+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:22:06.076395+0000","last_undegraded":"2024-03-24T09:22:06.076395+0000","last_fullsized":"2024-03-24T09:22:06.076395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2054,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431370+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:22:05.431370+0000","last_peered":"2024-03-24T09:22:05.431370+0000","last_clean":"2024-03-24T09:22:05.431370+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:22:05.431370+0000","last_undegraded":"2024-03-24T09:22:05.431370+0000","last_fullsized":"2024-03-24T09:22:05.431370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416208+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:22:05.416208+0000","last_peered":"2024-03-24T09:22:05.416208+0000","last_clean":"2024-03-24T09:22:05.416208+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:22:05.416208+0000","last_undegraded":"2024-03-24T09:22:05.416208+0000","last_fullsized":"2024-03-24T09:22:05.416208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.512009+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:22:05.512009+0000","last_peered":"2024-03-24T09:22:05.512009+0000","last_clean":"2024-03-24T09:22:05.512009+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:22:05.512009+0000","last_undegraded":"2024-03-24T09:22:05.512009+0000","last_fullsized":"2024-03-24T09:22:05.512009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431252+0000","last_change":"2024-03-24T09:20:35.031333+0000","last_active":"2024-03-24T09:22:05.431252+0000","last_peered":"2024-03-24T09:22:05.431252+0000","last_clean":"2024-03-24T09:22:05.431252+0000","last_became_active":"2024-03-24T09:20:35.030995+0000","last_became_peered":"2024-03-24T09:20:35.030995+0000","last_unstale":"2024-03-24T09:22:05.431252+0000","last_undegraded":"2024-03-24T09:22:05.431252+0000","last_fullsized":"2024-03-24T09:22:05.431252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:05.746740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391405+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:22:05.391405+0000","last_peered":"2024-03-24T09:22:05.391405+0000","last_clean":"2024-03-24T09:22:05.391405+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:22:05.391405+0000","last_undegraded":"2024-03-24T09:22:05.391405+0000","last_fullsized":"2024-03-24T09:22:05.391405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430787+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:22:05.430787+0000","last_peered":"2024-03-24T09:22:05.430787+0000","last_clean":"2024-03-24T09:22:05.430787+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:22:05.430787+0000","last_undegraded":"2024-03-24T09:22:05.430787+0000","last_fullsized":"2024-03-24T09:22:05.430787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431653+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:22:05.431653+0000","last_peered":"2024-03-24T09:22:05.431653+0000","last_clean":"2024-03-24T09:22:05.431653+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:22:05.431653+0000","last_undegraded":"2024-03-24T09:22:05.431653+0000","last_fullsized":"2024-03-24T09:22:05.431653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429626+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:22:05.429626+0000","last_peered":"2024-03-24T09:22:05.429626+0000","last_clean":"2024-03-24T09:22:05.429626+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:22:05.429626+0000","last_undegraded":"2024-03-24T09:22:05.429626+0000","last_fullsized":"2024-03-24T09:22:05.429626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265298+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:22:05.265298+0000","last_peered":"2024-03-24T09:22:05.265298+0000","last_clean":"2024-03-24T09:22:05.265298+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:22:05.265298+0000","last_undegraded":"2024-03-24T09:22:05.265298+0000","last_fullsized":"2024-03-24T09:22:05.265298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265210+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:22:05.265210+0000","last_peered":"2024-03-24T09:22:05.265210+0000","last_clean":"2024-03-24T09:22:05.265210+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:22:05.265210+0000","last_undegraded":"2024-03-24T09:22:05.265210+0000","last_fullsized":"2024-03-24T09:22:05.265210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416529+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:22:05.416529+0000","last_peered":"2024-03-24T09:22:05.416529+0000","last_clean":"2024-03-24T09:22:05.416529+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:22:05.416529+0000","last_undegraded":"2024-03-24T09:22:05.416529+0000","last_fullsized":"2024-03-24T09:22:05.416529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430754+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:22:05.430754+0000","last_peered":"2024-03-24T09:22:05.430754+0000","last_clean":"2024-03-24T09:22:05.430754+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:22:05.430754+0000","last_undegraded":"2024-03-24T09:22:05.430754+0000","last_fullsized":"2024-03-24T09:22:05.430754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430399+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:22:05.430399+0000","last_peered":"2024-03-24T09:22:05.430399+0000","last_clean":"2024-03-24T09:22:05.430399+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:22:05.430399+0000","last_undegraded":"2024-03-24T09:22:05.430399+0000","last_fullsized":"2024-03-24T09:22:05.430399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265749+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:22:05.265749+0000","last_peered":"2024-03-24T09:22:05.265749+0000","last_clean":"2024-03-24T09:22:05.265749+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:22:05.265749+0000","last_undegraded":"2024-03-24T09:22:05.265749+0000","last_fullsized":"2024-03-24T09:22:05.265749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262063+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:22:05.262063+0000","last_peered":"2024-03-24T09:22:05.262063+0000","last_clean":"2024-03-24T09:22:05.262063+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:22:05.262063+0000","last_undegraded":"2024-03-24T09:22:05.262063+0000","last_fullsized":"2024-03-24T09:22:05.262063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391993+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:22:05.391993+0000","last_peered":"2024-03-24T09:22:05.391993+0000","last_clean":"2024-03-24T09:22:05.391993+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:22:05.391993+0000","last_undegraded":"2024-03-24T09:22:05.391993+0000","last_fullsized":"2024-03-24T09:22:05.391993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261494+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:22:05.261494+0000","last_peered":"2024-03-24T09:22:05.261494+0000","last_clean":"2024-03-24T09:22:05.261494+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:22:05.261494+0000","last_undegraded":"2024-03-24T09:22:05.261494+0000","last_fullsized":"2024-03-24T09:22:05.261494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430502+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:22:05.430502+0000","last_peered":"2024-03-24T09:22:05.430502+0000","last_clean":"2024-03-24T09:22:05.430502+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:22:05.430502+0000","last_undegraded":"2024-03-24T09:22:05.430502+0000","last_fullsized":"2024-03-24T09:22:05.430502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430370+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:22:05.430370+0000","last_peered":"2024-03-24T09:22:05.430370+0000","last_clean":"2024-03-24T09:22:05.430370+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:22:05.430370+0000","last_undegraded":"2024-03-24T09:22:05.430370+0000","last_fullsized":"2024-03-24T09:22:05.430370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430156+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:22:05.430156+0000","last_peered":"2024-03-24T09:22:05.430156+0000","last_clean":"2024-03-24T09:22:05.430156+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:22:05.430156+0000","last_undegraded":"2024-03-24T09:22:05.430156+0000","last_fullsized":"2024-03-24T09:22:05.430156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107004,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604641,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1352,"num_read_kb":795,"num_write":380,"num_write_kb":1476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1813923,"data_compressed":80715,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":531197,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1172328,"data_compressed":63929,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1093632,"data_stored":927555,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1085440,"data_stored":923837,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":1129270,"data_compressed":65311,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526240,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":596464,"data_compressed":31246,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":528087,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":598100,"data_compressed":31246,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1050271,"data_compressed":54945,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":531628,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1142784,"data_stored":1254934,"data_compressed":71467,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71439,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":73075,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135445,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":204337,"data_compressed":7759,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:22:09.634 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-24T09:22:09.635 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 -- 172.21.15.116:0/350571885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 msgr2=0x7f4e741045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/350571885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e741045a0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009900 tx=0x7f4e6402bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 -- 172.21.15.116:0/350571885 shutdown_connections 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/350571885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e74107410 0x7f4e74109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/350571885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 0x7f4e74106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/350571885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e741045a0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 -- 172.21.15.116:0/350571885 >> 172.21.15.116:0/350571885 conn(0x7f4e740fbba0 msgr2=0x7f4e740fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 -- 172.21.15.116:0/350571885 shutdown_connections 2024-03-24T09:22:09.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.779+0000 7f4e79dc7640 1 -- 172.21.15.116:0/350571885 wait complete. 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 Processor -- start 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 -- start start 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e74197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 0x7f4e74197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e74107410 0x7f4e7419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7419f1f0 con 0x7f4e741021b0 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7419f360 con 0x7f4e74104ae0 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e79dc7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7419f520 con 0x7f4e74107410 2024-03-24T09:22:09.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e74197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e72ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 0x7f4e74197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e74197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55892/0 (socket says 172.21.15.116:55892) 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e72ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 0x7f4e74197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:53086/0 (socket says 172.21.15.116:53086) 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e737fe640 1 -- 172.21.15.116:0/2781154433 learned_addr learned my addr 172.21.15.116:0/2781154433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.780+0000 7f4e73fff640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e74107410 0x7f4e7419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e737fe640 1 -- 172.21.15.116:0/2781154433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e74107410 msgr2=0x7f4e7419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e737fe640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e74107410 0x7f4e7419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e737fe640 1 -- 172.21.15.116:0/2781154433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 msgr2=0x7f4e74197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e737fe640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 0x7f4e74197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e737fe640 1 -- 172.21.15.116:0/2781154433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e64009560 con 0x7f4e741021b0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e737fe640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e74197700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4e64005ab0 tx=0x7f4e640040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e640044c0 con 0x7f4e741021b0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4e64003aa0 con 0x7f4e741021b0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e64005580 con 0x7f4e741021b0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7419f7a0 con 0x7f4e741021b0 2024-03-24T09:22:09.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.781+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7419fcb0 con 0x7f4e741021b0 2024-03-24T09:22:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.782+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e740fd690 con 0x7f4e741021b0 2024-03-24T09:22:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.783+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136877+0+0 (secure 0 0 0) 0x7f4e64004dc0 con 0x7f4e741021b0 2024-03-24T09:22:09.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.784+0000 7f4e70ff9640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4e5c0a2230 0x7f4e5c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.784+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4e6402de40 con 0x7f4e741021b0 2024-03-24T09:22:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.784+0000 7f4e72ffd640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4e5c0a2230 0x7f4e5c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:09.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.785+0000 7f4e72ffd640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4e5c0a2230 0x7f4e5c0a46f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e74198c10 tx=0x7f4e68005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:09.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.788+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4e640b95c0 con 0x7f4e741021b0 2024-03-24T09:22:09.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.969+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4e7402d1d0 con 0x7f4e5c0a2230 2024-03-24T09:22:09.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.987+0000 7f4e70ff9640 1 -- 172.21.15.116:0/2781154433 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339740 (secure 0 0 0) 0x7f4e7402d1d0 con 0x7f4e5c0a2230 2024-03-24T09:22:09.991 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:09.992 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4e5c0a2230 msgr2=0x7f4e5c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4e5c0a2230 0x7f4e5c0a46f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e74198c10 tx=0x7f4e68005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 msgr2=0x7f4e74197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e74197700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f4e64005ab0 tx=0x7f4e640040c0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 shutdown_connections 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4e5c0a2230 0x7f4e5c0a46f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4e74107410 0x7f4e7419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e74104ae0 0x7f4e74197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 --2- 172.21.15.116:0/2781154433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e741021b0 0x7f4e74197700 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 >> 172.21.15.116:0/2781154433 conn(0x7f4e740fbba0 msgr2=0x7f4e74102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 shutdown_connections 2024-03-24T09:22:09.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:09.990+0000 7f4e79dc7640 1 -- 172.21.15.116:0/2781154433 wait complete. 2024-03-24T09:22:10.003 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-03-24T09:22:09.274608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1195243,"num_objects":104,"num_object_clones":0,"num_object_copies":314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1488,"num_read_kb":914,"num_write":563,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":948,"ondisk_log_size":948,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":378676,"kb_used_data":12836,"kb_used_omap":18,"kb_used_meta":365677,"kb_avail":1124563148,"statfs":{"total":1151940427776,"available":1151552663552,"internally_reserved":0,"allocated":13144064,"data_stored":9769911,"data_compressed":534294,"data_compressed_allocated":724992,"data_compressed_original":4194304,"omap_allocated":19143,"internal_metadata":374453561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.987864"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430660+0000","last_change":"2024-03-24T09:19:38.735596+0000","last_active":"2024-03-24T09:22:05.430660+0000","last_peered":"2024-03-24T09:22:05.430660+0000","last_clean":"2024-03-24T09:22:05.430660+0000","last_became_active":"2024-03-24T09:19:38.734357+0000","last_became_peered":"2024-03-24T09:19:38.734357+0000","last_unstale":"2024-03-24T09:22:05.430660+0000","last_undegraded":"2024-03-24T09:22:05.430660+0000","last_fullsized":"2024-03-24T09:22:05.430660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:44.939298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417006+0000","last_change":"2024-03-24T09:19:38.723898+0000","last_active":"2024-03-24T09:22:05.417006+0000","last_peered":"2024-03-24T09:22:05.417006+0000","last_clean":"2024-03-24T09:22:05.417006+0000","last_became_active":"2024-03-24T09:19:38.723461+0000","last_became_peered":"2024-03-24T09:19:38.723461+0000","last_unstale":"2024-03-24T09:22:05.417006+0000","last_undegraded":"2024-03-24T09:22:05.417006+0000","last_fullsized":"2024-03-24T09:22:05.417006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:17.243810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416543+0000","last_change":"2024-03-24T09:19:38.738221+0000","last_active":"2024-03-24T09:22:05.416543+0000","last_peered":"2024-03-24T09:22:05.416543+0000","last_clean":"2024-03-24T09:22:05.416543+0000","last_became_active":"2024-03-24T09:19:38.737966+0000","last_became_peered":"2024-03-24T09:19:38.737966+0000","last_unstale":"2024-03-24T09:22:05.416543+0000","last_undegraded":"2024-03-24T09:22:05.416543+0000","last_fullsized":"2024-03-24T09:22:05.416543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:17:55.984096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.264004+0000","last_change":"2024-03-24T09:19:38.730042+0000","last_active":"2024-03-24T09:22:05.264004+0000","last_peered":"2024-03-24T09:22:05.264004+0000","last_clean":"2024-03-24T09:22:05.264004+0000","last_became_active":"2024-03-24T09:19:38.729511+0000","last_became_peered":"2024-03-24T09:19:38.729511+0000","last_unstale":"2024-03-24T09:22:05.264004+0000","last_undegraded":"2024-03-24T09:22:05.264004+0000","last_fullsized":"2024-03-24T09:22:05.264004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:06.372268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383474+0000","last_change":"2024-03-24T09:19:38.729740+0000","last_active":"2024-03-24T09:22:05.383474+0000","last_peered":"2024-03-24T09:22:05.383474+0000","last_clean":"2024-03-24T09:22:05.383474+0000","last_became_active":"2024-03-24T09:19:38.729418+0000","last_became_peered":"2024-03-24T09:19:38.729418+0000","last_unstale":"2024-03-24T09:22:05.383474+0000","last_undegraded":"2024-03-24T09:22:05.383474+0000","last_fullsized":"2024-03-24T09:22:05.383474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:35:42.012187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395243+0000","last_change":"2024-03-24T09:20:35.029608+0000","last_active":"2024-03-24T09:22:05.395243+0000","last_peered":"2024-03-24T09:22:05.395243+0000","last_clean":"2024-03-24T09:22:05.395243+0000","last_became_active":"2024-03-24T09:20:35.029268+0000","last_became_peered":"2024-03-24T09:20:35.029268+0000","last_unstale":"2024-03-24T09:22:05.395243+0000","last_undegraded":"2024-03-24T09:22:05.395243+0000","last_fullsized":"2024-03-24T09:22:05.395243+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:06:47.087125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430275+0000","last_change":"2024-03-24T09:19:38.728276+0000","last_active":"2024-03-24T09:22:05.430275+0000","last_peered":"2024-03-24T09:22:05.430275+0000","last_clean":"2024-03-24T09:22:05.430275+0000","last_became_active":"2024-03-24T09:19:38.727912+0000","last_became_peered":"2024-03-24T09:19:38.727912+0000","last_unstale":"2024-03-24T09:22:05.430275+0000","last_undegraded":"2024-03-24T09:22:05.430275+0000","last_fullsized":"2024-03-24T09:22:05.430275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:34:45.510212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261421+0000","last_change":"2024-03-24T09:19:38.728684+0000","last_active":"2024-03-24T09:22:05.261421+0000","last_peered":"2024-03-24T09:22:05.261421+0000","last_clean":"2024-03-24T09:22:05.261421+0000","last_became_active":"2024-03-24T09:19:38.728366+0000","last_became_peered":"2024-03-24T09:19:38.728366+0000","last_unstale":"2024-03-24T09:22:05.261421+0000","last_undegraded":"2024-03-24T09:22:05.261421+0000","last_fullsized":"2024-03-24T09:22:05.261421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:20:41.133857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431980+0000","last_change":"2024-03-24T09:19:38.723798+0000","last_active":"2024-03-24T09:22:05.431980+0000","last_peered":"2024-03-24T09:22:05.431980+0000","last_clean":"2024-03-24T09:22:05.431980+0000","last_became_active":"2024-03-24T09:19:38.723446+0000","last_became_peered":"2024-03-24T09:19:38.723446+0000","last_unstale":"2024-03-24T09:22:05.431980+0000","last_undegraded":"2024-03-24T09:22:05.431980+0000","last_fullsized":"2024-03-24T09:22:05.431980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:49:37.380164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395656+0000","last_change":"2024-03-24T09:19:38.734113+0000","last_active":"2024-03-24T09:22:05.395656+0000","last_peered":"2024-03-24T09:22:05.395656+0000","last_clean":"2024-03-24T09:22:05.395656+0000","last_became_active":"2024-03-24T09:19:38.733723+0000","last_became_peered":"2024-03-24T09:19:38.733723+0000","last_unstale":"2024-03-24T09:22:05.395656+0000","last_undegraded":"2024-03-24T09:22:05.395656+0000","last_fullsized":"2024-03-24T09:22:05.395656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:53:07.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431628+0000","last_change":"2024-03-24T09:19:38.726767+0000","last_active":"2024-03-24T09:22:05.431628+0000","last_peered":"2024-03-24T09:22:05.431628+0000","last_clean":"2024-03-24T09:22:05.431628+0000","last_became_active":"2024-03-24T09:19:38.726423+0000","last_became_peered":"2024-03-24T09:19:38.726423+0000","last_unstale":"2024-03-24T09:22:05.431628+0000","last_undegraded":"2024-03-24T09:22:05.431628+0000","last_fullsized":"2024-03-24T09:22:05.431628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:14.040456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263913+0000","last_change":"2024-03-24T09:19:38.732039+0000","last_active":"2024-03-24T09:22:05.263913+0000","last_peered":"2024-03-24T09:22:05.263913+0000","last_clean":"2024-03-24T09:22:05.263913+0000","last_became_active":"2024-03-24T09:19:38.731671+0000","last_became_peered":"2024-03-24T09:19:38.731671+0000","last_unstale":"2024-03-24T09:22:05.263913+0000","last_undegraded":"2024-03-24T09:22:05.263913+0000","last_fullsized":"2024-03-24T09:22:05.263913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:05:18.544959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430584+0000","last_change":"2024-03-24T09:19:38.722205+0000","last_active":"2024-03-24T09:22:05.430584+0000","last_peered":"2024-03-24T09:22:05.430584+0000","last_clean":"2024-03-24T09:22:05.430584+0000","last_became_active":"2024-03-24T09:19:38.721831+0000","last_became_peered":"2024-03-24T09:19:38.721831+0000","last_unstale":"2024-03-24T09:22:05.430584+0000","last_undegraded":"2024-03-24T09:22:05.430584+0000","last_fullsized":"2024-03-24T09:22:05.430584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:10.605921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416250+0000","last_change":"2024-03-24T09:19:38.725569+0000","last_active":"2024-03-24T09:22:05.416250+0000","last_peered":"2024-03-24T09:22:05.416250+0000","last_clean":"2024-03-24T09:22:05.416250+0000","last_became_active":"2024-03-24T09:19:38.725223+0000","last_became_peered":"2024-03-24T09:19:38.725223+0000","last_unstale":"2024-03-24T09:22:05.416250+0000","last_undegraded":"2024-03-24T09:22:05.416250+0000","last_fullsized":"2024-03-24T09:22:05.416250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:08:11.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416249+0000","last_change":"2024-03-24T09:19:38.729006+0000","last_active":"2024-03-24T09:22:05.416249+0000","last_peered":"2024-03-24T09:22:05.416249+0000","last_clean":"2024-03-24T09:22:05.416249+0000","last_became_active":"2024-03-24T09:19:38.728673+0000","last_became_peered":"2024-03-24T09:19:38.728673+0000","last_unstale":"2024-03-24T09:22:05.416249+0000","last_undegraded":"2024-03-24T09:22:05.416249+0000","last_fullsized":"2024-03-24T09:22:05.416249+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:34.163388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429842+0000","last_change":"2024-03-24T09:19:38.730642+0000","last_active":"2024-03-24T09:22:05.429842+0000","last_peered":"2024-03-24T09:22:05.429842+0000","last_clean":"2024-03-24T09:22:05.429842+0000","last_became_active":"2024-03-24T09:19:38.730339+0000","last_became_peered":"2024-03-24T09:19:38.730339+0000","last_unstale":"2024-03-24T09:22:05.429842+0000","last_undegraded":"2024-03-24T09:22:05.429842+0000","last_fullsized":"2024-03-24T09:22:05.429842+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:24.316267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430550+0000","last_change":"2024-03-24T09:19:38.735813+0000","last_active":"2024-03-24T09:22:05.430550+0000","last_peered":"2024-03-24T09:22:05.430550+0000","last_clean":"2024-03-24T09:22:05.430550+0000","last_became_active":"2024-03-24T09:19:38.735188+0000","last_became_peered":"2024-03-24T09:19:38.735188+0000","last_unstale":"2024-03-24T09:22:05.430550+0000","last_undegraded":"2024-03-24T09:22:05.430550+0000","last_fullsized":"2024-03-24T09:22:05.430550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:54:01.580650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381311+0000","last_change":"2024-03-24T09:19:38.724395+0000","last_active":"2024-03-24T09:22:05.381311+0000","last_peered":"2024-03-24T09:22:05.381311+0000","last_clean":"2024-03-24T09:22:05.381311+0000","last_became_active":"2024-03-24T09:19:38.724078+0000","last_became_peered":"2024-03-24T09:19:38.724078+0000","last_unstale":"2024-03-24T09:22:05.381311+0000","last_undegraded":"2024-03-24T09:22:05.381311+0000","last_fullsized":"2024-03-24T09:22:05.381311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:50:19.196784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265648+0000","last_change":"2024-03-24T09:19:38.729891+0000","last_active":"2024-03-24T09:22:05.265648+0000","last_peered":"2024-03-24T09:22:05.265648+0000","last_clean":"2024-03-24T09:22:05.265648+0000","last_became_active":"2024-03-24T09:19:38.728951+0000","last_became_peered":"2024-03-24T09:19:38.728951+0000","last_unstale":"2024-03-24T09:22:05.265648+0000","last_undegraded":"2024-03-24T09:22:05.265648+0000","last_fullsized":"2024-03-24T09:22:05.265648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:18:55.905725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265189+0000","last_change":"2024-03-24T09:19:42.746980+0000","last_active":"2024-03-24T09:22:05.265189+0000","last_peered":"2024-03-24T09:22:05.265189+0000","last_clean":"2024-03-24T09:22:05.265189+0000","last_became_active":"2024-03-24T09:19:42.746536+0000","last_became_peered":"2024-03-24T09:19:42.746536+0000","last_unstale":"2024-03-24T09:22:05.265189+0000","last_undegraded":"2024-03-24T09:22:05.265189+0000","last_fullsized":"2024-03-24T09:22:05.265189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:19:17.777487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"93'14","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381586+0000","last_change":"2024-03-24T09:19:38.099701+0000","last_active":"2024-03-24T09:22:05.381586+0000","last_peered":"2024-03-24T09:22:05.381586+0000","last_clean":"2024-03-24T09:22:05.381586+0000","last_became_active":"2024-03-24T09:19:38.099371+0000","last_became_peered":"2024-03-24T09:19:38.099371+0000","last_unstale":"2024-03-24T09:22:05.381586+0000","last_undegraded":"2024-03-24T09:22:05.381586+0000","last_fullsized":"2024-03-24T09:22:05.381586+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:14:46.658987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262126+0000","last_change":"2024-03-24T09:19:38.731940+0000","last_active":"2024-03-24T09:22:05.262126+0000","last_peered":"2024-03-24T09:22:05.262126+0000","last_clean":"2024-03-24T09:22:05.262126+0000","last_became_active":"2024-03-24T09:19:38.731622+0000","last_became_peered":"2024-03-24T09:19:38.731622+0000","last_unstale":"2024-03-24T09:22:05.262126+0000","last_undegraded":"2024-03-24T09:22:05.262126+0000","last_fullsized":"2024-03-24T09:22:05.262126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:55:38.281715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262833+0000","last_change":"2024-03-24T09:20:35.302025+0000","last_active":"2024-03-24T09:22:05.262833+0000","last_peered":"2024-03-24T09:22:05.262833+0000","last_clean":"2024-03-24T09:22:05.262833+0000","last_became_active":"2024-03-24T09:20:35.301575+0000","last_became_peered":"2024-03-24T09:20:35.301575+0000","last_unstale":"2024-03-24T09:22:05.262833+0000","last_undegraded":"2024-03-24T09:22:05.262833+0000","last_fullsized":"2024-03-24T09:22:05.262833+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:20.305175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262343+0000","last_change":"2024-03-24T09:19:38.104557+0000","last_active":"2024-03-24T09:22:05.262343+0000","last_peered":"2024-03-24T09:22:05.262343+0000","last_clean":"2024-03-24T09:22:05.262343+0000","last_became_active":"2024-03-24T09:19:38.104133+0000","last_became_peered":"2024-03-24T09:19:38.104133+0000","last_unstale":"2024-03-24T09:22:05.262343+0000","last_undegraded":"2024-03-24T09:22:05.262343+0000","last_fullsized":"2024-03-24T09:22:05.262343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:47.561650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417089+0000","last_change":"2024-03-24T09:19:38.735100+0000","last_active":"2024-03-24T09:22:05.417089+0000","last_peered":"2024-03-24T09:22:05.417089+0000","last_clean":"2024-03-24T09:22:05.417089+0000","last_became_active":"2024-03-24T09:19:38.734686+0000","last_became_peered":"2024-03-24T09:19:38.734686+0000","last_unstale":"2024-03-24T09:22:05.417089+0000","last_undegraded":"2024-03-24T09:22:05.417089+0000","last_fullsized":"2024-03-24T09:22:05.417089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:29:42.439850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263181+0000","last_change":"2024-03-24T09:19:42.746559+0000","last_active":"2024-03-24T09:22:05.263181+0000","last_peered":"2024-03-24T09:22:05.263181+0000","last_clean":"2024-03-24T09:22:05.263181+0000","last_became_active":"2024-03-24T09:19:42.746209+0000","last_became_peered":"2024-03-24T09:19:42.746209+0000","last_unstale":"2024-03-24T09:22:05.263181+0000","last_undegraded":"2024-03-24T09:22:05.263181+0000","last_fullsized":"2024-03-24T09:22:05.263181+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.662298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384209+0000","last_change":"2024-03-24T09:19:38.104231+0000","last_active":"2024-03-24T09:22:05.384209+0000","last_peered":"2024-03-24T09:22:05.384209+0000","last_clean":"2024-03-24T09:22:05.384209+0000","last_became_active":"2024-03-24T09:19:38.103812+0000","last_became_peered":"2024-03-24T09:19:38.103812+0000","last_unstale":"2024-03-24T09:22:05.384209+0000","last_undegraded":"2024-03-24T09:22:05.384209+0000","last_fullsized":"2024-03-24T09:22:05.384209+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:47:43.404750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430840+0000","last_change":"2024-03-24T09:19:38.732753+0000","last_active":"2024-03-24T09:22:05.430840+0000","last_peered":"2024-03-24T09:22:05.430840+0000","last_clean":"2024-03-24T09:22:05.430840+0000","last_became_active":"2024-03-24T09:19:38.732282+0000","last_became_peered":"2024-03-24T09:19:38.732282+0000","last_unstale":"2024-03-24T09:22:05.430840+0000","last_undegraded":"2024-03-24T09:22:05.430840+0000","last_fullsized":"2024-03-24T09:22:05.430840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.748159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263015+0000","last_change":"2024-03-24T09:19:42.746331+0000","last_active":"2024-03-24T09:22:05.263015+0000","last_peered":"2024-03-24T09:22:05.263015+0000","last_clean":"2024-03-24T09:22:05.263015+0000","last_became_active":"2024-03-24T09:19:42.745932+0000","last_became_peered":"2024-03-24T09:19:42.745932+0000","last_unstale":"2024-03-24T09:22:05.263015+0000","last_undegraded":"2024-03-24T09:22:05.263015+0000","last_fullsized":"2024-03-24T09:22:05.263015+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:17:02.935273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383756+0000","last_change":"2024-03-24T09:19:38.095379+0000","last_active":"2024-03-24T09:22:05.383756+0000","last_peered":"2024-03-24T09:22:05.383756+0000","last_clean":"2024-03-24T09:22:05.383756+0000","last_became_active":"2024-03-24T09:19:38.094878+0000","last_became_peered":"2024-03-24T09:19:38.094878+0000","last_unstale":"2024-03-24T09:22:05.383756+0000","last_undegraded":"2024-03-24T09:22:05.383756+0000","last_fullsized":"2024-03-24T09:22:05.383756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:18.555890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391949+0000","last_change":"2024-03-24T09:20:35.032271+0000","last_active":"2024-03-24T09:22:05.391949+0000","last_peered":"2024-03-24T09:22:05.391949+0000","last_clean":"2024-03-24T09:22:05.391949+0000","last_became_active":"2024-03-24T09:20:35.031913+0000","last_became_peered":"2024-03-24T09:20:35.031913+0000","last_unstale":"2024-03-24T09:22:05.391949+0000","last_undegraded":"2024-03-24T09:22:05.391949+0000","last_fullsized":"2024-03-24T09:22:05.391949+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:36:40.322197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265519+0000","last_change":"2024-03-24T09:19:42.744920+0000","last_active":"2024-03-24T09:22:05.265519+0000","last_peered":"2024-03-24T09:22:05.265519+0000","last_clean":"2024-03-24T09:22:05.265519+0000","last_became_active":"2024-03-24T09:19:42.743809+0000","last_became_peered":"2024-03-24T09:19:42.743809+0000","last_unstale":"2024-03-24T09:22:05.265519+0000","last_undegraded":"2024-03-24T09:22:05.265519+0000","last_fullsized":"2024-03-24T09:22:05.265519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:56:38.431217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392028+0000","last_change":"2024-03-24T09:19:38.101774+0000","last_active":"2024-03-24T09:22:05.392028+0000","last_peered":"2024-03-24T09:22:05.392028+0000","last_clean":"2024-03-24T09:22:05.392028+0000","last_became_active":"2024-03-24T09:19:38.101415+0000","last_became_peered":"2024-03-24T09:19:38.101415+0000","last_unstale":"2024-03-24T09:22:05.392028+0000","last_undegraded":"2024-03-24T09:22:05.392028+0000","last_fullsized":"2024-03-24T09:22:05.392028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:17:10.086612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383732+0000","last_change":"2024-03-24T09:20:35.031176+0000","last_active":"2024-03-24T09:22:05.383732+0000","last_peered":"2024-03-24T09:22:05.383732+0000","last_clean":"2024-03-24T09:22:05.383732+0000","last_became_active":"2024-03-24T09:20:35.030888+0000","last_became_peered":"2024-03-24T09:20:35.030888+0000","last_unstale":"2024-03-24T09:22:05.383732+0000","last_undegraded":"2024-03-24T09:22:05.383732+0000","last_fullsized":"2024-03-24T09:22:05.383732+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:35:43.991847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265449+0000","last_change":"2024-03-24T09:19:42.745025+0000","last_active":"2024-03-24T09:22:05.265449+0000","last_peered":"2024-03-24T09:22:05.265449+0000","last_clean":"2024-03-24T09:22:05.265449+0000","last_became_active":"2024-03-24T09:19:42.744485+0000","last_became_peered":"2024-03-24T09:19:42.744485+0000","last_unstale":"2024-03-24T09:22:05.265449+0000","last_undegraded":"2024-03-24T09:22:05.265449+0000","last_fullsized":"2024-03-24T09:22:05.265449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:55:29.065183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395214+0000","last_change":"2024-03-24T09:19:38.104181+0000","last_active":"2024-03-24T09:22:05.395214+0000","last_peered":"2024-03-24T09:22:05.395214+0000","last_clean":"2024-03-24T09:22:05.395214+0000","last_became_active":"2024-03-24T09:19:38.103966+0000","last_became_peered":"2024-03-24T09:19:38.103966+0000","last_unstale":"2024-03-24T09:22:05.395214+0000","last_undegraded":"2024-03-24T09:22:05.395214+0000","last_fullsized":"2024-03-24T09:22:05.395214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:30.960998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391824+0000","last_change":"2024-03-24T09:20:35.301944+0000","last_active":"2024-03-24T09:22:05.391824+0000","last_peered":"2024-03-24T09:22:05.391824+0000","last_clean":"2024-03-24T09:22:05.391824+0000","last_became_active":"2024-03-24T09:20:35.301588+0000","last_became_peered":"2024-03-24T09:20:35.301588+0000","last_unstale":"2024-03-24T09:22:05.391824+0000","last_undegraded":"2024-03-24T09:22:05.391824+0000","last_fullsized":"2024-03-24T09:22:05.391824+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:33.185748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430406+0000","last_change":"2024-03-24T09:19:42.746517+0000","last_active":"2024-03-24T09:22:05.430406+0000","last_peered":"2024-03-24T09:22:05.430406+0000","last_clean":"2024-03-24T09:22:05.430406+0000","last_became_active":"2024-03-24T09:19:42.746071+0000","last_became_peered":"2024-03-24T09:19:42.746071+0000","last_unstale":"2024-03-24T09:22:05.430406+0000","last_undegraded":"2024-03-24T09:22:05.430406+0000","last_fullsized":"2024-03-24T09:22:05.430406+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:08.386959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430255+0000","last_change":"2024-03-24T09:19:38.097600+0000","last_active":"2024-03-24T09:22:05.430255+0000","last_peered":"2024-03-24T09:22:05.430255+0000","last_clean":"2024-03-24T09:22:05.430255+0000","last_became_active":"2024-03-24T09:19:38.097279+0000","last_became_peered":"2024-03-24T09:19:38.097279+0000","last_unstale":"2024-03-24T09:22:05.430255+0000","last_undegraded":"2024-03-24T09:22:05.430255+0000","last_fullsized":"2024-03-24T09:22:05.430255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:49.984958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261576+0000","last_change":"2024-03-24T09:19:38.733232+0000","last_active":"2024-03-24T09:22:05.261576+0000","last_peered":"2024-03-24T09:22:05.261576+0000","last_clean":"2024-03-24T09:22:05.261576+0000","last_became_active":"2024-03-24T09:19:38.732876+0000","last_became_peered":"2024-03-24T09:19:38.732876+0000","last_unstale":"2024-03-24T09:22:05.261576+0000","last_undegraded":"2024-03-24T09:22:05.261576+0000","last_fullsized":"2024-03-24T09:22:05.261576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:16:16.709512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416766+0000","last_change":"2024-03-24T09:19:42.750554+0000","last_active":"2024-03-24T09:22:05.416766+0000","last_peered":"2024-03-24T09:22:05.416766+0000","last_clean":"2024-03-24T09:22:05.416766+0000","last_became_active":"2024-03-24T09:19:42.750181+0000","last_became_peered":"2024-03-24T09:19:42.750181+0000","last_unstale":"2024-03-24T09:22:05.416766+0000","last_undegraded":"2024-03-24T09:22:05.416766+0000","last_fullsized":"2024-03-24T09:22:05.416766+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:40:44.267125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392042+0000","last_change":"2024-03-24T09:19:38.101838+0000","last_active":"2024-03-24T09:22:05.392042+0000","last_peered":"2024-03-24T09:22:05.392042+0000","last_clean":"2024-03-24T09:22:05.392042+0000","last_became_active":"2024-03-24T09:19:38.101442+0000","last_became_peered":"2024-03-24T09:19:38.101442+0000","last_unstale":"2024-03-24T09:22:05.392042+0000","last_undegraded":"2024-03-24T09:22:05.392042+0000","last_fullsized":"2024-03-24T09:22:05.392042+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:51:39.419418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380573+0000","last_change":"2024-03-24T09:19:38.728844+0000","last_active":"2024-03-24T09:22:05.380573+0000","last_peered":"2024-03-24T09:22:05.380573+0000","last_clean":"2024-03-24T09:22:05.380573+0000","last_became_active":"2024-03-24T09:19:38.728524+0000","last_became_peered":"2024-03-24T09:19:38.728524+0000","last_unstale":"2024-03-24T09:22:05.380573+0000","last_undegraded":"2024-03-24T09:22:05.380573+0000","last_fullsized":"2024-03-24T09:22:05.380573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:32.020438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261827+0000","last_change":"2024-03-24T09:19:42.745062+0000","last_active":"2024-03-24T09:22:05.261827+0000","last_peered":"2024-03-24T09:22:05.261827+0000","last_clean":"2024-03-24T09:22:05.261827+0000","last_became_active":"2024-03-24T09:19:42.744643+0000","last_became_peered":"2024-03-24T09:19:42.744643+0000","last_unstale":"2024-03-24T09:22:05.261827+0000","last_undegraded":"2024-03-24T09:22:05.261827+0000","last_fullsized":"2024-03-24T09:22:05.261827+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:41:29.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'7","reported_seq":284,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.075787+0000","last_change":"2024-03-24T09:19:38.103210+0000","last_active":"2024-03-24T09:22:06.075787+0000","last_peered":"2024-03-24T09:22:06.075787+0000","last_clean":"2024-03-24T09:22:06.075787+0000","last_became_active":"2024-03-24T09:19:38.102816+0000","last_became_peered":"2024-03-24T09:19:38.102816+0000","last_unstale":"2024-03-24T09:22:06.075787+0000","last_undegraded":"2024-03-24T09:22:06.075787+0000","last_fullsized":"2024-03-24T09:22:06.075787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:01:09.811308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":243,"num_read_kb":243,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431671+0000","last_change":"2024-03-24T09:19:38.735348+0000","last_active":"2024-03-24T09:22:05.431671+0000","last_peered":"2024-03-24T09:22:05.431671+0000","last_clean":"2024-03-24T09:22:05.431671+0000","last_became_active":"2024-03-24T09:19:38.733839+0000","last_became_peered":"2024-03-24T09:19:38.733839+0000","last_unstale":"2024-03-24T09:22:05.431671+0000","last_undegraded":"2024-03-24T09:22:05.431671+0000","last_fullsized":"2024-03-24T09:22:05.431671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:51:22.712941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380431+0000","last_change":"2024-03-24T09:19:42.749091+0000","last_active":"2024-03-24T09:22:05.380431+0000","last_peered":"2024-03-24T09:22:05.380431+0000","last_clean":"2024-03-24T09:22:05.380431+0000","last_became_active":"2024-03-24T09:19:42.748696+0000","last_became_peered":"2024-03-24T09:19:42.748696+0000","last_unstale":"2024-03-24T09:22:05.380431+0000","last_undegraded":"2024-03-24T09:22:05.380431+0000","last_fullsized":"2024-03-24T09:22:05.380431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:34:28.966626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431572+0000","last_change":"2024-03-24T09:19:38.099733+0000","last_active":"2024-03-24T09:22:05.431572+0000","last_peered":"2024-03-24T09:22:05.431572+0000","last_clean":"2024-03-24T09:22:05.431572+0000","last_became_active":"2024-03-24T09:19:38.099364+0000","last_became_peered":"2024-03-24T09:19:38.099364+0000","last_unstale":"2024-03-24T09:22:05.431572+0000","last_undegraded":"2024-03-24T09:22:05.431572+0000","last_fullsized":"2024-03-24T09:22:05.431572+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:25:57.361916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.417192+0000","last_change":"2024-03-24T09:20:35.301994+0000","last_active":"2024-03-24T09:22:05.417192+0000","last_peered":"2024-03-24T09:22:05.417192+0000","last_clean":"2024-03-24T09:22:05.417192+0000","last_became_active":"2024-03-24T09:20:35.301570+0000","last_became_peered":"2024-03-24T09:20:35.301570+0000","last_unstale":"2024-03-24T09:22:05.417192+0000","last_undegraded":"2024-03-24T09:22:05.417192+0000","last_fullsized":"2024-03-24T09:22:05.417192+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:13:34.896090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263428+0000","last_change":"2024-03-24T09:19:42.738993+0000","last_active":"2024-03-24T09:22:05.263428+0000","last_peered":"2024-03-24T09:22:05.263428+0000","last_clean":"2024-03-24T09:22:05.263428+0000","last_became_active":"2024-03-24T09:19:42.738497+0000","last_became_peered":"2024-03-24T09:19:42.738497+0000","last_unstale":"2024-03-24T09:22:05.263428+0000","last_undegraded":"2024-03-24T09:22:05.263428+0000","last_fullsized":"2024-03-24T09:22:05.263428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:48.092178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391847+0000","last_change":"2024-03-24T09:19:38.100323+0000","last_active":"2024-03-24T09:22:05.391847+0000","last_peered":"2024-03-24T09:22:05.391847+0000","last_clean":"2024-03-24T09:22:05.391847+0000","last_became_active":"2024-03-24T09:19:38.099668+0000","last_became_peered":"2024-03-24T09:19:38.099668+0000","last_unstale":"2024-03-24T09:22:05.391847+0000","last_undegraded":"2024-03-24T09:22:05.391847+0000","last_fullsized":"2024-03-24T09:22:05.391847+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:45:17.173370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":28,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380857+0000","last_change":"2024-03-24T09:20:35.027718+0000","last_active":"2024-03-24T09:22:05.380857+0000","last_peered":"2024-03-24T09:22:05.380857+0000","last_clean":"2024-03-24T09:22:05.380857+0000","last_became_active":"2024-03-24T09:20:35.027347+0000","last_became_peered":"2024-03-24T09:20:35.027347+0000","last_unstale":"2024-03-24T09:22:05.380857+0000","last_undegraded":"2024-03-24T09:22:05.380857+0000","last_fullsized":"2024-03-24T09:22:05.380857+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:19:20.097882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395060+0000","last_change":"2024-03-24T09:20:35.301625+0000","last_active":"2024-03-24T09:22:05.395060+0000","last_peered":"2024-03-24T09:22:05.395060+0000","last_clean":"2024-03-24T09:22:05.395060+0000","last_became_active":"2024-03-24T09:20:35.301035+0000","last_became_peered":"2024-03-24T09:20:35.301035+0000","last_unstale":"2024-03-24T09:22:05.395060+0000","last_undegraded":"2024-03-24T09:22:05.395060+0000","last_fullsized":"2024-03-24T09:22:05.395060+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:09.448556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":333,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.021394+0000","last_change":"2024-03-24T09:19:38.097922+0000","last_active":"2024-03-24T09:22:06.021394+0000","last_peered":"2024-03-24T09:22:06.021394+0000","last_clean":"2024-03-24T09:22:06.021394+0000","last_became_active":"2024-03-24T09:19:38.097606+0000","last_became_peered":"2024-03-24T09:19:38.097606+0000","last_unstale":"2024-03-24T09:22:06.021394+0000","last_undegraded":"2024-03-24T09:22:06.021394+0000","last_fullsized":"2024-03-24T09:22:06.021394+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:23:28.142282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69194,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":257,"num_read_kb":196,"num_write":44,"num_write_kb":88,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384057+0000","last_change":"2024-03-24T09:19:38.728650+0000","last_active":"2024-03-24T09:22:05.384057+0000","last_peered":"2024-03-24T09:22:05.384057+0000","last_clean":"2024-03-24T09:22:05.384057+0000","last_became_active":"2024-03-24T09:19:38.728317+0000","last_became_peered":"2024-03-24T09:19:38.728317+0000","last_unstale":"2024-03-24T09:22:05.384057+0000","last_undegraded":"2024-03-24T09:22:05.384057+0000","last_fullsized":"2024-03-24T09:22:05.384057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:10:52.467920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392229+0000","last_change":"2024-03-24T09:19:42.745156+0000","last_active":"2024-03-24T09:22:05.392229+0000","last_peered":"2024-03-24T09:22:05.392229+0000","last_clean":"2024-03-24T09:22:05.392229+0000","last_became_active":"2024-03-24T09:19:42.744741+0000","last_became_peered":"2024-03-24T09:19:42.744741+0000","last_unstale":"2024-03-24T09:22:05.392229+0000","last_undegraded":"2024-03-24T09:22:05.392229+0000","last_fullsized":"2024-03-24T09:22:05.392229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:46:05.642447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265368+0000","last_change":"2024-03-24T09:19:38.098750+0000","last_active":"2024-03-24T09:22:05.265368+0000","last_peered":"2024-03-24T09:22:05.265368+0000","last_clean":"2024-03-24T09:22:05.265368+0000","last_became_active":"2024-03-24T09:19:38.098386+0000","last_became_peered":"2024-03-24T09:19:38.098386+0000","last_unstale":"2024-03-24T09:22:05.265368+0000","last_undegraded":"2024-03-24T09:22:05.265368+0000","last_fullsized":"2024-03-24T09:22:05.265368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:54:31.973563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430401+0000","last_change":"2024-03-24T09:19:38.722108+0000","last_active":"2024-03-24T09:22:05.430401+0000","last_peered":"2024-03-24T09:22:05.430401+0000","last_clean":"2024-03-24T09:22:05.430401+0000","last_became_active":"2024-03-24T09:19:38.721772+0000","last_became_peered":"2024-03-24T09:19:38.721772+0000","last_unstale":"2024-03-24T09:22:05.430401+0000","last_undegraded":"2024-03-24T09:22:05.430401+0000","last_fullsized":"2024-03-24T09:22:05.430401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:45:30.867181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429872+0000","last_change":"2024-03-24T09:19:42.739976+0000","last_active":"2024-03-24T09:22:05.429872+0000","last_peered":"2024-03-24T09:22:05.429872+0000","last_clean":"2024-03-24T09:22:05.429872+0000","last_became_active":"2024-03-24T09:19:42.739502+0000","last_became_peered":"2024-03-24T09:19:42.739502+0000","last_unstale":"2024-03-24T09:22:05.429872+0000","last_undegraded":"2024-03-24T09:22:05.429872+0000","last_fullsized":"2024-03-24T09:22:05.429872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:51:11.934111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391701+0000","last_change":"2024-03-24T09:19:38.096487+0000","last_active":"2024-03-24T09:22:05.391701+0000","last_peered":"2024-03-24T09:22:05.391701+0000","last_clean":"2024-03-24T09:22:05.391701+0000","last_became_active":"2024-03-24T09:19:38.096177+0000","last_became_peered":"2024-03-24T09:19:38.096177+0000","last_unstale":"2024-03-24T09:22:05.391701+0000","last_undegraded":"2024-03-24T09:22:05.391701+0000","last_fullsized":"2024-03-24T09:22:05.391701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:31.922758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430945+0000","last_change":"2024-03-24T09:19:38.731548+0000","last_active":"2024-03-24T09:22:05.430945+0000","last_peered":"2024-03-24T09:22:05.430945+0000","last_clean":"2024-03-24T09:22:05.430945+0000","last_became_active":"2024-03-24T09:19:38.731236+0000","last_became_peered":"2024-03-24T09:19:38.731236+0000","last_unstale":"2024-03-24T09:22:05.430945+0000","last_undegraded":"2024-03-24T09:22:05.430945+0000","last_fullsized":"2024-03-24T09:22:05.430945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:12.280332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.381184+0000","last_change":"2024-03-24T09:19:42.748141+0000","last_active":"2024-03-24T09:22:05.381184+0000","last_peered":"2024-03-24T09:22:05.381184+0000","last_clean":"2024-03-24T09:22:05.381184+0000","last_became_active":"2024-03-24T09:19:42.747751+0000","last_became_peered":"2024-03-24T09:19:42.747751+0000","last_unstale":"2024-03-24T09:22:05.381184+0000","last_undegraded":"2024-03-24T09:22:05.381184+0000","last_fullsized":"2024-03-24T09:22:05.381184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:41:37.088122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'10","reported_seq":169,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.079573+0000","last_change":"2024-03-24T09:19:38.100815+0000","last_active":"2024-03-24T09:22:06.079573+0000","last_peered":"2024-03-24T09:22:06.079573+0000","last_clean":"2024-03-24T09:22:06.079573+0000","last_became_active":"2024-03-24T09:19:38.100508+0000","last_became_peered":"2024-03-24T09:19:38.100508+0000","last_unstale":"2024-03-24T09:22:06.079573+0000","last_undegraded":"2024-03-24T09:22:06.079573+0000","last_fullsized":"2024-03-24T09:22:06.079573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:08:57.000984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2395,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":6,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430294+0000","last_change":"2024-03-24T09:20:35.031305+0000","last_active":"2024-03-24T09:22:05.430294+0000","last_peered":"2024-03-24T09:22:05.430294+0000","last_clean":"2024-03-24T09:22:05.430294+0000","last_became_active":"2024-03-24T09:20:35.030952+0000","last_became_peered":"2024-03-24T09:20:35.030952+0000","last_unstale":"2024-03-24T09:22:05.430294+0000","last_undegraded":"2024-03-24T09:22:05.430294+0000","last_fullsized":"2024-03-24T09:22:05.430294+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:40.987185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430873+0000","last_change":"2024-03-24T09:19:42.740500+0000","last_active":"2024-03-24T09:22:05.430873+0000","last_peered":"2024-03-24T09:22:05.430873+0000","last_clean":"2024-03-24T09:22:05.430873+0000","last_became_active":"2024-03-24T09:19:42.740111+0000","last_became_peered":"2024-03-24T09:19:42.740111+0000","last_unstale":"2024-03-24T09:22:05.430873+0000","last_undegraded":"2024-03-24T09:22:05.430873+0000","last_fullsized":"2024-03-24T09:22:05.430873+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:47:48.226924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":52,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383466+0000","last_change":"2024-03-24T09:19:38.097283+0000","last_active":"2024-03-24T09:22:05.383466+0000","last_peered":"2024-03-24T09:22:05.383466+0000","last_clean":"2024-03-24T09:22:05.383466+0000","last_became_active":"2024-03-24T09:19:38.096987+0000","last_became_peered":"2024-03-24T09:19:38.096987+0000","last_unstale":"2024-03-24T09:22:05.383466+0000","last_undegraded":"2024-03-24T09:22:05.383466+0000","last_fullsized":"2024-03-24T09:22:05.383466+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:41:38.901991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391686+0000","last_change":"2024-03-24T09:19:38.726185+0000","last_active":"2024-03-24T09:22:05.391686+0000","last_peered":"2024-03-24T09:22:05.391686+0000","last_clean":"2024-03-24T09:22:05.391686+0000","last_became_active":"2024-03-24T09:19:38.725833+0000","last_became_peered":"2024-03-24T09:19:38.725833+0000","last_unstale":"2024-03-24T09:22:05.391686+0000","last_undegraded":"2024-03-24T09:22:05.391686+0000","last_fullsized":"2024-03-24T09:22:05.391686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:53:27.091377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391546+0000","last_change":"2024-03-24T09:19:42.750997+0000","last_active":"2024-03-24T09:22:05.391546+0000","last_peered":"2024-03-24T09:22:05.391546+0000","last_clean":"2024-03-24T09:22:05.391546+0000","last_became_active":"2024-03-24T09:19:42.750514+0000","last_became_peered":"2024-03-24T09:19:42.750514+0000","last_unstale":"2024-03-24T09:22:05.391546+0000","last_undegraded":"2024-03-24T09:22:05.391546+0000","last_fullsized":"2024-03-24T09:22:05.391546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:30:23.229209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"93'3","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384256+0000","last_change":"2024-03-24T09:19:38.094879+0000","last_active":"2024-03-24T09:22:05.384256+0000","last_peered":"2024-03-24T09:22:05.384256+0000","last_clean":"2024-03-24T09:22:05.384256+0000","last_became_active":"2024-03-24T09:19:38.094496+0000","last_became_peered":"2024-03-24T09:19:38.094496+0000","last_unstale":"2024-03-24T09:22:05.384256+0000","last_undegraded":"2024-03-24T09:22:05.384256+0000","last_fullsized":"2024-03-24T09:22:05.384256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.086867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430172+0000","last_change":"2024-03-24T09:19:38.731878+0000","last_active":"2024-03-24T09:22:05.430172+0000","last_peered":"2024-03-24T09:22:05.430172+0000","last_clean":"2024-03-24T09:22:05.430172+0000","last_became_active":"2024-03-24T09:19:38.731541+0000","last_became_peered":"2024-03-24T09:19:38.731541+0000","last_unstale":"2024-03-24T09:22:05.430172+0000","last_undegraded":"2024-03-24T09:22:05.430172+0000","last_fullsized":"2024-03-24T09:22:05.430172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:11:46.438113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383731+0000","last_change":"2024-03-24T09:19:43.057735+0000","last_active":"2024-03-24T09:22:05.383731+0000","last_peered":"2024-03-24T09:22:05.383731+0000","last_clean":"2024-03-24T09:22:05.383731+0000","last_became_active":"2024-03-24T09:19:43.057227+0000","last_became_peered":"2024-03-24T09:19:43.057227+0000","last_unstale":"2024-03-24T09:22:05.383731+0000","last_undegraded":"2024-03-24T09:22:05.383731+0000","last_fullsized":"2024-03-24T09:22:05.383731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:08:16.847524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.380914+0000","last_change":"2024-03-24T09:19:38.098329+0000","last_active":"2024-03-24T09:22:05.380914+0000","last_peered":"2024-03-24T09:22:05.380914+0000","last_clean":"2024-03-24T09:22:05.380914+0000","last_became_active":"2024-03-24T09:19:38.097999+0000","last_became_peered":"2024-03-24T09:19:38.097999+0000","last_unstale":"2024-03-24T09:22:05.380914+0000","last_undegraded":"2024-03-24T09:22:05.380914+0000","last_fullsized":"2024-03-24T09:22:05.380914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:52.969120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395507+0000","last_change":"2024-03-24T09:19:38.727959+0000","last_active":"2024-03-24T09:22:05.395507+0000","last_peered":"2024-03-24T09:22:05.395507+0000","last_clean":"2024-03-24T09:22:05.395507+0000","last_became_active":"2024-03-24T09:19:38.727642+0000","last_became_peered":"2024-03-24T09:19:38.727642+0000","last_unstale":"2024-03-24T09:22:05.395507+0000","last_undegraded":"2024-03-24T09:22:05.395507+0000","last_fullsized":"2024-03-24T09:22:05.395507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:14:10.295138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430587+0000","last_change":"2024-03-24T09:19:42.747129+0000","last_active":"2024-03-24T09:22:05.430587+0000","last_peered":"2024-03-24T09:22:05.430587+0000","last_clean":"2024-03-24T09:22:05.430587+0000","last_became_active":"2024-03-24T09:19:42.746712+0000","last_became_peered":"2024-03-24T09:19:42.746712+0000","last_unstale":"2024-03-24T09:22:05.430587+0000","last_undegraded":"2024-03-24T09:22:05.430587+0000","last_fullsized":"2024-03-24T09:22:05.430587+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:15:02.723610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"93'6","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430650+0000","last_change":"2024-03-24T09:19:38.101817+0000","last_active":"2024-03-24T09:22:05.430650+0000","last_peered":"2024-03-24T09:22:05.430650+0000","last_clean":"2024-03-24T09:22:05.430650+0000","last_became_active":"2024-03-24T09:19:38.101254+0000","last_became_peered":"2024-03-24T09:19:38.101254+0000","last_unstale":"2024-03-24T09:22:05.430650+0000","last_undegraded":"2024-03-24T09:22:05.430650+0000","last_fullsized":"2024-03-24T09:22:05.430650+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:09:41.329451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.395163+0000","last_change":"2024-03-24T09:19:38.733635+0000","last_active":"2024-03-24T09:22:05.395163+0000","last_peered":"2024-03-24T09:22:05.395163+0000","last_clean":"2024-03-24T09:22:05.395163+0000","last_became_active":"2024-03-24T09:19:38.732938+0000","last_became_peered":"2024-03-24T09:19:38.732938+0000","last_unstale":"2024-03-24T09:22:05.395163+0000","last_undegraded":"2024-03-24T09:22:05.395163+0000","last_fullsized":"2024-03-24T09:22:05.395163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:56:21.786654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430946+0000","last_change":"2024-03-24T09:19:43.057535+0000","last_active":"2024-03-24T09:22:05.430946+0000","last_peered":"2024-03-24T09:22:05.430946+0000","last_clean":"2024-03-24T09:22:05.430946+0000","last_became_active":"2024-03-24T09:19:43.057117+0000","last_became_peered":"2024-03-24T09:19:43.057117+0000","last_unstale":"2024-03-24T09:22:05.430946+0000","last_undegraded":"2024-03-24T09:22:05.430946+0000","last_fullsized":"2024-03-24T09:22:05.430946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:17:14.548497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392124+0000","last_change":"2024-03-24T09:19:38.727449+0000","last_active":"2024-03-24T09:22:05.392124+0000","last_peered":"2024-03-24T09:22:05.392124+0000","last_clean":"2024-03-24T09:22:05.392124+0000","last_became_active":"2024-03-24T09:19:38.727142+0000","last_became_peered":"2024-03-24T09:19:38.727142+0000","last_unstale":"2024-03-24T09:22:05.392124+0000","last_undegraded":"2024-03-24T09:22:05.392124+0000","last_fullsized":"2024-03-24T09:22:05.392124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.524491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'7","reported_seq":162,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430122+0000","last_change":"2024-03-24T09:19:38.102058+0000","last_active":"2024-03-24T09:22:05.430122+0000","last_peered":"2024-03-24T09:22:05.430122+0000","last_clean":"2024-03-24T09:22:05.430122+0000","last_became_active":"2024-03-24T09:19:38.101685+0000","last_became_peered":"2024-03-24T09:19:38.101685+0000","last_unstale":"2024-03-24T09:22:05.430122+0000","last_undegraded":"2024-03-24T09:22:05.430122+0000","last_fullsized":"2024-03-24T09:22:05.430122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:00:17.450235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431778+0000","last_change":"2024-03-24T09:19:42.741717+0000","last_active":"2024-03-24T09:22:05.431778+0000","last_peered":"2024-03-24T09:22:05.431778+0000","last_clean":"2024-03-24T09:22:05.431778+0000","last_became_active":"2024-03-24T09:19:42.741298+0000","last_became_peered":"2024-03-24T09:19:42.741298+0000","last_unstale":"2024-03-24T09:22:05.431778+0000","last_undegraded":"2024-03-24T09:22:05.431778+0000","last_fullsized":"2024-03-24T09:22:05.431778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:31.907322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261595+0000","last_change":"2024-03-24T09:19:38.731233+0000","last_active":"2024-03-24T09:22:05.261595+0000","last_peered":"2024-03-24T09:22:05.261595+0000","last_clean":"2024-03-24T09:22:05.261595+0000","last_became_active":"2024-03-24T09:19:38.730941+0000","last_became_peered":"2024-03-24T09:19:38.730941+0000","last_unstale":"2024-03-24T09:22:05.261595+0000","last_undegraded":"2024-03-24T09:22:05.261595+0000","last_fullsized":"2024-03-24T09:22:05.261595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:02.876844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263319+0000","last_change":"2024-03-24T09:19:38.096721+0000","last_active":"2024-03-24T09:22:05.263319+0000","last_peered":"2024-03-24T09:22:05.263319+0000","last_clean":"2024-03-24T09:22:05.263319+0000","last_became_active":"2024-03-24T09:19:38.096356+0000","last_became_peered":"2024-03-24T09:19:38.096356+0000","last_unstale":"2024-03-24T09:22:05.263319+0000","last_undegraded":"2024-03-24T09:22:05.263319+0000","last_fullsized":"2024-03-24T09:22:05.263319+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:35:01.662680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":368,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429912+0000","last_change":"2024-03-24T09:18:26.090509+0000","last_active":"2024-03-24T09:22:05.429912+0000","last_peered":"2024-03-24T09:22:05.429912+0000","last_clean":"2024-03-24T09:22:05.429912+0000","last_became_active":"2024-03-24T09:18:26.090101+0000","last_became_peered":"2024-03-24T09:18:26.090101+0000","last_unstale":"2024-03-24T09:22:05.429912+0000","last_undegraded":"2024-03-24T09:22:05.429912+0000","last_fullsized":"2024-03-24T09:22:05.429912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:15:00.427345+0000","last_clean_scrub_stamp":"2024-03-24T09:15:00.427345+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:53:50.923714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.392506+0000","last_change":"2024-03-24T09:19:43.058424+0000","last_active":"2024-03-24T09:22:05.392506+0000","last_peered":"2024-03-24T09:22:05.392506+0000","last_clean":"2024-03-24T09:22:05.392506+0000","last_became_active":"2024-03-24T09:19:43.057437+0000","last_became_peered":"2024-03-24T09:19:43.057437+0000","last_unstale":"2024-03-24T09:22:05.392506+0000","last_undegraded":"2024-03-24T09:22:05.392506+0000","last_fullsized":"2024-03-24T09:22:05.392506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:33:11.293573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429921+0000","last_change":"2024-03-24T09:19:38.722598+0000","last_active":"2024-03-24T09:22:05.429921+0000","last_peered":"2024-03-24T09:22:05.429921+0000","last_clean":"2024-03-24T09:22:05.429921+0000","last_became_active":"2024-03-24T09:19:38.721975+0000","last_became_peered":"2024-03-24T09:19:38.721975+0000","last_unstale":"2024-03-24T09:22:05.429921+0000","last_undegraded":"2024-03-24T09:22:05.429921+0000","last_fullsized":"2024-03-24T09:22:05.429921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:59.817055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430097+0000","last_change":"2024-03-24T09:19:38.102675+0000","last_active":"2024-03-24T09:22:05.430097+0000","last_peered":"2024-03-24T09:22:05.430097+0000","last_clean":"2024-03-24T09:22:05.430097+0000","last_became_active":"2024-03-24T09:19:38.102270+0000","last_became_peered":"2024-03-24T09:19:38.102270+0000","last_unstale":"2024-03-24T09:22:05.430097+0000","last_undegraded":"2024-03-24T09:22:05.430097+0000","last_fullsized":"2024-03-24T09:22:05.430097+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:37:21.958629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429932+0000","last_change":"2024-03-24T09:19:42.742248+0000","last_active":"2024-03-24T09:22:05.429932+0000","last_peered":"2024-03-24T09:22:05.429932+0000","last_clean":"2024-03-24T09:22:05.429932+0000","last_became_active":"2024-03-24T09:19:42.741865+0000","last_became_peered":"2024-03-24T09:19:42.741865+0000","last_unstale":"2024-03-24T09:22:05.429932+0000","last_undegraded":"2024-03-24T09:22:05.429932+0000","last_fullsized":"2024-03-24T09:22:05.429932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:50:38.956262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384084+0000","last_change":"2024-03-24T09:19:38.098397+0000","last_active":"2024-03-24T09:22:05.384084+0000","last_peered":"2024-03-24T09:22:05.384084+0000","last_clean":"2024-03-24T09:22:05.384084+0000","last_became_active":"2024-03-24T09:19:38.098074+0000","last_became_peered":"2024-03-24T09:19:38.098074+0000","last_unstale":"2024-03-24T09:22:05.384084+0000","last_undegraded":"2024-03-24T09:22:05.384084+0000","last_fullsized":"2024-03-24T09:22:05.384084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:10:16.536904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430884+0000","last_change":"2024-03-24T09:19:38.722117+0000","last_active":"2024-03-24T09:22:05.430884+0000","last_peered":"2024-03-24T09:22:05.430884+0000","last_clean":"2024-03-24T09:22:05.430884+0000","last_became_active":"2024-03-24T09:19:38.721816+0000","last_became_peered":"2024-03-24T09:19:38.721816+0000","last_unstale":"2024-03-24T09:22:05.430884+0000","last_undegraded":"2024-03-24T09:22:05.430884+0000","last_fullsized":"2024-03-24T09:22:05.430884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:03:35.865687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261523+0000","last_change":"2024-03-24T09:19:43.057638+0000","last_active":"2024-03-24T09:22:05.261523+0000","last_peered":"2024-03-24T09:22:05.261523+0000","last_clean":"2024-03-24T09:22:05.261523+0000","last_became_active":"2024-03-24T09:19:43.057240+0000","last_became_peered":"2024-03-24T09:19:43.057240+0000","last_unstale":"2024-03-24T09:22:05.261523+0000","last_undegraded":"2024-03-24T09:22:05.261523+0000","last_fullsized":"2024-03-24T09:22:05.261523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:30:11.477434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'48","reported_seq":105,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.514742+0000","last_change":"2024-03-24T09:19:38.098363+0000","last_active":"2024-03-24T09:22:05.514742+0000","last_peered":"2024-03-24T09:22:05.514742+0000","last_clean":"2024-03-24T09:22:05.514742+0000","last_became_active":"2024-03-24T09:19:38.097967+0000","last_became_peered":"2024-03-24T09:19:38.097967+0000","last_unstale":"2024-03-24T09:22:05.514742+0000","last_undegraded":"2024-03-24T09:22:05.514742+0000","last_fullsized":"2024-03-24T09:22:05.514742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:53:22.489735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":84,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262000+0000","last_change":"2024-03-24T09:19:38.727970+0000","last_active":"2024-03-24T09:22:05.262000+0000","last_peered":"2024-03-24T09:22:05.262000+0000","last_clean":"2024-03-24T09:22:05.262000+0000","last_became_active":"2024-03-24T09:19:38.727586+0000","last_became_peered":"2024-03-24T09:19:38.727586+0000","last_unstale":"2024-03-24T09:22:05.262000+0000","last_undegraded":"2024-03-24T09:22:05.262000+0000","last_fullsized":"2024-03-24T09:22:05.262000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:40.401349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265022+0000","last_change":"2024-03-24T09:19:42.738149+0000","last_active":"2024-03-24T09:22:05.265022+0000","last_peered":"2024-03-24T09:22:05.265022+0000","last_clean":"2024-03-24T09:22:05.265022+0000","last_became_active":"2024-03-24T09:19:42.737544+0000","last_became_peered":"2024-03-24T09:19:42.737544+0000","last_unstale":"2024-03-24T09:22:05.265022+0000","last_undegraded":"2024-03-24T09:22:05.265022+0000","last_fullsized":"2024-03-24T09:22:05.265022+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:22:58.406290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.264376+0000","last_change":"2024-03-24T09:19:38.096490+0000","last_active":"2024-03-24T09:22:05.264376+0000","last_peered":"2024-03-24T09:22:05.264376+0000","last_clean":"2024-03-24T09:22:05.264376+0000","last_became_active":"2024-03-24T09:19:38.096131+0000","last_became_peered":"2024-03-24T09:19:38.096131+0000","last_unstale":"2024-03-24T09:22:05.264376+0000","last_undegraded":"2024-03-24T09:22:05.264376+0000","last_fullsized":"2024-03-24T09:22:05.264376+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:06.740466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416864+0000","last_change":"2024-03-24T09:19:38.729552+0000","last_active":"2024-03-24T09:22:05.416864+0000","last_peered":"2024-03-24T09:22:05.416864+0000","last_clean":"2024-03-24T09:22:05.416864+0000","last_became_active":"2024-03-24T09:19:38.729248+0000","last_became_peered":"2024-03-24T09:19:38.729248+0000","last_unstale":"2024-03-24T09:22:05.416864+0000","last_undegraded":"2024-03-24T09:22:05.416864+0000","last_fullsized":"2024-03-24T09:22:05.416864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:54:12.682477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263725+0000","last_change":"2024-03-24T09:19:43.057522+0000","last_active":"2024-03-24T09:22:05.263725+0000","last_peered":"2024-03-24T09:22:05.263725+0000","last_clean":"2024-03-24T09:22:05.263725+0000","last_became_active":"2024-03-24T09:19:43.057092+0000","last_became_peered":"2024-03-24T09:19:43.057092+0000","last_unstale":"2024-03-24T09:22:05.263725+0000","last_undegraded":"2024-03-24T09:22:05.263725+0000","last_fullsized":"2024-03-24T09:22:05.263725+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:13:15.282316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429825+0000","last_change":"2024-03-24T09:19:38.098689+0000","last_active":"2024-03-24T09:22:05.429825+0000","last_peered":"2024-03-24T09:22:05.429825+0000","last_clean":"2024-03-24T09:22:05.429825+0000","last_became_active":"2024-03-24T09:19:38.098344+0000","last_became_peered":"2024-03-24T09:19:38.098344+0000","last_unstale":"2024-03-24T09:22:05.429825+0000","last_undegraded":"2024-03-24T09:22:05.429825+0000","last_fullsized":"2024-03-24T09:22:05.429825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:29.388939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.383425+0000","last_change":"2024-03-24T09:19:38.723483+0000","last_active":"2024-03-24T09:22:05.383425+0000","last_peered":"2024-03-24T09:22:05.383425+0000","last_clean":"2024-03-24T09:22:05.383425+0000","last_became_active":"2024-03-24T09:19:38.722210+0000","last_became_peered":"2024-03-24T09:19:38.722210+0000","last_unstale":"2024-03-24T09:22:05.383425+0000","last_undegraded":"2024-03-24T09:22:05.383425+0000","last_fullsized":"2024-03-24T09:22:05.383425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:58:17.401446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.384159+0000","last_change":"2024-03-24T09:21:36.042199+0000","last_active":"2024-03-24T09:22:05.384159+0000","last_peered":"2024-03-24T09:22:05.384159+0000","last_clean":"2024-03-24T09:22:05.384159+0000","last_became_active":"2024-03-24T09:21:36.041615+0000","last_became_peered":"2024-03-24T09:21:36.041615+0000","last_unstale":"2024-03-24T09:22:05.384159+0000","last_undegraded":"2024-03-24T09:22:05.384159+0000","last_fullsized":"2024-03-24T09:22:05.384159+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:40:34.301227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261666+0000","last_change":"2024-03-24T09:19:38.105112+0000","last_active":"2024-03-24T09:22:05.261666+0000","last_peered":"2024-03-24T09:22:05.261666+0000","last_clean":"2024-03-24T09:22:05.261666+0000","last_became_active":"2024-03-24T09:19:38.104102+0000","last_became_peered":"2024-03-24T09:19:38.104102+0000","last_unstale":"2024-03-24T09:22:05.261666+0000","last_undegraded":"2024-03-24T09:22:05.261666+0000","last_fullsized":"2024-03-24T09:22:05.261666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:30:45.570187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429552+0000","last_change":"2024-03-24T09:19:38.729895+0000","last_active":"2024-03-24T09:22:05.429552+0000","last_peered":"2024-03-24T09:22:05.429552+0000","last_clean":"2024-03-24T09:22:05.429552+0000","last_became_active":"2024-03-24T09:19:38.729513+0000","last_became_peered":"2024-03-24T09:19:38.729513+0000","last_unstale":"2024-03-24T09:22:05.429552+0000","last_undegraded":"2024-03-24T09:22:05.429552+0000","last_fullsized":"2024-03-24T09:22:05.429552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:10:56.521062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":26,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263733+0000","last_change":"2024-03-24T09:19:42.737674+0000","last_active":"2024-03-24T09:22:05.263733+0000","last_peered":"2024-03-24T09:22:05.263733+0000","last_clean":"2024-03-24T09:22:05.263733+0000","last_became_active":"2024-03-24T09:19:42.737219+0000","last_became_peered":"2024-03-24T09:19:42.737219+0000","last_unstale":"2024-03-24T09:22:05.263733+0000","last_undegraded":"2024-03-24T09:22:05.263733+0000","last_fullsized":"2024-03-24T09:22:05.263733+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:52:09.977981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429612+0000","last_change":"2024-03-24T09:21:36.048092+0000","last_active":"2024-03-24T09:22:05.429612+0000","last_peered":"2024-03-24T09:22:05.429612+0000","last_clean":"2024-03-24T09:22:05.429612+0000","last_became_active":"2024-03-24T09:21:36.041520+0000","last_became_peered":"2024-03-24T09:21:36.041520+0000","last_unstale":"2024-03-24T09:22:05.429612+0000","last_undegraded":"2024-03-24T09:22:05.429612+0000","last_fullsized":"2024-03-24T09:22:05.429612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:53.315658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262181+0000","last_change":"2024-03-24T09:19:38.729845+0000","last_active":"2024-03-24T09:22:05.262181+0000","last_peered":"2024-03-24T09:22:05.262181+0000","last_clean":"2024-03-24T09:22:05.262181+0000","last_became_active":"2024-03-24T09:19:38.729352+0000","last_became_peered":"2024-03-24T09:19:38.729352+0000","last_unstale":"2024-03-24T09:22:05.262181+0000","last_undegraded":"2024-03-24T09:22:05.262181+0000","last_fullsized":"2024-03-24T09:22:05.262181+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:33:22.583761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.263715+0000","last_change":"2024-03-24T09:19:43.057474+0000","last_active":"2024-03-24T09:22:05.263715+0000","last_peered":"2024-03-24T09:22:05.263715+0000","last_clean":"2024-03-24T09:22:05.263715+0000","last_became_active":"2024-03-24T09:19:43.057045+0000","last_became_peered":"2024-03-24T09:19:43.057045+0000","last_unstale":"2024-03-24T09:22:05.263715+0000","last_undegraded":"2024-03-24T09:22:05.263715+0000","last_fullsized":"2024-03-24T09:22:05.263715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:31:56.569657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":19,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429820+0000","last_change":"2024-03-24T09:21:36.044659+0000","last_active":"2024-03-24T09:22:05.429820+0000","last_peered":"2024-03-24T09:22:05.429820+0000","last_clean":"2024-03-24T09:22:05.429820+0000","last_became_active":"2024-03-24T09:21:36.039364+0000","last_became_peered":"2024-03-24T09:21:36.039364+0000","last_unstale":"2024-03-24T09:22:05.429820+0000","last_undegraded":"2024-03-24T09:22:05.429820+0000","last_fullsized":"2024-03-24T09:22:05.429820+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:51.993956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430424+0000","last_change":"2024-03-24T09:19:38.729413+0000","last_active":"2024-03-24T09:22:05.430424+0000","last_peered":"2024-03-24T09:22:05.430424+0000","last_clean":"2024-03-24T09:22:05.430424+0000","last_became_active":"2024-03-24T09:19:38.728778+0000","last_became_peered":"2024-03-24T09:19:38.728778+0000","last_unstale":"2024-03-24T09:22:05.430424+0000","last_undegraded":"2024-03-24T09:22:05.430424+0000","last_fullsized":"2024-03-24T09:22:05.430424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:17:02.755301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430230+0000","last_change":"2024-03-24T09:21:36.039252+0000","last_active":"2024-03-24T09:22:05.430230+0000","last_peered":"2024-03-24T09:22:05.430230+0000","last_clean":"2024-03-24T09:22:05.430230+0000","last_became_active":"2024-03-24T09:21:36.038637+0000","last_became_peered":"2024-03-24T09:21:36.038637+0000","last_unstale":"2024-03-24T09:22:05.430230+0000","last_undegraded":"2024-03-24T09:22:05.430230+0000","last_fullsized":"2024-03-24T09:22:05.430230+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:49.512786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"94'7","reported_seq":165,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:06.076395+0000","last_change":"2024-03-24T09:19:38.097176+0000","last_active":"2024-03-24T09:22:06.076395+0000","last_peered":"2024-03-24T09:22:06.076395+0000","last_clean":"2024-03-24T09:22:06.076395+0000","last_became_active":"2024-03-24T09:19:38.096859+0000","last_became_peered":"2024-03-24T09:19:38.096859+0000","last_unstale":"2024-03-24T09:22:06.076395+0000","last_undegraded":"2024-03-24T09:22:06.076395+0000","last_fullsized":"2024-03-24T09:22:06.076395+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:20.639063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2054,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":124,"num_read_kb":4,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431370+0000","last_change":"2024-03-24T09:19:38.723325+0000","last_active":"2024-03-24T09:22:05.431370+0000","last_peered":"2024-03-24T09:22:05.431370+0000","last_clean":"2024-03-24T09:22:05.431370+0000","last_became_active":"2024-03-24T09:19:38.723001+0000","last_became_peered":"2024-03-24T09:19:38.723001+0000","last_unstale":"2024-03-24T09:22:05.431370+0000","last_undegraded":"2024-03-24T09:22:05.431370+0000","last_fullsized":"2024-03-24T09:22:05.431370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:50.552303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416208+0000","last_change":"2024-03-24T09:19:42.752380+0000","last_active":"2024-03-24T09:22:05.416208+0000","last_peered":"2024-03-24T09:22:05.416208+0000","last_clean":"2024-03-24T09:22:05.416208+0000","last_became_active":"2024-03-24T09:19:42.751989+0000","last_became_peered":"2024-03-24T09:19:42.751989+0000","last_unstale":"2024-03-24T09:22:05.416208+0000","last_undegraded":"2024-03-24T09:22:05.416208+0000","last_fullsized":"2024-03-24T09:22:05.416208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:35.852906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'41","reported_seq":83,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.512009+0000","last_change":"2024-03-24T09:19:38.098226+0000","last_active":"2024-03-24T09:22:05.512009+0000","last_peered":"2024-03-24T09:22:05.512009+0000","last_clean":"2024-03-24T09:22:05.512009+0000","last_became_active":"2024-03-24T09:19:38.097144+0000","last_became_peered":"2024-03-24T09:19:38.097144+0000","last_unstale":"2024-03-24T09:22:05.512009+0000","last_undegraded":"2024-03-24T09:22:05.512009+0000","last_fullsized":"2024-03-24T09:22:05.512009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:18.659581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431252+0000","last_change":"2024-03-24T09:20:35.031333+0000","last_active":"2024-03-24T09:22:05.431252+0000","last_peered":"2024-03-24T09:22:05.431252+0000","last_clean":"2024-03-24T09:22:05.431252+0000","last_became_active":"2024-03-24T09:20:35.030995+0000","last_became_peered":"2024-03-24T09:20:35.030995+0000","last_unstale":"2024-03-24T09:22:05.431252+0000","last_undegraded":"2024-03-24T09:22:05.431252+0000","last_fullsized":"2024-03-24T09:22:05.431252+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:16:05.746740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391405+0000","last_change":"2024-03-24T09:19:42.744569+0000","last_active":"2024-03-24T09:22:05.391405+0000","last_peered":"2024-03-24T09:22:05.391405+0000","last_clean":"2024-03-24T09:22:05.391405+0000","last_became_active":"2024-03-24T09:19:42.744175+0000","last_became_peered":"2024-03-24T09:19:42.744175+0000","last_unstale":"2024-03-24T09:22:05.391405+0000","last_undegraded":"2024-03-24T09:22:05.391405+0000","last_fullsized":"2024-03-24T09:22:05.391405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:41.714442+0000","last_clean_scrub_stamp":"2024-03-24T09:19:41.714442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:17:27.036599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430787+0000","last_change":"2024-03-24T09:19:38.097293+0000","last_active":"2024-03-24T09:22:05.430787+0000","last_peered":"2024-03-24T09:22:05.430787+0000","last_clean":"2024-03-24T09:22:05.430787+0000","last_became_active":"2024-03-24T09:19:38.096957+0000","last_became_peered":"2024-03-24T09:19:38.096957+0000","last_unstale":"2024-03-24T09:22:05.430787+0000","last_undegraded":"2024-03-24T09:22:05.430787+0000","last_fullsized":"2024-03-24T09:22:05.430787+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:37.073950+0000","last_clean_scrub_stamp":"2024-03-24T09:19:37.073950+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:57.551873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.431653+0000","last_change":"2024-03-24T09:19:38.725190+0000","last_active":"2024-03-24T09:22:05.431653+0000","last_peered":"2024-03-24T09:22:05.431653+0000","last_clean":"2024-03-24T09:22:05.431653+0000","last_became_active":"2024-03-24T09:19:38.724383+0000","last_became_peered":"2024-03-24T09:19:38.724383+0000","last_unstale":"2024-03-24T09:22:05.431653+0000","last_undegraded":"2024-03-24T09:22:05.431653+0000","last_fullsized":"2024-03-24T09:22:05.431653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:48.032463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.429626+0000","last_change":"2024-03-24T09:19:38.727715+0000","last_active":"2024-03-24T09:22:05.429626+0000","last_peered":"2024-03-24T09:22:05.429626+0000","last_clean":"2024-03-24T09:22:05.429626+0000","last_became_active":"2024-03-24T09:19:38.727369+0000","last_became_peered":"2024-03-24T09:19:38.727369+0000","last_unstale":"2024-03-24T09:22:05.429626+0000","last_undegraded":"2024-03-24T09:22:05.429626+0000","last_fullsized":"2024-03-24T09:22:05.429626+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:36:46.491097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265298+0000","last_change":"2024-03-24T09:19:38.728266+0000","last_active":"2024-03-24T09:22:05.265298+0000","last_peered":"2024-03-24T09:22:05.265298+0000","last_clean":"2024-03-24T09:22:05.265298+0000","last_became_active":"2024-03-24T09:19:38.727851+0000","last_became_peered":"2024-03-24T09:19:38.727851+0000","last_unstale":"2024-03-24T09:22:05.265298+0000","last_undegraded":"2024-03-24T09:22:05.265298+0000","last_fullsized":"2024-03-24T09:22:05.265298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:10.486565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265210+0000","last_change":"2024-03-24T09:19:38.726527+0000","last_active":"2024-03-24T09:22:05.265210+0000","last_peered":"2024-03-24T09:22:05.265210+0000","last_clean":"2024-03-24T09:22:05.265210+0000","last_became_active":"2024-03-24T09:19:38.726195+0000","last_became_peered":"2024-03-24T09:19:38.726195+0000","last_unstale":"2024-03-24T09:22:05.265210+0000","last_undegraded":"2024-03-24T09:22:05.265210+0000","last_fullsized":"2024-03-24T09:22:05.265210+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:30:40.446298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.416529+0000","last_change":"2024-03-24T09:19:38.729306+0000","last_active":"2024-03-24T09:22:05.416529+0000","last_peered":"2024-03-24T09:22:05.416529+0000","last_clean":"2024-03-24T09:22:05.416529+0000","last_became_active":"2024-03-24T09:19:38.728748+0000","last_became_peered":"2024-03-24T09:19:38.728748+0000","last_unstale":"2024-03-24T09:22:05.416529+0000","last_undegraded":"2024-03-24T09:22:05.416529+0000","last_fullsized":"2024-03-24T09:22:05.416529+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:55:15.420677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430754+0000","last_change":"2024-03-24T09:19:38.731829+0000","last_active":"2024-03-24T09:22:05.430754+0000","last_peered":"2024-03-24T09:22:05.430754+0000","last_clean":"2024-03-24T09:22:05.430754+0000","last_became_active":"2024-03-24T09:19:38.731517+0000","last_became_peered":"2024-03-24T09:19:38.731517+0000","last_unstale":"2024-03-24T09:22:05.430754+0000","last_undegraded":"2024-03-24T09:22:05.430754+0000","last_fullsized":"2024-03-24T09:22:05.430754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:12:49.323742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430399+0000","last_change":"2024-03-24T09:19:38.724439+0000","last_active":"2024-03-24T09:22:05.430399+0000","last_peered":"2024-03-24T09:22:05.430399+0000","last_clean":"2024-03-24T09:22:05.430399+0000","last_became_active":"2024-03-24T09:19:38.724070+0000","last_became_peered":"2024-03-24T09:19:38.724070+0000","last_unstale":"2024-03-24T09:22:05.430399+0000","last_undegraded":"2024-03-24T09:22:05.430399+0000","last_fullsized":"2024-03-24T09:22:05.430399+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:06:40.757212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.265749+0000","last_change":"2024-03-24T09:19:38.732103+0000","last_active":"2024-03-24T09:22:05.265749+0000","last_peered":"2024-03-24T09:22:05.265749+0000","last_clean":"2024-03-24T09:22:05.265749+0000","last_became_active":"2024-03-24T09:19:38.731773+0000","last_became_peered":"2024-03-24T09:19:38.731773+0000","last_unstale":"2024-03-24T09:22:05.265749+0000","last_undegraded":"2024-03-24T09:22:05.265749+0000","last_fullsized":"2024-03-24T09:22:05.265749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:32:02.099618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.262063+0000","last_change":"2024-03-24T09:19:38.736145+0000","last_active":"2024-03-24T09:22:05.262063+0000","last_peered":"2024-03-24T09:22:05.262063+0000","last_clean":"2024-03-24T09:22:05.262063+0000","last_became_active":"2024-03-24T09:19:38.735823+0000","last_became_peered":"2024-03-24T09:19:38.735823+0000","last_unstale":"2024-03-24T09:22:05.262063+0000","last_undegraded":"2024-03-24T09:22:05.262063+0000","last_fullsized":"2024-03-24T09:22:05.262063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:53.644731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.391993+0000","last_change":"2024-03-24T09:19:38.722934+0000","last_active":"2024-03-24T09:22:05.391993+0000","last_peered":"2024-03-24T09:22:05.391993+0000","last_clean":"2024-03-24T09:22:05.391993+0000","last_became_active":"2024-03-24T09:19:38.722616+0000","last_became_peered":"2024-03-24T09:19:38.722616+0000","last_unstale":"2024-03-24T09:22:05.391993+0000","last_undegraded":"2024-03-24T09:22:05.391993+0000","last_fullsized":"2024-03-24T09:22:05.391993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:17:12.197915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.261494+0000","last_change":"2024-03-24T09:19:38.733263+0000","last_active":"2024-03-24T09:22:05.261494+0000","last_peered":"2024-03-24T09:22:05.261494+0000","last_clean":"2024-03-24T09:22:05.261494+0000","last_became_active":"2024-03-24T09:19:38.732557+0000","last_became_peered":"2024-03-24T09:19:38.732557+0000","last_unstale":"2024-03-24T09:22:05.261494+0000","last_undegraded":"2024-03-24T09:22:05.261494+0000","last_fullsized":"2024-03-24T09:22:05.261494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:46:26.560985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430502+0000","last_change":"2024-03-24T09:19:38.727372+0000","last_active":"2024-03-24T09:22:05.430502+0000","last_peered":"2024-03-24T09:22:05.430502+0000","last_clean":"2024-03-24T09:22:05.430502+0000","last_became_active":"2024-03-24T09:19:38.727025+0000","last_became_peered":"2024-03-24T09:19:38.727025+0000","last_unstale":"2024-03-24T09:22:05.430502+0000","last_undegraded":"2024-03-24T09:22:05.430502+0000","last_fullsized":"2024-03-24T09:22:05.430502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:50:43.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430370+0000","last_change":"2024-03-24T09:19:38.723490+0000","last_active":"2024-03-24T09:22:05.430370+0000","last_peered":"2024-03-24T09:22:05.430370+0000","last_clean":"2024-03-24T09:22:05.430370+0000","last_became_active":"2024-03-24T09:19:38.723163+0000","last_became_peered":"2024-03-24T09:19:38.723163+0000","last_unstale":"2024-03-24T09:22:05.430370+0000","last_undegraded":"2024-03-24T09:22:05.430370+0000","last_fullsized":"2024-03-24T09:22:05.430370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:55:09.084058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T09:22:05.430156+0000","last_change":"2024-03-24T09:19:38.732594+0000","last_active":"2024-03-24T09:22:05.430156+0000","last_peered":"2024-03-24T09:22:05.430156+0000","last_clean":"2024-03-24T09:22:05.430156+0000","last_became_active":"2024-03-24T09:19:38.732295+0000","last_became_peered":"2024-03-24T09:19:38.732295+0000","last_unstale":"2024-03-24T09:22:05.430156+0000","last_undegraded":"2024-03-24T09:22:05.430156+0000","last_fullsized":"2024-03-24T09:22:05.430156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T09:19:38.080268+0000","last_clean_scrub_stamp":"2024-03-24T09:19:38.080268+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:43:35.641510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107004,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":604641,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":1352,"num_read_kb":795,"num_write":380,"num_write_kb":1476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":1813923,"data_compressed":80715,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383722,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":531197,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645359,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1172328,"data_compressed":63929,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808885,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1093632,"data_stored":927555,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1085440,"data_stored":923837,"data_compressed":48789,"data_compressed_allocated":65536,"data_compressed_original":475136,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712884,"statfs":{"total":95995035648,"available":95961993216,"internally_reserved":0,"allocated":1114112,"data_stored":1129270,"data_compressed":65311,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495684,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":526240,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":596464,"data_compressed":31246,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1064960,"data_stored":528087,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1130496,"data_stored":598100,"data_compressed":31246,"data_compressed_allocated":45056,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1050271,"data_compressed":54945,"data_compressed_allocated":73728,"data_compressed_original":598016,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920846,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":531628,"data_compressed":29643,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757321,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36464,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708688,"statfs":{"total":95995035648,"available":95957696512,"internally_reserved":0,"allocated":1142784,"data_stored":1254934,"data_compressed":71467,"data_compressed_allocated":94208,"data_compressed_original":794624,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35668,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71439,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":73075,"data_compressed":1603,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":525246,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":135445,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":6467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":394716,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":394338,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":204337,"data_compressed":7759,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T09:22:10.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:22:10.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 -- 172.21.15.116:0/3710808862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b81076b0 msgr2=0x7f36b8107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/3710808862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b81076b0 0x7f36b8107a90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009930 tx=0x7f36a802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:10.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 -- 172.21.15.116:0/3710808862 shutdown_connections 2024-03-24T09:22:10.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/3710808862 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b8102060 0x7f36b810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/3710808862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b81016a0 0x7f36b8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/3710808862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b81076b0 0x7f36b8107a90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.151+0000 7f36bcd5e640 1 -- 172.21.15.116:0/3710808862 >> 172.21.15.116:0/3710808862 conn(0x7f36b8078b10 msgr2=0x7f36b8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:10.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.152+0000 7f36bcd5e640 1 -- 172.21.15.116:0/3710808862 shutdown_connections 2024-03-24T09:22:10.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.152+0000 7f36bcd5e640 1 -- 172.21.15.116:0/3710808862 wait complete. 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.152+0000 7f36bcd5e640 1 Processor -- start 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.152+0000 7f36bcd5e640 1 -- start start 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b81016a0 0x7f36b819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 0x7f36b819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b81076b0 0x7f36b8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36b81980e0 con 0x7f36b8102060 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8198250 con 0x7f36b81016a0 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f36b8198410 con 0x7f36b81076b0 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 0x7f36b819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 0x7f36b819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55896/0 (socket says 172.21.15.116:55896) 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 -- 172.21.15.116:0/948360474 learned_addr learned my addr 172.21.15.116:0/948360474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:10.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b659c640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b81016a0 0x7f36b819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b6d9d640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b81076b0 0x7f36b8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 -- 172.21.15.116:0/948360474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b81076b0 msgr2=0x7f36b8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b81076b0 0x7f36b8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 -- 172.21.15.116:0/948360474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b81016a0 msgr2=0x7f36b819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b81016a0 0x7f36b819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 -- 172.21.15.116:0/948360474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a8009560 con 0x7f36b8102060 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36b5d9b640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 0x7f36b819de20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00e9f0 tx=0x7f36ac00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac009700 con 0x7f36b8102060 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f36ac00cd80 con 0x7f36b8102060 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b81986f0 con 0x7f36b8102060 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac018840 con 0x7f36b8102060 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.153+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b81a4980 con 0x7f36b8102060 2024-03-24T09:22:10.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.155+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b8067c70 con 0x7f36b8102060 2024-03-24T09:22:10.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.155+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 23) ==== 136877+0+0 (secure 0 0 0) 0x7f36ac0189a0 con 0x7f36b8102060 2024-03-24T09:22:10.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.156+0000 7f36a77fe640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.156+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f36ac014070 con 0x7f36b8102060 2024-03-24T09:22:10.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.156+0000 7f36b659c640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.157+0000 7f36b659c640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36a8005ab0 tx=0x7f36a8005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:10.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.161+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f36ac053280 con 0x7f36b8102060 2024-03-24T09:22:10.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.188+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f36ac0cbd60 con 0x7f36b8102060 2024-03-24T09:22:10.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.358+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36b8109c10 con 0x7f36b8102060 2024-03-24T09:22:10.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.360+0000 7f36a77fe640 1 -- 172.21.15.116:0/948360474 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25293 (secure 0 0 0) 0x7f36ac02c220 con 0x7f36b8102060 2024-03-24T09:22:10.361 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:10.361 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:22:05.255869+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 msgr2=0x7f36a00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f36a8005ab0 tx=0x7f36a8005fd0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 msgr2=0x7f36b819de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 0x7f36b819de20 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00e9f0 tx=0x7f36ac00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 shutdown_connections 2024-03-24T09:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b81076b0 0x7f36b8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b8102060 0x7f36b819de20 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 --2- 172.21.15.116:0/948360474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b81016a0 0x7f36b819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 >> 172.21.15.116:0/948360474 conn(0x7f36b8078b10 msgr2=0x7f36b81042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 shutdown_connections 2024-03-24T09:22:10.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.362+0000 7f36bcd5e640 1 -- 172.21.15.116:0/948360474 wait complete. 2024-03-24T09:22:10.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:22:10.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/1458065566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c740fffa0 msgr2=0x7f0c74100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/1458065566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c740fffa0 0x7f0c74100380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000aa40 tx=0x7f0c7002cf50 comp rx=0 tx=0).stop 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/1458065566 shutdown_connections 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/1458065566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c74105070 0x7f0c7410cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/1458065566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c74100950 0x7f0c74104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/1458065566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c740fffa0 0x7f0c74100380 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.479+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/1458065566 >> 172.21.15.116:0/1458065566 conn(0x7f0c74076b20 msgr2=0x7f0c740fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/1458065566 shutdown_connections 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/1458065566 wait complete. 2024-03-24T09:22:10.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 Processor -- start 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 -- start start 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740fffa0 0x7f0c7419b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c74105070 0x7f0c7419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 0x7f0c74195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0c741962d0 con 0x7f0c741958e0 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74196440 con 0x7f0c74105070 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7bfc7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0c74196600 con 0x7f0c740fffa0 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7a564640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 0x7f0c74195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7a564640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 0x7f0c74195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55918/0 (socket says 172.21.15.116:55918) 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c7a564640 1 -- 172.21.15.116:0/2577923069 learned_addr learned my addr 172.21.15.116:0/2577923069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c79d63640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740fffa0 0x7f0c7419b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.480+0000 7f0c79562640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c74105070 0x7f0c7419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7a564640 1 -- 172.21.15.116:0/2577923069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740fffa0 msgr2=0x7f0c7419b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7a564640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740fffa0 0x7f0c7419b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7a564640 1 -- 172.21.15.116:0/2577923069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c74105070 msgr2=0x7f0c7419bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7a564640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c74105070 0x7f0c7419bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7a564640 1 -- 172.21.15.116:0/2577923069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7000a670 con 0x7f0c741958e0 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7a564640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 0x7f0c74195d90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00eb10 tx=0x7f0c6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c004b20 con 0x7f0c741958e0 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0c6c004c80 con 0x7f0c741958e0 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c741a25f0 con 0x7f0c741958e0 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c010690 con 0x7f0c741958e0 2024-03-24T09:22:10.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.481+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c741a2ad0 con 0x7f0c741958e0 2024-03-24T09:22:10.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.482+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c74068950 con 0x7f0c741958e0 2024-03-24T09:22:10.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.483+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f0c6c005020 con 0x7f0c741958e0 2024-03-24T09:22:10.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.484+0000 7f0c6affd640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0c600a2340 0x7f0c600a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.484+0000 7f0c79d63640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0c600a2340 0x7f0c600a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.484+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0c6c014070 con 0x7f0c741958e0 2024-03-24T09:22:10.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.484+0000 7f0c79d63640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0c600a2340 0x7f0c600a4800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c70005ab0 tx=0x7f0c70005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:10.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.488+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0c6c097260 con 0x7f0c741958e0 2024-03-24T09:22:10.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.662+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0c741082a0 con 0x7f0c741958e0 2024-03-24T09:22:10.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.663+0000 7f0c6affd640 1 -- 172.21.15.116:0/2577923069 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f0c6c09a920 con 0x7f0c741958e0 2024-03-24T09:22:10.663 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:22:10.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0c600a2340 msgr2=0x7f0c600a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0c600a2340 0x7f0c600a4800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c70005ab0 tx=0x7f0c70005fd0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 msgr2=0x7f0c74195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 0x7f0c74195d90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00eb10 tx=0x7f0c6c00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 shutdown_connections 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c741958e0 0x7f0c74195d90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c74105070 0x7f0c7419bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0c600a2340 0x7f0c600a4800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 --2- 172.21.15.116:0/2577923069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c740fffa0 0x7f0c7419b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 >> 172.21.15.116:0/2577923069 conn(0x7f0c74076b20 msgr2=0x7f0c741033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 shutdown_connections 2024-03-24T09:22:10.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.665+0000 7f0c7bfc7640 1 -- 172.21.15.116:0/2577923069 wait complete. 2024-03-24T09:22:10.676 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:22:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 -- 172.21.15.116:0/1093797434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 msgr2=0x7fba94100430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 --2- 172.21.15.116:0/1093797434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba94100430 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fba84009900 tx=0x7fba8402bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 -- 172.21.15.116:0/1093797434 shutdown_connections 2024-03-24T09:22:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 --2- 172.21.15.116:0/1093797434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba94100970 0x7fba9410cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 --2- 172.21.15.116:0/1093797434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba94100430 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 --2- 172.21.15.116:0/1093797434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 0x7fba94105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.783+0000 7fba9c064640 1 -- 172.21.15.116:0/1093797434 >> 172.21.15.116:0/1093797434 conn(0x7fba94075950 msgr2=0x7fba94075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 -- 172.21.15.116:0/1093797434 shutdown_connections 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 -- 172.21.15.116:0/1093797434 wait complete. 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 Processor -- start 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 -- start start 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba9419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba94100970 0x7fba9419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 0x7fba94197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fba94198080 con 0x7fba940fffd0 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fba941981f0 con 0x7fba94100970 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba9c064640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fba941983b0 con 0x7fba94104d40 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba99e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba9419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba99e00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba9419d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55946/0 (socket says 172.21.15.116:55946) 2024-03-24T09:22:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba99e00640 1 -- 172.21.15.116:0/3740826691 learned_addr learned my addr 172.21.15.116:0/3740826691 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.784+0000 7fba995ff640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba94100970 0x7fba9419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba9a601640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 0x7fba94197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba99e00640 1 -- 172.21.15.116:0/3740826691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 msgr2=0x7fba94197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba99e00640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 0x7fba94197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba99e00640 1 -- 172.21.15.116:0/3740826691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba94100970 msgr2=0x7fba9419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba99e00640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba94100970 0x7fba9419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba99e00640 1 -- 172.21.15.116:0/3740826691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba84009560 con 0x7fba940fffd0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba9a601640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 0x7fba94197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba99e00640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba9419d880 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fba9000eb90 tx=0x7fba9000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba900099b0 con 0x7fba940fffd0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fba9000ee90 con 0x7fba940fffd0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba94198690 con 0x7fba940fffd0 2024-03-24T09:22:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba90010690 con 0x7fba940fffd0 2024-03-24T09:22:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.785+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba941a4920 con 0x7fba940fffd0 2024-03-24T09:22:10.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.786+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba94101470 con 0x7fba940fffd0 2024-03-24T09:22:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.787+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fba90005020 con 0x7fba940fffd0 2024-03-24T09:22:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.788+0000 7fba8affd640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fba800a22c0 0x7fba800a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.788+0000 7fba995ff640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fba800a22c0 0x7fba800a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.788+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fba90014070 con 0x7fba940fffd0 2024-03-24T09:22:10.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.788+0000 7fba995ff640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fba800a22c0 0x7fba800a4780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba84006010 tx=0x7fba8402c6c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:10.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.792+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fba90097520 con 0x7fba940fffd0 2024-03-24T09:22:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.964+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fba941092a0 con 0x7fba940fffd0 2024-03-24T09:22:10.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.965+0000 7fba8affd640 1 -- 172.21.15.116:0/3740826691 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fba9009abe0 con 0x7fba940fffd0 2024-03-24T09:22:10.965 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:22:10.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.966+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fba800a22c0 msgr2=0x7fba800a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fba800a22c0 0x7fba800a4780 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba84006010 tx=0x7fba8402c6c0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 msgr2=0x7fba9419d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:10.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba9419d880 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fba9000eb90 tx=0x7fba9000c6a0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 shutdown_connections 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fba800a22c0 0x7fba800a4780 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fba94104d40 0x7fba94197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fba94100970 0x7fba9419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 --2- 172.21.15.116:0/3740826691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba940fffd0 0x7fba9419d880 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 >> 172.21.15.116:0/3740826691 conn(0x7fba94075950 msgr2=0x7fba9410d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 shutdown_connections 2024-03-24T09:22:10.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:10.967+0000 7fba9c064640 1 -- 172.21.15.116:0/3740826691 wait complete. 2024-03-24T09:22:10.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:22:11.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 -- 172.21.15.116:0/609417262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf410bc70 msgr2=0x7f5cf410e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/609417262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf410bc70 0x7f5cf410e110 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f5cec009ca0 tx=0x7f5cec02c180 comp rx=0 tx=0).stop 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 -- 172.21.15.116:0/609417262 shutdown_connections 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/609417262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf410bc70 0x7f5cf410e110 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/609417262 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf4103250 0x7f5cf410b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/609417262 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 0x7f5cf4102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 -- 172.21.15.116:0/609417262 >> 172.21.15.116:0/609417262 conn(0x7f5cf40777c0 msgr2=0x7f5cf4077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 -- 172.21.15.116:0/609417262 shutdown_connections 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 -- 172.21.15.116:0/609417262 wait complete. 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 Processor -- start 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.083+0000 7f5cfbc07640 1 -- start start 2024-03-24T09:22:11.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 0x7f5cf4100580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 0x7f5cf4100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf410bc70 0x7f5cf4101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41a5fa0 con 0x7f5cf4103250 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41a6110 con 0x7f5cf4102930 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf41a62d0 con 0x7f5cf410bc70 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 0x7f5cf4100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 0x7f5cf4100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55968/0 (socket says 172.21.15.116:55968) 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 -- 172.21.15.116:0/104521813 learned_addr learned my addr 172.21.15.116:0/104521813 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf99a3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 0x7f5cf4100580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfa1a4640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf410bc70 0x7f5cf4101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 -- 172.21.15.116:0/104521813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf410bc70 msgr2=0x7f5cf4101000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf410bc70 0x7f5cf4101000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 -- 172.21.15.116:0/104521813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 msgr2=0x7f5cf4100580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 0x7f5cf4100580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 -- 172.21.15.116:0/104521813 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cec009900 con 0x7f5cf4103250 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfa1a4640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf410bc70 0x7f5cf4101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf99a3640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 0x7f5cf4100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cf91a2640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 0x7f5cf4100ac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400e9f0 tx=0x7f5ce400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4009720 con 0x7f5cf4103250 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf41a65b0 con 0x7f5cf4103250 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5ce400cd80 con 0x7f5cf4103250 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce4010430 con 0x7f5cf4103250 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.084+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf41a6a70 con 0x7f5cf4103250 2024-03-24T09:22:11.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.085+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf4068500 con 0x7f5cf4103250 2024-03-24T09:22:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.086+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f5ce4005020 con 0x7f5cf4103250 2024-03-24T09:22:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.087+0000 7f5ceaffd640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5ce00a22c0 0x7f5ce00a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.087+0000 7f5cf99a3640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5ce00a22c0 0x7f5ce00a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.087+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5ce4014070 con 0x7f5cf4103250 2024-03-24T09:22:11.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.087+0000 7f5cf99a3640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5ce00a22c0 0x7f5ce00a4780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009fc0 tx=0x7f5cf0005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:11.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.091+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5ce4097120 con 0x7f5cf4103250 2024-03-24T09:22:11.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.267+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5cf4106df0 con 0x7f5cf4103250 2024-03-24T09:22:11.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.267+0000 7f5ceaffd640 1 -- 172.21.15.116:0/104521813 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f5ce409a7e0 con 0x7f5cf4103250 2024-03-24T09:22:11.268 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5ce00a22c0 msgr2=0x7f5ce00a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5ce00a22c0 0x7f5ce00a4780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5cf0009fc0 tx=0x7f5cf0005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 msgr2=0x7f5cf4100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 0x7f5cf4100ac0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5ce400e9f0 tx=0x7f5ce400edf0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 shutdown_connections 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5ce00a22c0 0x7f5ce00a4780 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5cf410bc70 0x7f5cf4101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf4103250 0x7f5cf4100ac0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 --2- 172.21.15.116:0/104521813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cf4102930 0x7f5cf4100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 >> 172.21.15.116:0/104521813 conn(0x7f5cf40777c0 msgr2=0x7f5cf40fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:11.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 shutdown_connections 2024-03-24T09:22:11.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.269+0000 7f5cfbc07640 1 -- 172.21.15.116:0/104521813 wait complete. 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:22:11.282 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/948360474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:11.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2577923069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:11.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:11 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3740826691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:11.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:22:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.387+0000 7f7868525640 1 -- 172.21.15.116:0/1958211369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78600692d0 msgr2=0x7f786010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- 172.21.15.116:0/1958211369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78600692d0 0x7f786010e570 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7858009c10 tx=0x7f785802bef0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- 172.21.15.116:0/1958211369 shutdown_connections 2024-03-24T09:22:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- 172.21.15.116:0/1958211369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78600692d0 0x7f786010e570 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- 172.21.15.116:0/1958211369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7860068910 0x7f7860068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- 172.21.15.116:0/1958211369 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7860107760 0x7f7860107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- 172.21.15.116:0/1958211369 >> 172.21.15.116:0/1958211369 conn(0x7f7860077710 msgr2=0x7f7860077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- 172.21.15.116:0/1958211369 shutdown_connections 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- 172.21.15.116:0/1958211369 wait complete. 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 Processor -- start 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- start start 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7860068910 0x7f786019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78600692d0 0x7f786019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 0x7f7860197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7860198050 con 0x7f7860107760 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f78601981c0 con 0x7f7860068910 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.388+0000 7f7868525640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7860198380 con 0x7f78600692d0 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 0x7f7860197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 0x7f7860197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55998/0 (socket says 172.21.15.116:55998) 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f78662c1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7860068910 0x7f786019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 -- 172.21.15.116:0/139891957 learned_addr learned my addr 172.21.15.116:0/139891957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7865ac0640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78600692d0 0x7f786019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 -- 172.21.15.116:0/139891957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78600692d0 msgr2=0x7f786019dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78600692d0 0x7f786019dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 -- 172.21.15.116:0/139891957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7860068910 msgr2=0x7f786019d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7860068910 0x7f786019d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 -- 172.21.15.116:0/139891957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7858009840 con 0x7f7860107760 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7866ac2640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 0x7f7860197a00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7858036040 tx=0x7f785802cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858003970 con 0x7f7860107760 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7858004040 con 0x7f7860107760 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7860198600 con 0x7f7860107760 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858034690 con 0x7f7860107760 2024-03-24T09:22:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.389+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78601a48f0 con 0x7f7860107760 2024-03-24T09:22:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.390+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7860068500 con 0x7f7860107760 2024-03-24T09:22:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.391+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f7858003ad0 con 0x7f7860107760 2024-03-24T09:22:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.392+0000 7f78577fe640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f784c0a2340 0x7f784c0a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.392+0000 7f78662c1640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f784c0a2340 0x7f784c0a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.392+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f78580f0ea0 con 0x7f7860107760 2024-03-24T09:22:11.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.392+0000 7f78662c1640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f784c0a2340 0x7f784c0a4800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7860198b40 tx=0x7f785c005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:11.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.396+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f78580b9720 con 0x7f7860107760 2024-03-24T09:22:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/948360474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2577923069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:11.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:11 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3740826691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='client.15687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T09:22:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='client.15693 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: Updating smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/948360474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2577923069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:22:11.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:11 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3740826691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:22:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.569+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7860109c70 con 0x7f7860107760 2024-03-24T09:22:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.570+0000 7f78577fe640 1 -- 172.21.15.116:0/139891957 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f78580bcde0 con 0x7f7860107760 2024-03-24T09:22:11.570 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:22:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f784c0a2340 msgr2=0x7f784c0a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f784c0a2340 0x7f784c0a4800 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7860198b40 tx=0x7f785c005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 msgr2=0x7f7860197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 0x7f7860197a00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7858036040 tx=0x7f785802cac0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 shutdown_connections 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7860107760 0x7f7860197a00 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f784c0a2340 0x7f784c0a4800 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78600692d0 0x7f786019dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 --2- 172.21.15.116:0/139891957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7860068910 0x7f786019d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 >> 172.21.15.116:0/139891957 conn(0x7f7860077710 msgr2=0x7f78600fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 shutdown_connections 2024-03-24T09:22:11.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.572+0000 7f7868525640 1 -- 172.21.15.116:0/139891957 wait complete. 2024-03-24T09:22:11.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:22:11.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 -- 172.21.15.116:0/3632387558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe81005c0 msgr2=0x7f0fe8100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 --2- 172.21.15.116:0/3632387558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe81005c0 0x7f0fe8100a40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009900 tx=0x7f0fd802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:11.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 -- 172.21.15.116:0/3632387558 shutdown_connections 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 --2- 172.21.15.116:0/3632387558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe8100f80 0x7f0fe810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 --2- 172.21.15.116:0/3632387558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe81005c0 0x7f0fe8100a40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 --2- 172.21.15.116:0/3632387558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8106660 0x7f0fe8106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 -- 172.21.15.116:0/3632387558 >> 172.21.15.116:0/3632387558 conn(0x7f0fe8076990 msgr2=0x7f0fe8076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 -- 172.21.15.116:0/3632387558 shutdown_connections 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.684+0000 7f0ff0781640 1 -- 172.21.15.116:0/3632387558 wait complete. 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 Processor -- start 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 -- start start 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe81005c0 0x7f0fe819c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 0x7f0fe819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe8106660 0x7f0fe8196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe8196fb0 con 0x7f0fe8100f80 2024-03-24T09:22:11.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe8197120 con 0x7f0fe8106660 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0ff0781640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe81972e0 con 0x7f0fe81005c0 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 0x7f0fe819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 0x7f0fe819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56008/0 (socket says 172.21.15.116:56008) 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 -- 172.21.15.116:0/4212761198 learned_addr learned my addr 172.21.15.116:0/4212761198 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fee51d640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe81005c0 0x7f0fe819c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0feed1e640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe8106660 0x7f0fe8196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 -- 172.21.15.116:0/4212761198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe81005c0 msgr2=0x7f0fe819c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe81005c0 0x7f0fe819c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 -- 172.21.15.116:0/4212761198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe8106660 msgr2=0x7f0fe8196960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe8106660 0x7f0fe8196960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fedd1c640 1 -- 172.21.15.116:0/4212761198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe0009840 con 0x7f0fe8100f80 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.685+0000 7f0fee51d640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe81005c0 0x7f0fe819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.686+0000 7f0fedd1c640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 0x7f0fe819cca0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8005940 tx=0x7f0fd8005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:11.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.686+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80035f0 con 0x7f0fe8100f80 2024-03-24T09:22:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.686+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd8009560 con 0x7f0fe8100f80 2024-03-24T09:22:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.686+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0fd8040070 con 0x7f0fe8100f80 2024-03-24T09:22:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.686+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8005320 con 0x7f0fe8100f80 2024-03-24T09:22:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.686+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe8069d60 con 0x7f0fe8100f80 2024-03-24T09:22:11.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.687+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe8101d00 con 0x7f0fe8100f80 2024-03-24T09:22:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.688+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f0fd8005480 con 0x7f0fe8100f80 2024-03-24T09:22:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.688+0000 7f0fdf7fe640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0fd40a22c0 0x7f0fd40a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.688+0000 7f0fee51d640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0fd40a22c0 0x7f0fd40a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.688+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0fd8042070 con 0x7f0fe8100f80 2024-03-24T09:22:11.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.689+0000 7f0fee51d640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0fd40a22c0 0x7f0fd40a4780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8105750 tx=0x7f0fe4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:11.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.693+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f0fd80bb0e0 con 0x7f0fe8100f80 2024-03-24T09:22:11.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.907+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fe8110600 con 0x7f0fe8100f80 2024-03-24T09:22:11.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.908+0000 7f0fdf7fe640 1 -- 172.21.15.116:0/4212761198 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7f0fd80be7a0 con 0x7f0fe8100f80 2024-03-24T09:22:11.908 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:11.908 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:22:05.111617+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24422,24443,24451,24484,24515]},"id":1}]} 2024-03-24T09:22:11.909 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0fd40a22c0 msgr2=0x7f0fd40a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0fd40a22c0 0x7f0fd40a4780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8105750 tx=0x7f0fe4005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 msgr2=0x7f0fe819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 0x7f0fe819cca0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8005940 tx=0x7f0fd8005df0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 shutdown_connections 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0fe8106660 0x7f0fe8196960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fe8100f80 0x7f0fe819cca0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0fd40a22c0 0x7f0fd40a4780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 --2- 172.21.15.116:0/4212761198 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0fe81005c0 0x7f0fe819c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 >> 172.21.15.116:0/4212761198 conn(0x7f0fe8076990 msgr2=0x7f0fe80fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 shutdown_connections 2024-03-24T09:22:11.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:11.910+0000 7f0ff0781640 1 -- 172.21.15.116:0/4212761198 wait complete. 2024-03-24T09:22:11.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/3578433923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 msgr2=0x7fcd88100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/3578433923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd88100a50 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009900 tx=0x7fcd7802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/3578433923 shutdown_connections 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/3578433923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd88100f90 0x7fcd8810d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/3578433923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd88100a50 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/3578433923 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd88106690 0x7fcd88106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/3578433923 >> 172.21.15.116:0/3578433923 conn(0x7fcd88076990 msgr2=0x7fcd88076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/3578433923 shutdown_connections 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/3578433923 wait complete. 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 Processor -- start 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.024+0000 7fcd8d4ae640 1 -- start start 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd8819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd88100f90 0x7fcd8819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd88106690 0x7fcd881979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd88198040 con 0x7fcd881005d0 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcd881981b0 con 0x7fcd88106690 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcd88198370 con 0x7fcd88100f90 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd8819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd8819d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56036/0 (socket says 172.21.15.116:56036) 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 -- 172.21.15.116:0/2309051213 learned_addr learned my addr 172.21.15.116:0/2309051213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd877fe640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd88106690 0x7fcd881979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd867fc640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd88100f90 0x7fcd8819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 -- 172.21.15.116:0/2309051213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd88100f90 msgr2=0x7fcd8819dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd88100f90 0x7fcd8819dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 -- 172.21.15.116:0/2309051213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd88106690 msgr2=0x7fcd881979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd88106690 0x7fcd881979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 -- 172.21.15.116:0/2309051213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd7c009490 con 0x7fcd881005d0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd86ffd640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd8819d7f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00ea60 tx=0x7fcd7c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c00cde0 con 0x7fcd881005d0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcd7c0044d0 con 0x7fcd881005d0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd78009560 con 0x7fcd881005d0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7c010690 con 0x7fcd881005d0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.025+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd881a46b0 con 0x7fcd881005d0 2024-03-24T09:22:12.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.026+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd88101cf0 con 0x7fcd881005d0 2024-03-24T09:22:12.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.027+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fcd7c003e40 con 0x7fcd881005d0 2024-03-24T09:22:12.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.028+0000 7fcd6ffff640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcd700a22c0 0x7fcd700a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.028+0000 7fcd867fc640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcd700a22c0 0x7fcd700a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.028+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcd7c014070 con 0x7fcd881005d0 2024-03-24T09:22:12.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.028+0000 7fcd867fc640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcd700a22c0 0x7fcd700a4780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd78006010 tx=0x7fcd7802c6c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.032+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fcd7c0971e0 con 0x7fcd881005d0 2024-03-24T09:22:12.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.211+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd88068950 con 0x7fcd881005d0 2024-03-24T09:22:12.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.213+0000 7fcd6ffff640 1 -- 172.21.15.116:0/2309051213 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25293 (secure 0 0 0) 0x7fcd88068950 con 0x7fcd881005d0 2024-03-24T09:22:12.213 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:12.213 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:22:05.255869+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:22:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcd700a22c0 msgr2=0x7fcd700a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcd700a22c0 0x7fcd700a4780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd78006010 tx=0x7fcd7802c6c0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 msgr2=0x7fcd8819d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd8819d7f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c00ea60 tx=0x7fcd7c00ef20 comp rx=0 tx=0).stop 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 shutdown_connections 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd88106690 0x7fcd881979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcd700a22c0 0x7fcd700a4780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd88100f90 0x7fcd8819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 --2- 172.21.15.116:0/2309051213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd881005d0 0x7fcd8819d7f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 >> 172.21.15.116:0/2309051213 conn(0x7fcd88076990 msgr2=0x7fcd88069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 shutdown_connections 2024-03-24T09:22:12.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.215+0000 7fcd8d4ae640 1 -- 172.21.15.116:0/2309051213 wait complete. 2024-03-24T09:22:12.226 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:12 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/104521813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:12.226 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:12 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/139891957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:12.226 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:12 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4212761198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:12.227 DEBUG:tasks.fwd_scrub:Yielding 2024-03-24T09:22:12.227 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T09:22:12.237 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T09:22:12.238 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:22:12.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-24T09:22:12.238 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-24T09:22:12.238 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:22:12.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:12.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 -- 172.21.15.116:0/389071075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96901076e0 msgr2=0x7f9690107ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 --2- 172.21.15.116:0/389071075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96901076e0 0x7f9690107ac0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f968c009930 tx=0x7f968c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 -- 172.21.15.116:0/389071075 shutdown_connections 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 --2- 172.21.15.116:0/389071075 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690069750 0x7f969010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 --2- 172.21.15.116:0/389071075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9690068d90 0x7f9690069210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 --2- 172.21.15.116:0/389071075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96901076e0 0x7f9690107ac0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.344+0000 7f96972eb640 1 -- 172.21.15.116:0/389071075 >> 172.21.15.116:0/389071075 conn(0x7f9690077460 msgr2=0x7f9690077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 -- 172.21.15.116:0/389071075 shutdown_connections 2024-03-24T09:22:12.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 -- 172.21.15.116:0/389071075 wait complete. 2024-03-24T09:22:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 -- 172.21.15.116:0/2111959281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6181018e0 msgr2=0x7fb6181051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 --2- 172.21.15.116:0/2111959281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6181018e0 0x7fb6181051b0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb608009860 tx=0x7fb60802bb70 comp rx=0 tx=0).stop 2024-03-24T09:22:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 -- 172.21.15.116:0/2111959281 shutdown_connections 2024-03-24T09:22:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 --2- 172.21.15.116:0/2111959281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb6181058e0 0x7fb61810d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 --2- 172.21.15.116:0/2111959281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6181018e0 0x7fb6181051b0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 --2- 172.21.15.116:0/2111959281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb618100f30 0x7fb618101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 -- 172.21.15.116:0/2111959281 >> 172.21.15.116:0/2111959281 conn(0x7fb618076b30 msgr2=0x7fb6180fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 Processor -- start 2024-03-24T09:22:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 -- 172.21.15.116:0/2111959281 shutdown_connections 2024-03-24T09:22:12.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7fb61f61e640 1 -- 172.21.15.116:0/2111959281 wait complete. 2024-03-24T09:22:12.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 -- start start 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690068d90 0x7f969019d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 0x7f969019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901076e0 0x7f9690197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9690198090 con 0x7f9690069750 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9690198200 con 0x7f96901076e0 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f96972eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f96901983c0 con 0x7f9690068d90 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f9694886640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 0x7f969019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f9694886640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 0x7f969019ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56060/0 (socket says 172.21.15.116:56060) 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f9695087640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690068d90 0x7f969019d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.345+0000 7f9694886640 1 -- 172.21.15.116:0/963170048 learned_addr learned my addr 172.21.15.116:0/963170048 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9695888640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901076e0 0x7f9690197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 Processor -- start 2024-03-24T09:22:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9694886640 1 -- 172.21.15.116:0/963170048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690068d90 msgr2=0x7f969019d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9694886640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690068d90 0x7f969019d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9694886640 1 -- 172.21.15.116:0/963170048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901076e0 msgr2=0x7f9690197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9694886640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901076e0 0x7f9690197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 -- start start 2024-03-24T09:22:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9694886640 1 -- 172.21.15.116:0/963170048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c009560 con 0x7f9690069750 2024-03-24T09:22:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9695087640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690068d90 0x7f969019d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f9694886640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 0x7f969019ddd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f968000e9f0 tx=0x7f968000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb618100f30 0x7fb618199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6181058e0 0x7fb61819a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 0x7fb61819ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb61819f170 con 0x7fb61819e780 2024-03-24T09:22:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb61819f2e0 con 0x7fb6181058e0 2024-03-24T09:22:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61f61e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb61819f450 con 0x7fb618100f30 2024-03-24T09:22:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680009700 con 0x7f9690069750 2024-03-24T09:22:12.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61dbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 0x7fb61819ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96901986a0 con 0x7f9690069750 2024-03-24T09:22:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61d3ba640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb618100f30 0x7fb618199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61dbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 0x7fb61819ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56072/0 (socket says 172.21.15.116:56072) 2024-03-24T09:22:12.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f968000cd80 con 0x7f9690069750 2024-03-24T09:22:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680018840 con 0x7f9690069750 2024-03-24T09:22:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61dbbb640 1 -- 172.21.15.116:0/3636889116 learned_addr learned my addr 172.21.15.116:0/3636889116 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6181058e0 0x7fb61819a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96901a4930 con 0x7f9690069750 2024-03-24T09:22:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61d3ba640 1 -- 172.21.15.116:0/3636889116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6181058e0 msgr2=0x7fb61819a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6181058e0 0x7fb61819a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61d3ba640 1 -- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 msgr2=0x7fb61819ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 0x7fb61819ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61d3ba640 1 -- 172.21.15.116:0/3636889116 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb608009490 con 0x7fb618100f30 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.346+0000 7fb61dbbb640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 0x7fb61819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb618100f30 0x7fb618199ec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb61400e790 tx=0x7fb61400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7fb60e7fc640 1 -- 172.21.15.116:0/3636889116 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61400ccc0 con 0x7fb618100f30 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6181a6a30 con 0x7fb618100f30 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7fb60e7fc640 1 -- 172.21.15.116:0/3636889116 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb61400eea0 con 0x7fb618100f30 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7fb60e7fc640 1 -- 172.21.15.116:0/3636889116 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614018680 con 0x7fb618100f30 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6181a6f00 con 0x7fb618100f30 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.347+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f969004f470 con 0x7f9690069750 2024-03-24T09:22:12.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.349+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f96800189a0 con 0x7f9690069750 2024-03-24T09:22:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.349+0000 7fb60e7fc640 1 -- 172.21.15.116:0/3636889116 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fb614010430 con 0x7fb618100f30 2024-03-24T09:22:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7f96867fc640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2370 0x7f967c0a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7f9695087640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2370 0x7f967c0a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9680014070 con 0x7f9690069750 2024-03-24T09:22:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb60e7fc640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb6040a2340 0x7fb6040a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 Processor -- start 2024-03-24T09:22:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 -- start start 2024-03-24T09:22:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb6040a2340 0x7fb6040a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb60e7fc640 1 -- 172.21.15.116:0/3636889116 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb614014070 con 0x7fb618100f30 2024-03-24T09:22:12.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7f9695087640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2370 0x7f967c0a4830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f968c005ab0 tx=0x7f968c02c6d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc53b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a6fbc1530 con 0x564a6fc54080 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a6fbbea60 con 0x564a6fc53760 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.350+0000 7fb6221fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a6fbba240 con 0x564a6fc5cc60 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc53b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61cbb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc53b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:53276/0 (socket says 172.21.15.116:53276) 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 -- 172.21.15.116:0/297930713 learned_addr learned my addr 172.21.15.116:0/297930713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61dbbb640 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 -- 172.21.15.116:0/297930713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 msgr2=0x564a6fc5f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 -- 172.21.15.116:0/297930713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 msgr2=0x564a6fc5c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 -- 172.21.15.116:0/297930713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6fbb88c0 con 0x564a6fc53760 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb6040a2340 0x7fb6040a4800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb608009830 tx=0x7fb608034e80 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc53b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c6a0 tx=0x7fb61409a840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d4c9c93ac78e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb6037fe640 1 -- 172.21.15.116:0/297930713 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61409ed20 con 0x564a6fc53760 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb6037fe640 1 -- 172.21.15.116:0/297930713 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb6140a3370 con 0x564a6fc53760 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.351+0000 7fb6037fe640 1 -- 172.21.15.116:0/297930713 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6140cbc90 con 0x564a6fc53760 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 -- 172.21.15.116:0/297930713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 msgr2=0x564a6fc53b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc53b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c6a0 tx=0x7fb61409a840 comp rx=0 tx=0).stop 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 -- 172.21.15.116:0/297930713 shutdown_connections 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 --2- 172.21.15.116:0/297930713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc53b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 -- 172.21.15.116:0/297930713 >> 172.21.15.116:0/297930713 conn(0x564a6fc4de90 msgr2=0x564a6fc4e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 -- 172.21.15.116:0/297930713 shutdown_connections 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 -- 172.21.15.116:0/297930713 wait complete. 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 -- start start 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.352+0000 7fb6221fd740 1 Processor -- start 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc59c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a6fb5ef80 con 0x564a6fc54080 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a6f88d4b0 con 0x564a6fc53760 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a6fb3fad0 con 0x564a6fc5cc60 2024-03-24T09:22:12.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb6221fd740 10 client.0 _ll_register_callbacks cb 0x564a6fb72b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:12.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc59c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61cbb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc59c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:53280/0 (socket says 172.21.15.116:53280) 2024-03-24T09:22:12.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 -- 172.21.15.116:0/96224971 learned_addr learned my addr 172.21.15.116:0/96224971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61dbbb640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9680097590 con 0x7f9690069750 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 -- 172.21.15.116:0/96224971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 msgr2=0x564a6fc5a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 -- 172.21.15.116:0/96224971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 msgr2=0x564a6fc5a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 -- 172.21.15.116:0/96224971 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6181a6f00 con 0x564a6fc53760 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61dbbb640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc59c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6140cbac0 tx=0x7fb614009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911c7003ff8ccd8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.353+0000 7fb600ff9640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb600ff9640 1 -- 172.21.15.116:0/96224971 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61409ed20 con 0x564a6fc53760 2024-03-24T09:22:12.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb600ff9640 1 -- 172.21.15.116:0/96224971 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb6140cbc90 con 0x564a6fc53760 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a6fbde7a0 con 0x564a6fc53760 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb600ff9640 1 -- 172.21.15.116:0/96224971 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb6140cbe50 con 0x564a6fc53760 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 10 client.25595 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564a6fb706d0 con 0x564a6fc53760 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb600ff9640 1 -- 172.21.15.116:0/96224971 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb6140a7020 con 0x564a6fc53760 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb600ff9640 1 -- 172.21.15.116:0/96224971 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb614096af0 con 0x564a6fc53760 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 10 client.25595 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 10 client.25595 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 20 client.25595 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fc6ca80 0x564a6fc6ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 4 client.25595 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a6fbb01f0 con 0x564a6fc6ca80 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.354+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fc6ca80 0x564a6fc6ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fc6ca80 0x564a6fc6ee90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10c0ac81eff8b66a server_cookie=6826f70a8baf7b9d in_seq=0 out_seq=0 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb600ff9640 10 client.25595 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb600ff9640 1 -- 172.21.15.116:0/96224971 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a6fbb01f0 con 0x564a6fc6ca80 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb600ff9640 10 client.25595 handle_command_reply: tid=0 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb6221fd740 1 client.25595 shutdown 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 msgr2=0x564a6fc59c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.355+0000 7fb6221fd740 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc59c90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6140cbac0 tx=0x7fb614009600 comp rx=0 tx=0).stop 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 shutdown_connections 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fc5a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc5a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fc6ca80 0x564a6fc6ee90 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 --2- 172.21.15.116:0/96224971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc59c90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 >> 172.21.15.116:0/96224971 conn(0x564a6fc4de90 msgr2=0x564a6fc4fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb600ff9640 0 client.25595 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 shutdown_connections 2024-03-24T09:22:12.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 1 -- 172.21.15.116:0/96224971 wait complete. 2024-03-24T09:22:12.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.356+0000 7fb6221fd740 20 client.25595 trim_cache size 0 max 16384 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 Processor -- start 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 -- start start 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc53760 0x564a6fc5c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc684f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc5cc60 0x564a6fc74770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a6fc4ee90 con 0x564a6fc54080 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a6fc4c350 con 0x564a6fc5cc60 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb6221fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a6fc697e0 con 0x564a6fc53760 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb61cbb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc684f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb61dbbb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc5cc60 0x564a6fc74770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb61cbb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc684f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56104/0 (socket says 172.21.15.116:56104) 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb61cbb9640 1 -- 172.21.15.116:0/774165496 learned_addr learned my addr 172.21.15.116:0/774165496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.373+0000 7fb61d3ba640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc53760 0x564a6fc5c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61cbb9640 1 -- 172.21.15.116:0/774165496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc53760 msgr2=0x564a6fc5c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc53760 0x564a6fc5c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61cbb9640 1 -- 172.21.15.116:0/774165496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc5cc60 msgr2=0x564a6fc74770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc5cc60 0x564a6fc74770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61cbb9640 1 -- 172.21.15.116:0/774165496 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a6fb54c60 con 0x564a6fc54080 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61dbbb640 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc5cc60 0x564a6fc74770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc684f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb608000c00 tx=0x7fb60802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d857addf67550f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb602ffd640 1 -- 172.21.15.116:0/774165496 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608053070 con 0x564a6fc54080 2024-03-24T09:22:12.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb602ffd640 1 -- 172.21.15.116:0/774165496 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb60802ce90 con 0x564a6fc54080 2024-03-24T09:22:12.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.374+0000 7fb602ffd640 1 -- 172.21.15.116:0/774165496 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608055570 con 0x564a6fc54080 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 -- 172.21.15.116:0/774165496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 msgr2=0x564a6fc684f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc684f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb608000c00 tx=0x7fb60802e050 comp rx=0 tx=0).stop 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 -- 172.21.15.116:0/774165496 shutdown_connections 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc5cc60 0x564a6fc74770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc684f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 --2- 172.21.15.116:0/774165496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc53760 0x564a6fc5c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 -- 172.21.15.116:0/774165496 >> 172.21.15.116:0/774165496 conn(0x564a6fc4de90 msgr2=0x564a6fc4f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 -- 172.21.15.116:0/774165496 shutdown_connections 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 -- 172.21.15.116:0/774165496 wait complete. 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.375+0000 7fb6221fd740 1 -- start start 2024-03-24T09:22:12.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 Processor -- start 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc56f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc574b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fcd1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a6fcd22e0 con 0x564a6fc54080 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a6fcd2420 con 0x564a6fc53760 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a6fcd2560 con 0x564a6fc5cc60 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb6221fd740 10 client.0 _ll_register_callbacks cb 0x7fb61409ab20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb61cbb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb61cbb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56106/0 (socket says 172.21.15.116:56106) 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb61cbb9640 1 -- 172.21.15.116:0/985444174 learned_addr learned my addr 172.21.15.116:0/985444174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc56f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.376+0000 7fb61dbbb640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fcd1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb61cbb9640 1 -- 172.21.15.116:0/985444174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 msgr2=0x564a6fcd1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fcd1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb61cbb9640 1 -- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 msgr2=0x564a6fc56f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc56f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb61cbb9640 1 -- 172.21.15.116:0/985444174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61000f910 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb61cbb9640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc574b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb608037040 tx=0x7fb608005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992f9afd2414ee74 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb600ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb600ff9640 1 -- 172.21.15.116:0/985444174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608053070 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb600ff9640 1 -- 172.21.15.116:0/985444174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb608034650 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6140a3370 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb600ff9640 1 -- 172.21.15.116:0/985444174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb60802c710 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 10 client.15765 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x7fb608049070 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb600ff9640 1 -- 172.21.15.116:0/985444174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb6140a3370 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb600ff9640 1 -- 172.21.15.116:0/985444174 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fb608066070 con 0x564a6fc54080 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 10 client.15765 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:12.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 20 client.15765 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fcd3af0 0x564a6fcd5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 4 client.15765 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.377+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564a6f8b9860 con 0x564a6fcd3af0 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.378+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fcd3af0 0x564a6fcd5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.378+0000 7fb600ff9640 10 client.15765 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.378+0000 7fb61d3ba640 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fcd3af0 0x564a6fcd5f10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93327933e09f0790 server_cookie=1d3a96fd6eb82 in_seq=0 out_seq=0 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.379+0000 7fb600ff9640 1 -- 172.21.15.116:0/985444174 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564a6f8b9860 con 0x564a6fcd3af0 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.379+0000 7fb600ff9640 10 client.15765 handle_command_reply: tid=0 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.379+0000 7fb6221fd740 1 client.15765 shutdown 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.379+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 msgr2=0x564a6fc574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.379+0000 7fb6221fd740 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc574b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb608037040 tx=0x7fb608005fa0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 shutdown_connections 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a6fc5cc60 0x564a6fcd1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a6fc54080 0x564a6fc574b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a6fcd3af0 0x564a6fcd5f10 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb600ff9640 0 client.15765 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 --2- 172.21.15.116:0/985444174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a6fc53760 0x564a6fc56f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 >> 172.21.15.116:0/985444174 conn(0x564a6fc4de90 msgr2=0x564a6fc4fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 shutdown_connections 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 1 -- 172.21.15.116:0/985444174 wait complete. 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb6221fd740 20 client.15765 trim_cache size 0 max 16384 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "30e181c9-0ec4-4905-8d12-d63bfb3c8df3", 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:22:12.385 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:22:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb6040a2340 msgr2=0x7fb6040a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb6040a2340 0x7fb6040a4800 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb608009830 tx=0x7fb608034e80 comp rx=0 tx=0).stop 2024-03-24T09:22:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb618100f30 msgr2=0x7fb618199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb618100f30 0x7fb618199ec0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb61400e790 tx=0x7fb61400ec50 comp rx=0 tx=0).stop 2024-03-24T09:22:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 shutdown_connections 2024-03-24T09:22:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61819e780 0x7fb61819ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6181058e0 0x7fb61819a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb6040a2340 0x7fb6040a4800 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 --2- 172.21.15.116:0/3636889116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb618100f30 0x7fb618199ec0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.380+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 >> 172.21.15.116:0/3636889116 conn(0x7fb618076b30 msgr2=0x7fb618104520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.381+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 shutdown_connections 2024-03-24T09:22:12.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.381+0000 7fb61f61e640 1 -- 172.21.15.116:0/3636889116 wait complete. 2024-03-24T09:22:12.392 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:22:12.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:22:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3210997600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 msgr2=0x7f5fc010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3210997600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc010d420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009ca0 tx=0x7f5fb802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3210997600 shutdown_connections 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3210997600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc010d420 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3210997600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 0x7f5fc0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3210997600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc0106660 0x7f5fc0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3210997600 >> 172.21.15.116:0/3210997600 conn(0x7f5fc0077710 msgr2=0x7f5fc0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3210997600 shutdown_connections 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3210997600 wait complete. 2024-03-24T09:22:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.498+0000 7f5fc60c2640 1 Processor -- start 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 -- start start 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 0x7f5fc019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc0106660 0x7f5fc0196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc0196ec0 con 0x7f5fc0100f60 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc0197030 con 0x7f5fc01005a0 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc01971f0 con 0x7f5fc0106660 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 0x7f5fc019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc019cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56128/0 (socket says 172.21.15.116:56128) 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 -- 172.21.15.116:0/3873673701 learned_addr learned my addr 172.21.15.116:0/3873673701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbffff640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc0106660 0x7f5fc0196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 -- 172.21.15.116:0/3873673701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc0106660 msgr2=0x7f5fc0196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc0106660 0x7f5fc0196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 -- 172.21.15.116:0/3873673701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 msgr2=0x7f5fc019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 0x7f5fc019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 -- 172.21.15.116:0/3873673701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb8009900 con 0x7f5fc0100f60 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 0x7f5fc019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc019cc00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000e9f0 tx=0x7f5fb000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fbcff9640 1 -- 172.21.15.116:0/3873673701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0009700 con 0x7f5fc0100f60 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.499+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc01974d0 con 0x7f5fc0100f60 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.500+0000 7f5fbcff9640 1 -- 172.21.15.116:0/3873673701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fb000cd80 con 0x7f5fc0100f60 2024-03-24T09:22:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.500+0000 7f5fbcff9640 1 -- 172.21.15.116:0/3873673701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0018840 con 0x7f5fc0100f60 2024-03-24T09:22:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.500+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc01a3760 con 0x7f5fc0100f60 2024-03-24T09:22:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fbcff9640 1 -- 172.21.15.116:0/3873673701 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f5fb00189a0 con 0x7f5fc0100f60 2024-03-24T09:22:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 Processor -- start 2024-03-24T09:22:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 -- start start 2024-03-24T09:22:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fbcff9640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5fa80a22c0 0x7f5fa80a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e445b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 0x563b40e4d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e4d6d0 0x563b40e4fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563b40d68f30 con 0x563b40e4d6d0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b40db2810 con 0x563b40e441d0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563b40dafd40 con 0x563b40e44af0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.502+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 0x563b40e4d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbffff640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5fa80a22c0 0x7f5fa80a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e445b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:53328/0 (socket says 172.21.15.116:53328) 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/1940171579 learned_addr learned my addr 172.21.15.116:0/1940171579 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 0x563b40e4d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40364/0 (socket says 172.21.15.116:40364) 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbcff9640 1 -- 172.21.15.116:0/3873673701 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5fb0014070 con 0x7f5fc0100f60 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbffff640 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e4d6d0 0x563b40e4fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/1940171579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 msgr2=0x563b40e4d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 0x563b40e4d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/1940171579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e4d6d0 msgr2=0x563b40e4fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e4d6d0 0x563b40e4fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/1940171579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b40d6cea0 con 0x563b40e441d0 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 0x563b40e4d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e445b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40114f0 tx=0x7f5fb40119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eab5026e78b7e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fbffff640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5fa80a22c0 0x7f5fa80a4780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8004af0 tx=0x7f5fb8032510 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fa5ffb640 1 -- 172.21.15.116:0/1940171579 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4004030 con 0x563b40e441d0 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fa5ffb640 1 -- 172.21.15.116:0/1940171579 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fb40035f0 con 0x563b40e441d0 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.503+0000 7f5fa5ffb640 1 -- 172.21.15.116:0/1940171579 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb400f840 con 0x563b40e441d0 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 -- 172.21.15.116:0/1940171579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 msgr2=0x563b40e445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e445b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5fb40114f0 tx=0x7f5fb40119b0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 -- 172.21.15.116:0/1940171579 shutdown_connections 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e4d6d0 0x563b40e4fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e44af0 0x563b40e4d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/1940171579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e445b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 -- 172.21.15.116:0/1940171579 >> 172.21.15.116:0/1940171579 conn(0x563b40e3e900 msgr2=0x563b40e3ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 -- 172.21.15.116:0/1940171579 shutdown_connections 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.504+0000 7f5fc8d76740 1 -- 172.21.15.116:0/1940171579 wait complete. 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 -- start start 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 Processor -- start 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 0x563b40e4a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e4ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e4d6d0 0x563b40e4b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563b40dab520 con 0x563b40e44af0 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b40a7e430 con 0x563b40e4d6d0 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563b40a3ae00 con 0x563b40e441d0 2024-03-24T09:22:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fc8d76740 10 client.0 _ll_register_callbacks cb 0x563b40d49d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e4ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 0x563b40e4a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fbffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e4d6d0 0x563b40e4b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e4ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56144/0 (socket says 172.21.15.116:56144) 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 0x563b40e4a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40378/0 (socket says 172.21.15.116:40378) 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.505+0000 7f5fbeffd640 1 -- 172.21.15.116:0/393364688 learned_addr learned my addr 172.21.15.116:0/393364688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbffff640 1 -- 172.21.15.116:0/393364688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 msgr2=0x563b40e4a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbffff640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 0x563b40e4a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbffff640 1 -- 172.21.15.116:0/393364688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 msgr2=0x563b40e4ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbffff640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e4ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbffff640 1 -- 172.21.15.116:0/393364688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb4009cc0 con 0x563b40e4d6d0 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e4ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 0x563b40e4a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fbffff640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e4d6d0 0x563b40e4b1c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5fb802c630 tx=0x7f5fb8005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415322beecd0d02b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5f9effd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5f9effd640 1 -- 172.21.15.116:0/393364688 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8003d80 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b40aaa7e0 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5f9effd640 1 -- 172.21.15.116:0/393364688 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5fb8004040 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5f9effd640 1 -- 172.21.15.116:0/393364688 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5fb8032e80 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fc8d76740 10 client.25613 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563b40da9ba0 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.506+0000 7f5f9effd640 1 -- 172.21.15.116:0/393364688 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563b40aaa7e0 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5f9effd640 1 -- 172.21.15.116:0/393364688 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5fb804d070 con 0x563b40e4d6d0 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fc8d76740 10 client.25613 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fc8d76740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fc8d76740 20 client.25613 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40e5db70 0x563b40e5ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fc8d76740 4 client.25613 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b40ae7720 con 0x563b40e5db70 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40e5db70 0x563b40e5ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5f9effd640 10 client.25613 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.507+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40e5db70 0x563b40e5ff60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e56ef9c81c02293 server_cookie=98ae1fb5bc91b4c4 in_seq=0 out_seq=0 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5f9effd640 1 -- 172.21.15.116:0/393364688 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563b40ae7720 con 0x563b40e5db70 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5f9effd640 10 client.25613 handle_command_reply: tid=0 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 client.25613 shutdown 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e4d6d0 msgr2=0x563b40e4b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e4d6d0 0x563b40e4b1c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5fb802c630 tx=0x7f5fb8005c50 comp rx=0 tx=0).stop 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 shutdown_connections 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40e5db70 0x563b40e5ff60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e4d6d0 0x563b40e4b1c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e4ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5f9effd640 0 client.25613 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/393364688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e441d0 0x563b40e4a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 >> 172.21.15.116:0/393364688 conn(0x563b40e3e900 msgr2=0x563b40e3db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 shutdown_connections 2024-03-24T09:22:12.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 1 -- 172.21.15.116:0/393364688 wait complete. 2024-03-24T09:22:12.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.508+0000 7f5fc8d76740 20 client.25613 trim_cache size 0 max 16384 2024-03-24T09:22:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:12 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/104521813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:12 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/139891957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:12 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4212761198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 Processor -- start 2024-03-24T09:22:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 -- start start 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e44ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e58f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40e4afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563b40d30430 con 0x563b40e44af0 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b40e5a630 con 0x563b40e441d0 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563b40e43970 con 0x563b40e4d6d0 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e58f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e58f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56146/0 (socket says 172.21.15.116:56146) 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fbeffd640 1 -- 172.21.15.116:0/4281050603 learned_addr learned my addr 172.21.15.116:0/4281050603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e44ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.526+0000 7f5fbffff640 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40e4afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fbeffd640 1 -- 172.21.15.116:0/4281050603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 msgr2=0x563b40e4afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40e4afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fbeffd640 1 -- 172.21.15.116:0/4281050603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 msgr2=0x563b40e44ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e44ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fbeffd640 1 -- 172.21.15.116:0/4281050603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b40d45b20 con 0x563b40e44af0 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e58f60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5fb009ab90 tx=0x7f5fb001de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b372b2a3b8fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fa57fa640 1 -- 172.21.15.116:0/4281050603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb009a7f0 con 0x563b40e44af0 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fa57fa640 1 -- 172.21.15.116:0/4281050603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fb00b3cf0 con 0x563b40e44af0 2024-03-24T09:22:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.527+0000 7f5fa57fa640 1 -- 172.21.15.116:0/4281050603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb00b3ed0 con 0x563b40e44af0 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- 172.21.15.116:0/4281050603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 msgr2=0x563b40e58f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e58f60 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5fb009ab90 tx=0x7f5fb001de60 comp rx=0 tx=0).stop 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- 172.21.15.116:0/4281050603 shutdown_connections 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40e4afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e44af0 0x563b40e58f60 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/4281050603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e441d0 0x563b40e44ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- 172.21.15.116:0/4281050603 >> 172.21.15.116:0/4281050603 conn(0x563b40e3e900 msgr2=0x563b40e3db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- 172.21.15.116:0/4281050603 shutdown_connections 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- 172.21.15.116:0/4281050603 wait complete. 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- start start 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 Processor -- start 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 0x563b40e476a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e44af0 0x563b40e47be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40ec2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563b40e48390 con 0x563b40e441d0 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b40e48500 con 0x563b40e44af0 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563b40ec3410 con 0x563b40e4d6d0 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.528+0000 7f5fc8d76740 10 client.0 _ll_register_callbacks cb 0x7f5fb0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 0x563b40e476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 0x563b40e476a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56162/0 (socket says 172.21.15.116:56162) 2024-03-24T09:22:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/2508719834 learned_addr learned my addr 172.21.15.116:0/2508719834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e44af0 0x563b40e47be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbffff640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40ec2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/2508719834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 msgr2=0x563b40ec2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40ec2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e44af0 msgr2=0x563b40e47be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e44af0 0x563b40e47be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 -- 172.21.15.116:0/2508719834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc01974d0 con 0x563b40e441d0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e44af0 0x563b40e47be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fbf7fe640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 0x563b40e476a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009250 tx=0x7f5fb40113d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a7e584c0fb52944 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5f9effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5f9effd640 1 -- 172.21.15.116:0/2508719834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb4004120 con 0x563b40e441d0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5f9effd640 1 -- 172.21.15.116:0/2508719834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5fb4005250 con 0x563b40e441d0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.529+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fb40035f0 con 0x563b40e441d0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5f9effd640 1 -- 172.21.15.116:0/2508719834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5fb4005430 con 0x563b40e441d0 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 10 client.15783 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563b40d6b700 con 0x563b40e441d0 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5f9effd640 1 -- 172.21.15.116:0/2508719834 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5fb40035f0 con 0x563b40e441d0 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5f9effd640 1 -- 172.21.15.116:0/2508719834 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5fb400f820 con 0x563b40e441d0 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 10 client.15783 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 20 client.15783 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40ec4c70 0x563b40ec7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 4 client.15783 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b40b36890 con 0x563b40ec4c70 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.530+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40ec4c70 0x563b40ec7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fbeffd640 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40ec4c70 0x563b40ec7090 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac6620dc98591344 server_cookie=99e1b522faf1ae81 in_seq=0 out_seq=0 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5f9effd640 10 client.15783 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5f9effd640 1 -- 172.21.15.116:0/2508719834 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563b40b36890 con 0x563b40ec4c70 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5f9effd640 10 client.15783 handle_command_reply: tid=0 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 client.15783 shutdown 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 msgr2=0x563b40e476a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 0x563b40e476a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f5fb4009250 tx=0x7f5fb40113d0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 shutdown_connections 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563b40e4d6d0 0x563b40ec2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x563b40ec4c70 0x563b40ec7090 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5f9effd640 0 client.15783 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b40e44af0 0x563b40e47be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 --2- 172.21.15.116:0/2508719834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563b40e441d0 0x563b40e476a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 >> 172.21.15.116:0/2508719834 conn(0x563b40e3e900 msgr2=0x563b40e3fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 shutdown_connections 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 1 -- 172.21.15.116:0/2508719834 wait complete. 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.531+0000 7f5fc8d76740 20 client.15783 trim_cache size 0 max 16384 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:22:12.537 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "30e181c9-0ec4-4905-8d12-d63bfb3c8df3": { 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "30e181c9-0ec4-4905-8d12-d63bfb3c8df3", 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:22:12.538 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5fa80a22c0 msgr2=0x7f5fa80a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5fa80a22c0 0x7f5fa80a4780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8004af0 tx=0x7f5fb8032510 comp rx=0 tx=0).stop 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 msgr2=0x7f5fc019cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc019cc00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5fb000e9f0 tx=0x7f5fb000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 shutdown_connections 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5fa80a22c0 0x7f5fa80a4780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5fc0106660 0x7f5fc0196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5fc0100f60 0x7f5fc019cc00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 --2- 172.21.15.116:0/3873673701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc01005a0 0x7f5fc019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 >> 172.21.15.116:0/3873673701 conn(0x7f5fc0077710 msgr2=0x7f5fc00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 shutdown_connections 2024-03-24T09:22:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.532+0000 7f5fc60c2640 1 -- 172.21.15.116:0/3873673701 wait complete. 2024-03-24T09:22:12.543 INFO:tasks.cephfs.filesystem:scrub status for tag:30e181c9-0ec4-4905-8d12-d63bfb3c8df3 - {'path': '/', 'tag': '30e181c9-0ec4-4905-8d12-d63bfb3c8df3', 'options': 'recursive,force'} 2024-03-24T09:22:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:12 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/104521813' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:22:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:12 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/139891957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:22:12.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:12 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4212761198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:12.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.566+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f9690111720 con 0x7f967c0a2370 2024-03-24T09:22:12.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.626+0000 7f96867fc640 1 -- 172.21.15.116:0/963170048 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9690111720 con 0x7f967c0a2370 2024-03-24T09:22:12.626 INFO:teuthology.orchestra.run.smithi116.stdout:Schedule set for path / 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2370 msgr2=0x7f967c0a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2370 0x7f967c0a4830 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f968c005ab0 tx=0x7f968c02c6d0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 msgr2=0x7f969019ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 0x7f969019ddd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f968000e9f0 tx=0x7f968000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 shutdown_connections 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901076e0 0x7f9690197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690069750 0x7f969019ddd0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2370 0x7f967c0a4830 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 --2- 172.21.15.116:0/963170048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690068d90 0x7f969019d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 >> 172.21.15.116:0/963170048 conn(0x7f9690077460 msgr2=0x7f96900fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 shutdown_connections 2024-03-24T09:22:12.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.628+0000 7f96972eb640 1 -- 172.21.15.116:0/963170048 wait complete. 2024-03-24T09:22:12.639 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 -- 172.21.15.116:0/755776782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601020a0 msgr2=0x7fc86010e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 --2- 172.21.15.116:0/755776782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601020a0 0x7fc86010e5b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc858009cd0 tx=0x7fc85802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 -- 172.21.15.116:0/755776782 shutdown_connections 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 --2- 172.21.15.116:0/755776782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601020a0 0x7fc86010e5b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 --2- 172.21.15.116:0/755776782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8601016e0 0x7fc860101b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 --2- 172.21.15.116:0/755776782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601076f0 0x7fc860107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 -- 172.21.15.116:0/755776782 >> 172.21.15.116:0/755776782 conn(0x7fc8600fd1f0 msgr2=0x7fc8600ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 -- 172.21.15.116:0/755776782 shutdown_connections 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.784+0000 7fc865db6640 1 -- 172.21.15.116:0/755776782 wait complete. 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 Processor -- start 2024-03-24T09:22:12.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 -- start start 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601016e0 0x7fc86019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8601020a0 0x7fc86019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 0x7fc8601979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc860198020 con 0x7fc8601076f0 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc860198190 con 0x7fc8601016e0 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc865db6640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc860198350 con 0x7fc8601020a0 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 0x7fc8601979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc85ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 0x7fc8601979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56170/0 (socket says 172.21.15.116:56170) 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc85f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601016e0 0x7fc86019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc85ffff640 1 -- 172.21.15.116:0/723709420 learned_addr learned my addr 172.21.15.116:0/723709420 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.785+0000 7fc85effd640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8601020a0 0x7fc86019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85ffff640 1 -- 172.21.15.116:0/723709420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8601020a0 msgr2=0x7fc86019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85ffff640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8601020a0 0x7fc86019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85ffff640 1 -- 172.21.15.116:0/723709420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601016e0 msgr2=0x7fc86019d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:12.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85ffff640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601016e0 0x7fc86019d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85ffff640 1 -- 172.21.15.116:0/723709420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc858009900 con 0x7fc8601076f0 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85f7fe640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601016e0 0x7fc86019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85ffff640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 0x7fc8601979d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc858004af0 tx=0x7fc858032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580325d0 con 0x7fc8601076f0 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc858003970 con 0x7fc8601076f0 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8601985d0 con 0x7fc8601076f0 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580046b0 con 0x7fc8601076f0 2024-03-24T09:22:12.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.786+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8601a48c0 con 0x7fc8601076f0 2024-03-24T09:22:12.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.787+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc86004f470 con 0x7fc8601076f0 2024-03-24T09:22:12.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.789+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fc85803b040 con 0x7fc8601076f0 2024-03-24T09:22:12.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.789+0000 7fc85cff9640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc8480a2370 0x7fc8480a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:12.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.789+0000 7fc85f7fe640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc8480a2370 0x7fc8480a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:12.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.790+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc8580f1230 con 0x7fc8601076f0 2024-03-24T09:22:12.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.790+0000 7fc85f7fe640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc8480a2370 0x7fc8480a4830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc860198b10 tx=0x7fc850005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:12.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.793+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc8580b9b40 con 0x7fc8601076f0 2024-03-24T09:22:12.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:12.991+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fc860111730 con 0x7fc8480a2370 2024-03-24T09:22:13.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.044+0000 7fc85cff9640 1 -- 172.21.15.116:0/723709420 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fc860111730 con 0x7fc8480a2370 2024-03-24T09:22:13.045 INFO:teuthology.orchestra.run.smithi116.stdout:Retention added to path / 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc8480a2370 msgr2=0x7fc8480a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc8480a2370 0x7fc8480a4830 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc860198b10 tx=0x7fc850005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 msgr2=0x7fc8601979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 0x7fc8601979d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc858004af0 tx=0x7fc858032f30 comp rx=0 tx=0).stop 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 shutdown_connections 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8601076f0 0x7fc8601979d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc8480a2370 0x7fc8480a4830 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8601020a0 0x7fc86019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 --2- 172.21.15.116:0/723709420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8601016e0 0x7fc86019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 >> 172.21.15.116:0/723709420 conn(0x7fc8600fd1f0 msgr2=0x7fc8600fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 shutdown_connections 2024-03-24T09:22:13.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.046+0000 7fc865db6640 1 -- 172.21.15.116:0/723709420 wait complete. 2024-03-24T09:22:13.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T09:22:13.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 -- 172.21.15.116:0/669686731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 msgr2=0x7f7c100691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 --2- 172.21.15.116:0/669686731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c100691f0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009930 tx=0x7f7c0002bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 -- 172.21.15.116:0/669686731 shutdown_connections 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 --2- 172.21.15.116:0/669686731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10069730 0x7f7c1010e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 --2- 172.21.15.116:0/669686731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c100691f0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 --2- 172.21.15.116:0/669686731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c10107780 0x7f7c10107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 -- 172.21.15.116:0/669686731 >> 172.21.15.116:0/669686731 conn(0x7f7c10077460 msgr2=0x7f7c10077870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 -- 172.21.15.116:0/669686731 shutdown_connections 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 -- 172.21.15.116:0/669686731 wait complete. 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 Processor -- start 2024-03-24T09:22:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.161+0000 7f7c18380640 1 -- start start 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c10100930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10069730 0x7f7c10100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c10107780 0x7f7c101013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c101a5f50 con 0x7f7c10068d90 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7c101a60c0 con 0x7f7c10107780 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7c101a6280 con 0x7f7c10069730 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c10100930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c10100930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56186/0 (socket says 172.21.15.116:56186) 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 -- 172.21.15.116:0/1461324876 learned_addr learned my addr 172.21.15.116:0/1461324876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1691d640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c10107780 0x7f7c101013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1591b640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10069730 0x7f7c10100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 -- 172.21.15.116:0/1461324876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10069730 msgr2=0x7f7c10100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10069730 0x7f7c10100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 -- 172.21.15.116:0/1461324876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c10107780 msgr2=0x7f7c101013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c10107780 0x7f7c101013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 -- 172.21.15.116:0/1461324876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c00009560 con 0x7f7c10068d90 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c1611c640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c10100930 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00e9f0 tx=0x7f7c0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c009700 con 0x7f7c10068d90 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c101a6560 con 0x7f7c10068d90 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c0c00cd80 con 0x7f7c10068d90 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c018840 con 0x7f7c10068d90 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.162+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c101a6a90 con 0x7f7c10068d90 2024-03-24T09:22:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.163+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c1004f470 con 0x7f7c10068d90 2024-03-24T09:22:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.164+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f7c0c0189a0 con 0x7f7c10068d90 2024-03-24T09:22:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.165+0000 7f7c077fe640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7bfc0a2370 0x7f7bfc0a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.165+0000 7f7c1591b640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7bfc0a2370 0x7f7bfc0a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.165+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7c0c014070 con 0x7f7c10068d90 2024-03-24T09:22:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.165+0000 7f7c1591b640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7bfc0a2370 0x7f7bfc0a4830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c00005ab0 tx=0x7f7c00005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.169+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c0c053280 con 0x7f7c10068d90 2024-03-24T09:22:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:13 smithi116 ceph-mon[32207]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T09:22:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2309051213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:13 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:22:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:13 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:13 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:22:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:13 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:13.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.365+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f7c10111700 con 0x7f7bfc0a2370 2024-03-24T09:22:13.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.373+0000 7f7c077fe640 1 -- 172.21.15.116:0/1461324876 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7f7c10111700 con 0x7f7bfc0a2370 2024-03-24T09:22:13.373 INFO:teuthology.orchestra.run.smithi116.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T09:22:12", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-24T09:22:13.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7bfc0a2370 msgr2=0x7f7bfc0a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7bfc0a2370 0x7f7bfc0a4830 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7c00005ab0 tx=0x7f7c00005fd0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 msgr2=0x7f7c10100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c10100930 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c00e9f0 tx=0x7f7c0c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 shutdown_connections 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c10107780 0x7f7c101013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7bfc0a2370 0x7f7bfc0a4830 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10069730 0x7f7c10100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 --2- 172.21.15.116:0/1461324876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c10068d90 0x7f7c10100930 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 >> 172.21.15.116:0/1461324876 conn(0x7f7c10077460 msgr2=0x7f7c100ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 shutdown_connections 2024-03-24T09:22:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.375+0000 7f7c18380640 1 -- 172.21.15.116:0/1461324876 wait complete. 2024-03-24T09:22:13.387 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T09:22:13.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:13 smithi143 ceph-mon[35965]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T09:22:13.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2309051213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:13.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:13 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:22:13.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:13 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:13.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:13 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:22:13.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:13 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 -- 172.21.15.116:0/1189209752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c341050a0 msgr2=0x7f4c3410cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/1189209752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c341050a0 0x7f4c3410cc70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00aa10 tx=0x7f4c2c02cf70 comp rx=0 tx=0).stop 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 -- 172.21.15.116:0/1189209752 shutdown_connections 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/1189209752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c341050a0 0x7f4c3410cc70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/1189209752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c34100980 0x7f4c341049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/1189209752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c340fffd0 0x7f4c341003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 -- 172.21.15.116:0/1189209752 >> 172.21.15.116:0/1189209752 conn(0x7f4c34076ba0 msgr2=0x7f4c340fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 -- 172.21.15.116:0/1189209752 shutdown_connections 2024-03-24T09:22:13.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 -- 172.21.15.116:0/1189209752 wait complete. 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.490+0000 7f4c3b71e640 1 Processor -- start 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 -- start start 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c340fffd0 0x7f4c34197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 0x7f4c341982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c3419c600 0x7f4c3419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3419cff0 con 0x7f4c341050a0 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3419d160 con 0x7f4c340fffd0 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4c3419d2d0 con 0x7f4c3419c600 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 0x7f4c341982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 0x7f4c341982d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56208/0 (socket says 172.21.15.116:56208) 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 -- 172.21.15.116:0/3281780876 learned_addr learned my addr 172.21.15.116:0/3281780876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c394ba640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c340fffd0 0x7f4c34197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c39cbb640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c3419c600 0x7f4c3419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 -- 172.21.15.116:0/3281780876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c3419c600 msgr2=0x7f4c3419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c3419c600 0x7f4c3419cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 -- 172.21.15.116:0/3281780876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c340fffd0 msgr2=0x7f4c34197d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c340fffd0 0x7f4c34197d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 -- 172.21.15.116:0/3281780876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c30009840 con 0x7f4c341050a0 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c394ba640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c340fffd0 0x7f4c34197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:13.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c38cb9640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 0x7f4c341982d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400e9c0 tx=0x7f4c2400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24009700 con 0x7f4c341050a0 2024-03-24T09:22:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2c00a670 con 0x7f4c341050a0 2024-03-24T09:22:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4c2400cd80 con 0x7f4c341050a0 2024-03-24T09:22:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.491+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c24010430 con 0x7f4c341050a0 2024-03-24T09:22:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.492+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c341a4bd0 con 0x7f4c341050a0 2024-03-24T09:22:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.493+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c3404f470 con 0x7f4c341050a0 2024-03-24T09:22:13.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.493+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f4c24004f50 con 0x7f4c341050a0 2024-03-24T09:22:13.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.494+0000 7f4c2a7fc640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4c200a2370 0x7f4c200a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:13.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.494+0000 7f4c394ba640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4c200a2370 0x7f4c200a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:13.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.494+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4c24014070 con 0x7f4c341050a0 2024-03-24T09:22:13.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.494+0000 7f4c394ba640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4c200a2370 0x7f4c200a4830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009fd0 tx=0x7f4c30009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:13.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.498+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4c24097070 con 0x7f4c341050a0 2024-03-24T09:22:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:13 smithi117 ceph-mon[35788]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T09:22:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2309051213' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:22:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:13 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:22:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:13 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:13 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:22:13.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:13 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.693+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f4c3410a3a0 con 0x7f4c200a2370 2024-03-24T09:22:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.699+0000 7f4c2a7fc640 1 -- 172.21.15.116:0/3281780876 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f4c3410a3a0 con 0x7f4c200a2370 2024-03-24T09:22:13.700 INFO:teuthology.orchestra.run.smithi116.stdout:/ 1m 6m3h 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4c200a2370 msgr2=0x7f4c200a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4c200a2370 0x7f4c200a4830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4c30009fd0 tx=0x7f4c30009500 comp rx=0 tx=0).stop 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 msgr2=0x7f4c341982d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 0x7f4c341982d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4c2400e9c0 tx=0x7f4c2400ee80 comp rx=0 tx=0).stop 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 shutdown_connections 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4c200a2370 0x7f4c200a4830 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4c3419c600 0x7f4c3419cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c341050a0 0x7f4c341982d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 --2- 172.21.15.116:0/3281780876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c340fffd0 0x7f4c34197d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:13.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 >> 172.21.15.116:0/3281780876 conn(0x7f4c34076ba0 msgr2=0x7f4c34103630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:13.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 shutdown_connections 2024-03-24T09:22:13.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:13.701+0000 7f4c3b71e640 1 -- 172.21.15.116:0/3281780876 wait complete. 2024-03-24T09:22:13.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-24T09:22:13.741 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-24T09:22:13.753 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T09:22:13.800 INFO:tasks.workunit:Pulling workunits from ref d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T09:22:13.801 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T09:22:13.801 INFO:tasks.workunit:timeout=3h 2024-03-24T09:22:13.801 INFO:tasks.workunit:cleanup=True 2024-03-24T09:22:13.801 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:13.816 INFO:teuthology.orchestra.run.smithi116.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:13.816 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 77h/119d Inode: 1 Links: 2 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:fusefs_t:s0 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-24 09:20:05.590216212 +0000 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-24 09:20:05.590216212 +0000 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-24 09:22:04.711936164 +0000 2024-03-24T09:22:13.817 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-03-24T09:22:13.817 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T09:22:13.817 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T09:22:13.890 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b 2024-03-24T09:22:13.945 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T09:22:14.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:14 smithi143 ceph-mon[35965]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:14.437 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:14 smithi143 ceph-mon[35965]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:14 smithi117 ceph-mon[35788]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:14 smithi117 ceph-mon[35788]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:14.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:14 smithi116 ceph-mon[32207]: from='client.15750 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:14.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:14 smithi116 ceph-mon[32207]: from='client.15789 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:15 smithi143 ceph-mon[35965]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T09:22:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:15 smithi143 ceph-mon[35965]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:15 smithi143 ceph-mon[35965]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:15.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:15 smithi117 ceph-mon[35788]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T09:22:15.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:15 smithi117 ceph-mon[35788]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:15.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:15 smithi117 ceph-mon[35788]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:15.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:15 smithi116 ceph-mon[32207]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-24T09:22:15.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:15 smithi116 ceph-mon[32207]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:15.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:15 smithi116 ceph-mon[32207]: from='client.15801 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:22:17.367 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:17 smithi117 ceph-mon[35788]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-24T09:22:17.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:17 smithi143 ceph-mon[35965]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-24T09:22:17.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:17 smithi116 ceph-mon[32207]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-24T09:22:19.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:19 smithi143 ceph-mon[35965]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-24T09:22:19.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:19 smithi117 ceph-mon[35788]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-24T09:22:19.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:19 smithi116 ceph-mon[32207]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-24T09:22:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:21 smithi143 ceph-mon[35965]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-24T09:22:21.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:21 smithi117 ceph-mon[35788]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-24T09:22:21.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:21 smithi116 ceph-mon[32207]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-24T09:22:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:23 smithi143 ceph-mon[35965]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-24T09:22:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:23 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:22:23.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:23 smithi117 ceph-mon[35788]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-24T09:22:23.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:23 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:22:23.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:23 smithi116 ceph-mon[32207]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-03-24T09:22:23.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:23 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:22:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:25 smithi143 ceph-mon[35965]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-03-24T09:22:25.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:25 smithi117 ceph-mon[35788]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-03-24T09:22:25.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:25 smithi116 ceph-mon[32207]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 77 KiB/s wr, 54 op/s 2024-03-24T09:22:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:27 smithi143 ceph-mon[35965]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T09:22:27.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:27 smithi117 ceph-mon[35788]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T09:22:27.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:27 smithi116 ceph-mon[32207]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 40 op/s 2024-03-24T09:22:29.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:29 smithi116 ceph-mon[32207]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-24T09:22:29.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:29 smithi143 ceph-mon[35965]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-24T09:22:29.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:29 smithi117 ceph-mon[35788]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 77 KiB/s wr, 44 op/s 2024-03-24T09:22:31.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:31 smithi143 ceph-mon[35965]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:22:31.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:31 smithi117 ceph-mon[35788]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:22:31.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:31 smithi116 ceph-mon[32207]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:22:33.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:33 smithi143 ceph-mon[35965]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-24T09:22:33.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:33 smithi117 ceph-mon[35788]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-24T09:22:33.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:33 smithi116 ceph-mon[32207]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-24T09:22:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:35 smithi143 ceph-mon[35965]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-24T09:22:35.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:35 smithi117 ceph-mon[35788]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-24T09:22:35.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:35 smithi116 ceph-mon[32207]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-24T09:22:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:37 smithi143 ceph-mon[35965]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-24T09:22:37.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:37 smithi117 ceph-mon[35788]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-24T09:22:37.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:37 smithi116 ceph-mon[32207]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-24T09:22:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:39 smithi143 ceph-mon[35965]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-24T09:22:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:39 smithi117 ceph-mon[35788]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-24T09:22:39.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:39 smithi116 ceph-mon[32207]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-24T09:22:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:41 smithi143 ceph-mon[35965]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:22:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:41 smithi117 ceph-mon[35788]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:22:41.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:41 smithi116 ceph-mon[32207]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:22:42.545 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:22:42.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:22:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 -- 172.21.15.116:0/831904688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 msgr2=0x7fa95806a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/831904688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95806a100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa948009900 tx=0x7fa94802bc30 comp rx=0 tx=0).stop 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 -- 172.21.15.116:0/831904688 shutdown_connections 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/831904688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 0x7fa95810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/831904688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95806a100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/831904688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 0x7fa958107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 -- 172.21.15.116:0/831904688 >> 172.21.15.116:0/831904688 conn(0x7fa9580775f0 msgr2=0x7fa958077a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 -- 172.21.15.116:0/831904688 shutdown_connections 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.660+0000 7fa95fa6b640 1 -- 172.21.15.116:0/831904688 wait complete. 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 Processor -- start 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 -- start start 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 0x7fa95819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 0x7fa958197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa9581980e0 con 0x7fa9581076b0 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa958198250 con 0x7fa958069c80 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.661+0000 7fa95fa6b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa958198410 con 0x7fa95806a640 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95e008640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 0x7fa958197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95e008640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 0x7fa958197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40232/0 (socket says 172.21.15.116:40232) 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95e008640 1 -- 172.21.15.116:0/549209014 learned_addr learned my addr 172.21.15.116:0/549209014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d006640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 0x7fa95819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 -- 172.21.15.116:0/549209014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 msgr2=0x7fa95819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 0x7fa95819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 -- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 msgr2=0x7fa958197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 0x7fa958197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 -- 172.21.15.116:0/549209014 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa950009840 con 0x7fa958069c80 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d006640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 0x7fa95819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95d807640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95819d8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa95400e9f0 tx=0x7fa95400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa94effd640 1 -- 172.21.15.116:0/549209014 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95400ccc0 con 0x7fa958069c80 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa948009560 con 0x7fa958069c80 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa94effd640 1 -- 172.21.15.116:0/549209014 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa95400ce20 con 0x7fa958069c80 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa94effd640 1 -- 172.21.15.116:0/549209014 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540187a0 con 0x7fa958069c80 2024-03-24T09:22:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.662+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9581a47a0 con 0x7fa958069c80 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.664+0000 7fa94effd640 1 -- 172.21.15.116:0/549209014 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fa954010430 con 0x7fa958069c80 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.665+0000 7fa94effd640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa9440a2340 0x7fa9440a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.665+0000 7fa95d006640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa9440a2340 0x7fa9440a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.665+0000 7fa94effd640 1 -- 172.21.15.116:0/549209014 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa954014070 con 0x7fa958069c80 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa95d006640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa9440a2340 0x7fa9440a4800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa948006010 tx=0x7fa94802c6c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 Processor -- start 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 -- start start 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99856020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99856a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd9986a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd9977af30 con 0x55dd99867e00 2024-03-24T09:22:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dd997c4810 con 0x55dd99855c40 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa96264a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dd997c1d40 con 0x55dd998565f0 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa95e008640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd9986a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa95d807640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99856020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa95e008640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd9986a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40240/0 (socket says 172.21.15.116:40240) 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa95e008640 1 -- 172.21.15.116:0/4216436556 learned_addr learned my addr 172.21.15.116:0/4216436556 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.666+0000 7fa95d006640 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99856a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d807640 1 -- 172.21.15.116:0/4216436556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 msgr2=0x55dd99856a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d807640 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99856a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d807640 1 -- 172.21.15.116:0/4216436556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 msgr2=0x55dd9986a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d807640 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd9986a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d807640 1 -- 172.21.15.116:0/4216436556 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd9977eea0 con 0x55dd99855c40 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d006640 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99856a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa95d807640 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99856020 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9540cbb40 tx=0x7fa95409a9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dcfff506dc7552f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa943fff640 1 -- 172.21.15.116:0/4216436556 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540a34d0 con 0x55dd99855c40 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa943fff640 1 -- 172.21.15.116:0/4216436556 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa954010b00 con 0x55dd99855c40 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.667+0000 7fa943fff640 1 -- 172.21.15.116:0/4216436556 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540a32a0 con 0x55dd99855c40 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 -- 172.21.15.116:0/4216436556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 msgr2=0x55dd99856020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99856020 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9540cbb40 tx=0x7fa95409a9e0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 -- 172.21.15.116:0/4216436556 shutdown_connections 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd9986a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99856a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 --2- 172.21.15.116:0/4216436556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99856020 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 -- 172.21.15.116:0/4216436556 >> 172.21.15.116:0/4216436556 conn(0x55dd99850900 msgr2=0x55dd99850ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 -- 172.21.15.116:0/4216436556 shutdown_connections 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 -- 172.21.15.116:0/4216436556 wait complete. 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 -- start start 2024-03-24T09:22:42.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.668+0000 7fa96264a740 1 Processor -- start 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99861620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99861b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd99867340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd997bd520 con 0x55dd99867e00 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dd99490430 con 0x55dd99855c40 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dd9944ce00 con 0x55dd998565f0 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99861620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95e008640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd99867340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99861620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58156/0 (socket says 172.21.15.116:58156) 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 -- 172.21.15.116:0/2982586292 learned_addr learned my addr 172.21.15.116:0/2982586292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d006640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99861b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 10 client.0 _ll_register_callbacks cb 0x55dd9975bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 -- 172.21.15.116:0/2982586292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 msgr2=0x55dd99861b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99861b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 -- 172.21.15.116:0/2982586292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 msgr2=0x55dd99867340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd99867340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95d807640 1 -- 172.21.15.116:0/2982586292 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9581a47a0 con 0x55dd99855c40 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dd994bc7e0 con 0x55dd99855c40 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.669+0000 7fa95e008640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd99867340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa95d807640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99861620 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa9540977b0 tx=0x7fa9540a32f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bce46408793812e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa9417fa640 10 client.25649 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa9417fa640 1 -- 172.21.15.116:0/2982586292 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540b92c0 con 0x55dd99855c40 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa9417fa640 1 -- 172.21.15.116:0/2982586292 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa9540ce6a0 con 0x55dd99855c40 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa9417fa640 1 -- 172.21.15.116:0/2982586292 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9540a34d0 con 0x55dd99855c40 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 10 client.25649 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55dd997bbba0 con 0x55dd99855c40 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa9417fa640 1 -- 172.21.15.116:0/2982586292 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa95409a230 con 0x55dd99855c40 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 10 client.25649 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 10 client.25649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 20 client.25649 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd99880700 0x55dd99882b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 4 client.25649 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd994f9720 con 0x55dd99880700 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.670+0000 7fa95d006640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd99880700 0x55dd99882b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.671+0000 7fa9417fa640 10 client.25649 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.671+0000 7fa95d006640 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd99880700 0x55dd99882b70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39ebb096d97f0f5c server_cookie=f8d403f5eb53d87b in_seq=0 out_seq=0 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.671+0000 7fa9417fa640 1 -- 172.21.15.116:0/2982586292 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd994f9720 con 0x55dd99880700 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.671+0000 7fa9417fa640 10 client.25649 handle_command_reply: tid=0 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.671+0000 7fa96264a740 1 client.25649 shutdown 2024-03-24T09:22:42.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 msgr2=0x55dd99861620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99861620 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa9540977b0 tx=0x7fa9540a32f0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 shutdown_connections 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99867e00 0x55dd99867340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd998565f0 0x55dd99861b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd99880700 0x55dd99882b70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 --2- 172.21.15.116:0/2982586292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd99855c40 0x55dd99861620 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 >> 172.21.15.116:0/2982586292 conn(0x55dd99850900 msgr2=0x55dd99852650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa9417fa640 0 client.25649 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 shutdown_connections 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 1 -- 172.21.15.116:0/2982586292 wait complete. 2024-03-24T09:22:42.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.672+0000 7fa96264a740 20 client.25649 trim_cache size 0 max 16384 2024-03-24T09:22:42.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 Processor -- start 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 -- start start 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 0x55dd99850ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998565f0 0x55dd99888410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd998937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd99742430 con 0x55dd99855c40 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dd99851b30 con 0x55dd998565f0 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa96264a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dd9985bd50 con 0x55dd99867e00 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.689+0000 7fa95d807640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 0x55dd99850ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 0x55dd99850ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40256/0 (socket says 172.21.15.116:40256) 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 -- 172.21.15.116:0/1352783305 learned_addr learned my addr 172.21.15.116:0/1352783305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d006640 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998565f0 0x55dd99888410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95e008640 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd998937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 -- 172.21.15.116:0/1352783305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 msgr2=0x55dd998937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd998937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 -- 172.21.15.116:0/1352783305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998565f0 msgr2=0x55dd99888410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998565f0 0x55dd99888410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 -- 172.21.15.116:0/1352783305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd99757b20 con 0x55dd99855c40 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d006640 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998565f0 0x55dd99888410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa95d807640 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 0x55dd99850ce0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa9540cbcb0 tx=0x7fa954022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f47e2367389fb0d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa9437fe640 1 -- 172.21.15.116:0/1352783305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540a32c0 con 0x55dd99855c40 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa9437fe640 1 -- 172.21.15.116:0/1352783305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dd994bc7e0 con 0x55dd99855c40 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.690+0000 7fa9437fe640 1 -- 172.21.15.116:0/1352783305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540ce430 con 0x55dd99855c40 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 -- 172.21.15.116:0/1352783305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 msgr2=0x55dd99850ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 0x55dd99850ce0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa9540cbcb0 tx=0x7fa954022000 comp rx=0 tx=0).stop 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 -- 172.21.15.116:0/1352783305 shutdown_connections 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd998937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998565f0 0x55dd99888410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 --2- 172.21.15.116:0/1352783305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd99855c40 0x55dd99850ce0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 -- 172.21.15.116:0/1352783305 >> 172.21.15.116:0/1352783305 conn(0x55dd99850900 msgr2=0x55dd9984fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 -- 172.21.15.116:0/1352783305 shutdown_connections 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 -- 172.21.15.116:0/1352783305 wait complete. 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 -- start start 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.691+0000 7fa96264a740 1 Processor -- start 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 0x55dd99890980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd99890ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998e4400 0x55dd998e67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dd998e6d30 con 0x55dd998565f0 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dd998e6e70 con 0x55dd998e4400 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 0x55dd99890980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dd998e6fb0 con 0x55dd99867e00 2024-03-24T09:22:42.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 0x55dd99890980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40268/0 (socket says 172.21.15.116:40268) 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 -- 172.21.15.116:0/454660344 learned_addr learned my addr 172.21.15.116:0/454660344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa96264a740 10 client.0 _ll_register_callbacks cb 0x7fa950009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95e008640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998e4400 0x55dd998e67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d006640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd99890ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 -- 172.21.15.116:0/454660344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 msgr2=0x55dd99890ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd99890ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 -- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998e4400 msgr2=0x55dd998e67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998e4400 0x55dd998e67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.692+0000 7fa95d807640 1 -- 172.21.15.116:0/454660344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa948030070 con 0x55dd998565f0 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa95d807640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 0x55dd99890980 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa954010ab0 tx=0x7fa95409a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10734006bb1427e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa9417fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:22:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa9417fa640 1 -- 172.21.15.116:0/454660344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540a32c0 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa9417fa640 1 -- 172.21.15.116:0/454660344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa95409ee80 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa954010b00 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa9417fa640 1 -- 172.21.15.116:0/454660344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9540a34d0 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 10 client.15819 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55dd997bbba0 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa9417fa640 1 -- 172.21.15.116:0/454660344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa954010b00 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa9417fa640 1 -- 172.21.15.116:0/454660344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa95409fe30 con 0x55dd998565f0 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 10 client.15819 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 20 client.15819 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd998e83c0 0x55dd998ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 4 client.15819 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:22:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.693+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd99548890 con 0x55dd998e83c0 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa95d006640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd998e83c0 0x55dd998ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa9417fa640 10 client.15819 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa95d006640 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd998e83c0 0x55dd998ea7b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5bb853c1d6592a8 server_cookie=37963a81f19e49 in_seq=0 out_seq=0 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa9417fa640 1 -- 172.21.15.116:0/454660344 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55dd99548890 con 0x55dd998e83c0 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa9417fa640 10 client.15819 handle_command_reply: tid=0 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 client.15819 shutdown 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 msgr2=0x55dd99890980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 0x55dd99890980 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa954010ab0 tx=0x7fa95409a460 comp rx=0 tx=0).stop 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 shutdown_connections 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55dd998e83c0 0x55dd998ea7b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dd998e4400 0x55dd998e67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dd99867e00 0x55dd99890ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 --2- 172.21.15.116:0/454660344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dd998565f0 0x55dd99890980 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa9417fa640 0 client.15819 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 >> 172.21.15.116:0/454660344 conn(0x55dd99850900 msgr2=0x55dd9984f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 shutdown_connections 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.694+0000 7fa96264a740 1 -- 172.21.15.116:0/454660344 wait complete. 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa96264a740 20 client.15819 trim_cache size 0 max 16384 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:22:42.702 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:22:42.703 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:22:42.703 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:22:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa9440a2340 msgr2=0x7fa9440a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa9440a2340 0x7fa9440a4800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa948006010 tx=0x7fa94802c6c0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 msgr2=0x7fa95819d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95819d8e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa95400e9f0 tx=0x7fa95400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 shutdown_connections 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9581076b0 0x7fa958197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa9440a2340 0x7fa9440a4800 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95806a640 0x7fa95819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 --2- 172.21.15.116:0/549209014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa958069c80 0x7fa95819d8e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 >> 172.21.15.116:0/549209014 conn(0x7fa9580775f0 msgr2=0x7fa9580ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 shutdown_connections 2024-03-24T09:22:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.695+0000 7fa95fa6b640 1 -- 172.21.15.116:0/549209014 wait complete. 2024-03-24T09:22:42.706 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:22:42.707 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:22:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.813+0000 7fe8af30c640 1 -- 172.21.15.116:0/3498310169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 msgr2=0x7fe8a8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.813+0000 7fe8af30c640 1 --2- 172.21.15.116:0/3498310169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a8106a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4009900 tx=0x7fe8a402bcd0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 -- 172.21.15.116:0/3498310169 shutdown_connections 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 --2- 172.21.15.116:0/3498310169 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8a8100d40 0x7fe8a810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 --2- 172.21.15.116:0/3498310169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a8106fd0 0x7fe8a8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 --2- 172.21.15.116:0/3498310169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a8106a90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 -- 172.21.15.116:0/3498310169 >> 172.21.15.116:0/3498310169 conn(0x7fe8a8077710 msgr2=0x7fe8a8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 -- 172.21.15.116:0/3498310169 shutdown_connections 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 -- 172.21.15.116:0/3498310169 wait complete. 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 Processor -- start 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 -- start start 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8a8100d40 0x7fe8a819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.814+0000 7fe8af30c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8af30c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a8106fd0 0x7fe8a8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58212/0 (socket says 172.21.15.116:58212) 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 -- 172.21.15.116:0/1829171195 learned_addr learned my addr 172.21.15.116:0/1829171195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ad0a8640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8a8100d40 0x7fe8a819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ad8a9640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a8106fd0 0x7fe8a8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a81980f0 con 0x7fe8a8106fd0 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a8198260 con 0x7fe8a81066b0 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a8198420 con 0x7fe8a8100d40 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 -- 172.21.15.116:0/1829171195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8a8100d40 msgr2=0x7fe8a819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8a8100d40 0x7fe8a819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 -- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a8106fd0 msgr2=0x7fe8a8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a8106fd0 0x7fe8a8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a4009560 con 0x7fe8a81066b0 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8ac8a7640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a819de30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e9f0 tx=0x7fe89800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8980096a0 con 0x7fe8a81066b0 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a8198700 con 0x7fe8a81066b0 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.815+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe89800ccc0 con 0x7fe8a81066b0 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.816+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a81a4990 con 0x7fe8a81066b0 2024-03-24T09:22:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.816+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8980187e0 con 0x7fe8a81066b0 2024-03-24T09:22:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.816+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a8101d50 con 0x7fe8a81066b0 2024-03-24T09:22:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.817+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fe898018940 con 0x7fe8a81066b0 2024-03-24T09:22:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.818+0000 7fe89e7fc640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8940a21a0 0x7fe8940a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.818+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe898014070 con 0x7fe8a81066b0 2024-03-24T09:22:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.818+0000 7fe8ad0a8640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8940a21a0 0x7fe8940a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.819+0000 7fe8ad0a8640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8940a21a0 0x7fe8940a4660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8198be0 tx=0x7fe8a4005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:42.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:42.822+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe898097080 con 0x7fe8a81066b0 2024-03-24T09:22:43.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.044+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8a8198fd0 con 0x7fe8a81066b0 2024-03-24T09:22:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.044+0000 7fe89e7fc640 1 -- 172.21.15.116:0/1829171195 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11953 (secure 0 0 0) 0x7fe89809a740 con 0x7fe8a81066b0 2024-03-24T09:22:43.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:22:43.045 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:22:05.111617+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24422,24443,24451,24484,24515]},"id":1}]} 2024-03-24T09:22:43.045 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 29 2024-03-24T09:22:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8940a21a0 msgr2=0x7fe8940a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8940a21a0 0x7fe8940a4660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8198be0 tx=0x7fe8a4005fd0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 msgr2=0x7fe8a819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a819de30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe89800e9f0 tx=0x7fe89800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 shutdown_connections 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8a8106fd0 0x7fe8a8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a81066b0 0x7fe8a819de30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8940a21a0 0x7fe8940a4660 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 --2- 172.21.15.116:0/1829171195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe8a8100d40 0x7fe8a819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.046+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 >> 172.21.15.116:0/1829171195 conn(0x7fe8a8077710 msgr2=0x7fe8a80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.047+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 shutdown_connections 2024-03-24T09:22:43.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.047+0000 7fe8af30c640 1 -- 172.21.15.116:0/1829171195 wait complete. 2024-03-24T09:22:43.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:22:43.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 -- 172.21.15.116:0/3328818104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3010e080 msgr2=0x7f1c30110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 --2- 172.21.15.116:0/3328818104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3010e080 0x7f1c30110470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009ca0 tx=0x7f1c2802c120 comp rx=0 tx=0).stop 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 -- 172.21.15.116:0/3328818104 shutdown_connections 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 --2- 172.21.15.116:0/3328818104 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c3010e080 0x7f1c30110470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 --2- 172.21.15.116:0/3328818104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c30100df0 0x7f1c3010db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 --2- 172.21.15.116:0/3328818104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 0x7f1c301008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 -- 172.21.15.116:0/3328818104 >> 172.21.15.116:0/3328818104 conn(0x7f1c30076990 msgr2=0x7f1c30076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 -- 172.21.15.116:0/3328818104 shutdown_connections 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.166+0000 7f1c366bb640 1 -- 172.21.15.116:0/3328818104 wait complete. 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 Processor -- start 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 -- start start 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 0x7f1c30199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c30100df0 0x7f1c3019a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c3010e080 0x7f1c3019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3019ed00 con 0x7f1c3010e080 2024-03-24T09:22:43.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c34c58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c3010e080 0x7f1c3019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 0x7f1c30199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 0x7f1c30199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58230/0 (socket says 172.21.15.116:58230) 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c34c58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c3010e080 0x7f1c3019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40290/0 (socket says 172.21.15.116:40290) 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1577892956 learned_addr learned my addr 172.21.15.116:0/1577892956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c30100df0 0x7f1c3019a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3019ee70 con 0x7f1c301004d0 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.167+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3019f030 con 0x7f1c30100df0 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2f7fe640 1 -- 172.21.15.116:0/1577892956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 msgr2=0x7f1c30199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 0x7f1c30199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2f7fe640 1 -- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c3010e080 msgr2=0x7f1c3019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c3010e080 0x7f1c3019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2f7fe640 1 -- 172.21.15.116:0/1577892956 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c24009490 con 0x7f1c30100df0 2024-03-24T09:22:43.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c30100df0 0x7f1c3019a3d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400e9f0 tx=0x7f1c2400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2d7fa640 1 -- 172.21.15.116:0/1577892956 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400cd80 con 0x7f1c30100df0 2024-03-24T09:22:43.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2d7fa640 1 -- 172.21.15.116:0/1577892956 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c240044d0 con 0x7f1c30100df0 2024-03-24T09:22:43.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c2d7fa640 1 -- 172.21.15.116:0/1577892956 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c24010430 con 0x7f1c30100df0 2024-03-24T09:22:43.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c28009900 con 0x7f1c30100df0 2024-03-24T09:22:43.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.168+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c301a69b0 con 0x7f1c30100df0 2024-03-24T09:22:43.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.170+0000 7f1c2d7fa640 1 -- 172.21.15.116:0/1577892956 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f1c24003fa0 con 0x7f1c30100df0 2024-03-24T09:22:43.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c2d7fa640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1c180a23c0 0x7f1c180a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 Processor -- start 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 -- start start 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1c180a23c0 0x7f1c180a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e06343f0 0x5566e06347d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e063d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566e05a2840 con 0x5566e0634d10 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566e059fd70 con 0x5566e063d8f0 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c39370740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5566e059b550 con 0x5566e06343f0 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c2f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e063d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58240/0 (socket says 172.21.15.116:58240) 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 -- 172.21.15.116:0/1845184981 learned_addr learned my addr 172.21.15.116:0/1845184981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 -- 172.21.15.116:0/1845184981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e06343f0 msgr2=0x5566e06347d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e06343f0 0x5566e06347d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 -- 172.21.15.116:0/1845184981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 msgr2=0x5566e063d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e063d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.171+0000 7f1c34c58640 1 -- 172.21.15.116:0/1845184981 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566e0530de0 con 0x5566e063d8f0 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.172+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1c180a23c0 0x7f1c180a4880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1c20017fd0 tx=0x7f1c20005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.172+0000 7f1c2d7fa640 1 -- 172.21.15.116:0/1577892956 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1c24014070 con 0x7f1c30100df0 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.172+0000 7f1c34c58640 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063fce0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1c28005cc0 tx=0x7f1c28005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=118790e370549fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.172+0000 7f1c167fc640 1 -- 172.21.15.116:0/1845184981 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28032440 con 0x5566e063d8f0 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.172+0000 7f1c167fc640 1 -- 172.21.15.116:0/1845184981 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c28032e90 con 0x5566e063d8f0 2024-03-24T09:22:43.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.172+0000 7f1c167fc640 1 -- 172.21.15.116:0/1845184981 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2803f480 con 0x5566e063d8f0 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 -- 172.21.15.116:0/1845184981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 msgr2=0x5566e063fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063fce0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1c28005cc0 tx=0x7f1c28005c50 comp rx=0 tx=0).stop 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 -- 172.21.15.116:0/1845184981 shutdown_connections 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063fce0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e063d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 --2- 172.21.15.116:0/1845184981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e06343f0 0x5566e06347d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 -- 172.21.15.116:0/1845184981 >> 172.21.15.116:0/1845184981 conn(0x5566e062eb20 msgr2=0x5566e062ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 -- 172.21.15.116:0/1845184981 shutdown_connections 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 -- 172.21.15.116:0/1845184981 wait complete. 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 -- start start 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 Processor -- start 2024-03-24T09:22:43.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 0x5566e063a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.173+0000 7f1c39370740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e063ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c39370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c39370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566e026e430 con 0x5566e06343f0 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c39370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566e022ae00 con 0x5566e063d8f0 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c39370740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5566e0520590 con 0x5566e0634d10 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 0x5566e063a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 0x5566e063a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40318/0 (socket says 172.21.15.116:40318) 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1168895393 learned_addr learned my addr 172.21.15.116:0/1168895393 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c39370740 10 client.0 _ll_register_callbacks cb 0x5566e058ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e063ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c34c58640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1168895393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 msgr2=0x5566e063ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e063ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 msgr2=0x5566e063b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1168895393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c30111000 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 0x5566e063a920 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000ef10 tx=0x7f1c200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127d4f39f7c3e962 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1168895393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20028070 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5566e029a7e0 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.174+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1168895393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1c200238e0 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1168895393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1c2001c070 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 10 client.15837 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566e0599bd0 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1168895393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5566e029a7e0 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1168895393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1c20021ba0 con 0x5566e06343f0 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 10 client.15837 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 10 client.15837 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:22:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 20 client.15837 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e064d980 0x5566e064fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 4 client.15837 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566e02d7720 con 0x5566e064d980 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.175+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e064d980 0x5566e064fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.176+0000 7f1c0f7fe640 10 client.15837 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.176+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e064d980 0x5566e064fdb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65457bfdd48a3fe6 server_cookie=df555f873e93693e in_seq=0 out_seq=0 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1168895393 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5566e02d7720 con 0x5566e064d980 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c0f7fe640 10 client.15837 handle_command_reply: tid=0 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 client.15837 shutdown 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 msgr2=0x5566e063a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 0x5566e063a920 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000ef10 tx=0x7f1c200046c0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 shutdown_connections 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e063d8f0 0x5566e063b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e063ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e064d980 0x5566e064fdb0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c0f7fe640 0 client.15837 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:22:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 --2- 172.21.15.116:0/1168895393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e06343f0 0x5566e063a920 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 >> 172.21.15.116:0/1168895393 conn(0x5566e062eb20 msgr2=0x5566e0630870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 shutdown_connections 2024-03-24T09:22:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 1 -- 172.21.15.116:0/1168895393 wait complete. 2024-03-24T09:22:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.177+0000 7f1c39370740 20 client.15837 trim_cache size 0 max 16384 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 Processor -- start 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 -- start start 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e06556c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e0645740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e063d8f0 0x5566e0630580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566e06334e0 con 0x5566e063d8f0 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566e064c1b0 con 0x5566e06343f0 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.190+0000 7f1c39370740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5566e063fce0 con 0x5566e0634d10 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e06556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e06556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58248/0 (socket says 172.21.15.116:58248) 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 -- 172.21.15.116:0/3928292244 learned_addr learned my addr 172.21.15.116:0/3928292244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c34c58640 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e063d8f0 0x5566e0630580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e0645740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 -- 172.21.15.116:0/3928292244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 msgr2=0x5566e0645740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e0645740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 -- 172.21.15.116:0/3928292244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e063d8f0 msgr2=0x5566e0630580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e063d8f0 0x5566e0630580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 -- 172.21.15.116:0/3928292244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566e0535c80 con 0x5566e06343f0 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e06556c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e780 tx=0x7f1c2000e8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12ff7166b4ded1ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c15ffb640 1 -- 172.21.15.116:0/3928292244 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20028040 con 0x5566e06343f0 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c15ffb640 1 -- 172.21.15.116:0/3928292244 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c20023d90 con 0x5566e06343f0 2024-03-24T09:22:43.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.191+0000 7f1c15ffb640 1 -- 172.21.15.116:0/3928292244 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20004990 con 0x5566e06343f0 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 -- 172.21.15.116:0/3928292244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 msgr2=0x5566e06556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e06556c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1c2000e780 tx=0x7f1c2000e8d0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 -- 172.21.15.116:0/3928292244 shutdown_connections 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e063d8f0 0x5566e0630580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e0634d10 0x5566e0645740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 --2- 172.21.15.116:0/3928292244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e06556c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 -- 172.21.15.116:0/3928292244 >> 172.21.15.116:0/3928292244 conn(0x5566e062eb20 msgr2=0x5566e062fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 -- 172.21.15.116:0/3928292244 shutdown_connections 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 -- 172.21.15.116:0/3928292244 wait complete. 2024-03-24T09:22:43.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 -- start start 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.192+0000 7f1c39370740 1 Processor -- start 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e0641860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e0641da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e063d8f0 0x5566e06accc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5566e06ad200 con 0x5566e0634d10 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566e06ad370 con 0x5566e06343f0 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5566e06ad530 con 0x5566e063d8f0 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e0641860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e0641860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58260/0 (socket says 172.21.15.116:58260) 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1552826897 learned_addr learned my addr 172.21.15.116:0/1552826897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e0641da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c39370740 10 client.0 _ll_register_callbacks cb 0x7f1c20009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1552826897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e063d8f0 msgr2=0x5566e06accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e063d8f0 0x5566e06accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1552826897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 msgr2=0x5566e0641da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e0641da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2ffff640 1 -- 172.21.15.116:0/1552826897 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566e0599bd0 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.193+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e0641da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c2ffff640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e0641860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1c200237f0 tx=0x7f1c20023b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97ea46da4bfcb208 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1552826897 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20028040 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c28032e90 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1552826897 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1c200217b0 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1552826897 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1c2000edd0 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 10 client.25679 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5566e055b920 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1552826897 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1c28032e90 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1552826897 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1c20036030 con 0x5566e06343f0 2024-03-24T09:22:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 10 client.25679 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 10 client.25679 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 20 client.25679 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e06b49c0 0x5566e06b6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 4 client.25679 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.194+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566e0326890 con 0x5566e06b49c0 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e06b49c0 0x5566e06b6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c2f7fe640 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e06b49c0 0x5566e06b6de0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bbbe8a98c60328ff server_cookie=91c0656295811705 in_seq=0 out_seq=0 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c0f7fe640 10 client.25679 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c0f7fe640 1 -- 172.21.15.116:0/1552826897 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566e0326890 con 0x5566e06b49c0 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c0f7fe640 10 client.25679 handle_command_reply: tid=0 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 client.25679 shutdown 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 msgr2=0x5566e0641860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e0641860 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1c200237f0 tx=0x7f1c20023b10 comp rx=0 tx=0).stop 2024-03-24T09:22:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 shutdown_connections 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5566e063d8f0 0x5566e06accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5566e06b49c0 0x5566e06b6de0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5566e0634d10 0x5566e0641da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c0f7fe640 0 client.25679 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 --2- 172.21.15.116:0/1552826897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566e06343f0 0x5566e0641860 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 >> 172.21.15.116:0/1552826897 conn(0x5566e062eb20 msgr2=0x5566e0630700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 shutdown_connections 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.195+0000 7f1c39370740 1 -- 172.21.15.116:0/1552826897 wait complete. 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c39370740 20 client.25679 trim_cache size 0 max 16384 2024-03-24T09:22:43.202 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:22:43.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1c180a23c0 msgr2=0x7f1c180a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1c180a23c0 0x7f1c180a4880 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1c20017fd0 tx=0x7f1c20005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:43.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c30100df0 msgr2=0x7f1c3019a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c30100df0 0x7f1c3019a3d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400e9f0 tx=0x7f1c2400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 shutdown_connections 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c3010e080 0x7f1c3019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1c180a23c0 0x7f1c180a4880 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c30100df0 0x7f1c3019a3d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 --2- 172.21.15.116:0/1577892956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c301004d0 0x7f1c30199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 >> 172.21.15.116:0/1577892956 conn(0x7f1c30076990 msgr2=0x7f1c3010f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 shutdown_connections 2024-03-24T09:22:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.196+0000 7f1c366bb640 1 -- 172.21.15.116:0/1577892956 wait complete. 2024-03-24T09:22:43.207 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 -- 172.21.15.116:0/2652026574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 msgr2=0x7f2f58068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/2652026574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f58068d90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f48009960 tx=0x7f2f4802bc20 comp rx=0 tx=0).stop 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 -- 172.21.15.116:0/2652026574 shutdown_connections 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/2652026574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/2652026574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f58068d90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/2652026574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f58107760 0x7f2f58107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 -- 172.21.15.116:0/2652026574 >> 172.21.15.116:0/2652026574 conn(0x7f2f58077710 msgr2=0x7f2f58077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 -- 172.21.15.116:0/2652026574 shutdown_connections 2024-03-24T09:22:43.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.314+0000 7f2f5fe23640 1 -- 172.21.15.116:0/2652026574 wait complete. 2024-03-24T09:22:43.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 Processor -- start 2024-03-24T09:22:43.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 -- start start 2024-03-24T09:22:43.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f5819d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5819de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f58107760 0x7f2f58197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58198120 con 0x7f2f580692d0 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58198290 con 0x7f2f58068910 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5fe23640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2f58198450 con 0x7f2f58107760 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5d3be640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5819de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5d3be640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5819de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40350/0 (socket says 172.21.15.116:40350) 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3399424304 learned_addr learned my addr 172.21.15.116:0/3399424304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f5819d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.315+0000 7f2f5e3c0640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f58107760 0x7f2f58197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3399424304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f58107760 msgr2=0x7f2f58197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f58107760 0x7f2f58197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3399424304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 msgr2=0x7f2f5819d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f5819d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3399424304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f48009560 con 0x7f2f580692d0 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f5819d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5819de60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2f4802c230 tx=0x7f2f48005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f4effd640 1 -- 172.21.15.116:0/3399424304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f480042d0 con 0x7f2f580692d0 2024-03-24T09:22:43.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f581986d0 con 0x7f2f580692d0 2024-03-24T09:22:43.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f4effd640 1 -- 172.21.15.116:0/3399424304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f48004430 con 0x7f2f580692d0 2024-03-24T09:22:43.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f4effd640 1 -- 172.21.15.116:0/3399424304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f48005310 con 0x7f2f580692d0 2024-03-24T09:22:43.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.316+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f581006e0 con 0x7f2f580692d0 2024-03-24T09:22:43.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:43 smithi116 ceph-mon[32207]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-24T09:22:43.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:43 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1829171195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:43.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.319+0000 7f2f62a00740 1 Processor -- start 2024-03-24T09:22:43.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.319+0000 7f2f62a00740 1 -- start start 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161220f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x5616122183b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x56161221ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f62a00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56161217d840 con 0x56161220fd10 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f62a00740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56161217ad70 con 0x56161220f3f0 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f62a00740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561612176550 con 0x5616122188f0 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5e3c0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161220f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x5616122183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x5616122183b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40352/0 (socket says 172.21.15.116:40352) 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1111544429 learned_addr learned my addr 172.21.15.116:0/1111544429 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5e3c0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161220f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58278/0 (socket says 172.21.15.116:58278) 2024-03-24T09:22:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1111544429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 msgr2=0x56161221ace0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x56161221ace0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1111544429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 msgr2=0x56161220f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f4effd640 1 -- 172.21.15.116:0/3399424304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f2f48005470 con 0x7f2f580692d0 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161220f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1111544429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56161210bde0 con 0x56161220fd10 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.320+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x5616122183b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009850 tx=0x7f2f54005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf742f66ebdace0f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.321+0000 7f2f43fff640 1 -- 172.21.15.116:0/1111544429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54015050 con 0x56161220fd10 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.321+0000 7f2f43fff640 1 -- 172.21.15.116:0/1111544429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f54016040 con 0x56161220fd10 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.321+0000 7f2f43fff640 1 -- 172.21.15.116:0/1111544429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5401ad20 con 0x56161220fd10 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.321+0000 7f2f4effd640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2f440a2340 0x7f2f440a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.321+0000 7f2f5e3c0640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2f440a2340 0x7f2f440a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 -- 172.21.15.116:0/1111544429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 msgr2=0x5616122183b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x5616122183b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009850 tx=0x7f2f54005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f5e3c0640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2f440a2340 0x7f2f440a4800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005fd0 tx=0x7f2f50009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f4effd640 1 -- 172.21.15.116:0/3399424304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2f480f0830 con 0x7f2f580692d0 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 -- 172.21.15.116:0/1111544429 shutdown_connections 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x56161221ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x5616122183b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1111544429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161220f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 -- 172.21.15.116:0/1111544429 >> 172.21.15.116:0/1111544429 conn(0x561612209b20 msgr2=0x561612209f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 -- 172.21.15.116:0/1111544429 shutdown_connections 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 -- 172.21.15.116:0/1111544429 wait complete. 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 -- start start 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.322+0000 7f2f62a00740 1 Processor -- start 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x5616122159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x561612215f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612216440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f62a00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561611e49430 con 0x56161220fd10 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x5616122159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x5616122159c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58294/0 (socket says 172.21.15.116:58294) 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1360823021 learned_addr learned my addr 172.21.15.116:0/1360823021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x561612215f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f5e3c0640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612216440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561611e05e00 con 0x56161220f3f0 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.323+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5616120fb590 con 0x5616122188f0 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1360823021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 msgr2=0x561612216440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612216440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1360823021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 msgr2=0x561612215f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x561612215f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/1360823021 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f54009530 con 0x56161220f3f0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x561612215f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5e3c0640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612216440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f62a00740 10 client.0 _ll_register_callbacks cb 0x56161216afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x5616122159c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f54004600 tx=0x7f2f5401bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d1b8ac581fa93b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f417fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f417fa640 1 -- 172.21.15.116:0/1360823021 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5401bed0 con 0x56161220f3f0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.324+0000 7f2f417fa640 1 -- 172.21.15.116:0/1360823021 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2f5400bcb0 con 0x56161220f3f0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561611e757e0 con 0x56161220f3f0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f417fa640 1 -- 172.21.15.116:0/1360823021 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f5400be70 con 0x56161220f3f0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 10 client.25691 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=29}) -- 0x561612174bd0 con 0x56161220f3f0 2024-03-24T09:22:43.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f417fa640 1 -- 172.21.15.116:0/1360823021 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2f54035070 con 0x56161220f3f0 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 10 client.25691 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 10 client.25691 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 20 client.25691 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.325+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x56161222ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.326+0000 7f2f62a00740 4 client.25691 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.326+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561611e757e0 con 0x561612228640 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.326+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x56161222ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.326+0000 7f2f417fa640 10 client.25691 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:22:43.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.326+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x56161222ab00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ae122bb41da07dc6 server_cookie=bc2ac22dc5bc7980 in_seq=0 out_seq=0 2024-03-24T09:22:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f417fa640 1 -- 172.21.15.116:0/1360823021 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561611e757e0 con 0x561612228640 2024-03-24T09:22:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f417fa640 10 client.25691 handle_command_reply: tid=0 2024-03-24T09:22:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 client.25691 shutdown 2024-03-24T09:22:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 msgr2=0x5616122159c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x5616122159c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2f54004600 tx=0x7f2f5401bce0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 shutdown_connections 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x56161222ab00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f417fa640 0 client.25691 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612216440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220fd10 0x561612215f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 --2- 172.21.15.116:0/1360823021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x5616122159c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 >> 172.21.15.116:0/1360823021 conn(0x561612209b20 msgr2=0x56161220b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 shutdown_connections 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 1 -- 172.21.15.116:0/1360823021 wait complete. 2024-03-24T09:22:43.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.327+0000 7f2f62a00740 20 client.25691 trim_cache size 0 max 16384 2024-03-24T09:22:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 Processor -- start 2024-03-24T09:22:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 -- start start 2024-03-24T09:22:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220f3f0 0x561612224180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 0x561612208430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612215170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56161221b2b0 con 0x56161220f3f0 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5616122246c0 con 0x56161220fd10 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f62a00740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561612208970 con 0x5616122188f0 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f5d3be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 0x561612208430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220f3f0 0x561612224180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f5d3be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 0x561612208430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58304/0 (socket says 172.21.15.116:58304) 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f5e3c0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612215170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.341+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3697779636 learned_addr learned my addr 172.21.15.116:0/3697779636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3697779636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 msgr2=0x561612215170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612215170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3697779636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220f3f0 msgr2=0x561612224180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220f3f0 0x561612224180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f5d3be640 1 -- 172.21.15.116:0/3697779636 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561612110c80 con 0x56161220fd10 2024-03-24T09:22:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 0x561612208430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2f480bcda0 tx=0x7f2f48041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb548803e80c2a80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f437fe640 1 -- 172.21.15.116:0/3697779636 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f480d9070 con 0x56161220fd10 2024-03-24T09:22:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f437fe640 1 -- 172.21.15.116:0/3697779636 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f480d4070 con 0x56161220fd10 2024-03-24T09:22:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.342+0000 7f2f437fe640 1 -- 172.21.15.116:0/3697779636 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f480eddc0 con 0x56161220fd10 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 -- 172.21.15.116:0/3697779636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 msgr2=0x561612208430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 0x561612208430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2f480bcda0 tx=0x7f2f48041040 comp rx=0 tx=0).stop 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 -- 172.21.15.116:0/3697779636 shutdown_connections 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5616122188f0 0x561612215170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220fd10 0x561612208430 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 --2- 172.21.15.116:0/3697779636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56161220f3f0 0x561612224180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 -- 172.21.15.116:0/3697779636 >> 172.21.15.116:0/3697779636 conn(0x561612209b20 msgr2=0x561612216210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 -- 172.21.15.116:0/3697779636 shutdown_connections 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 -- 172.21.15.116:0/3697779636 wait complete. 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 -- start start 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.343+0000 7f2f62a00740 1 Processor -- start 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161221c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56161220fd10 0x56161221ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616122188f0 0x56161228da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56161221d400 con 0x5616122188f0 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56161228df80 con 0x56161220f3f0 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56161228e110 con 0x56161220fd10 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161221c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f62a00740 10 client.0 _ll_register_callbacks cb 0x7f2f50009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f5dbbf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161221c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58312/0 (socket says 172.21.15.116:58312) 2024-03-24T09:22:43.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/4245092923 learned_addr learned my addr 172.21.15.116:0/4245092923 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56161220fd10 0x56161221ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.344+0000 7f2f5e3c0640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616122188f0 0x56161228da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/4245092923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56161220fd10 msgr2=0x56161221ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56161220fd10 0x56161221ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/4245092923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616122188f0 msgr2=0x56161228da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616122188f0 0x56161228da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f5dbbf640 1 -- 172.21.15.116:0/4245092923 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f581006e0 con 0x56161220f3f0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f5dbbf640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161221c780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2f5401a7e0 tx=0x7f2f54012040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8b608f36b36f1ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f417fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f417fa640 1 -- 172.21.15.116:0/4245092923 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f54015040 con 0x56161220f3f0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f417fa640 1 -- 172.21.15.116:0/4245092923 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2f54000e40 con 0x56161220f3f0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f54016040 con 0x56161220f3f0 2024-03-24T09:22:43.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f417fa640 1 -- 172.21.15.116:0/4245092923 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2f5401aa10 con 0x56161220f3f0 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f62a00740 10 client.25697 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561612136920 con 0x56161220f3f0 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.345+0000 7f2f417fa640 1 -- 172.21.15.116:0/4245092923 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2f5401ab70 con 0x56161220f3f0 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f417fa640 1 -- 172.21.15.116:0/4245092923 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f2f54035070 con 0x56161220f3f0 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 10 client.25697 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 10 client.25697 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 20 client.25697 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x561612291660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 4 client.25697 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2f54016040 con 0x561612228640 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x561612291660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f417fa640 10 client.25697 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:22:43.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f5d3be640 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x561612291660 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f159c246b282e5d server_cookie=9aa4493dc4fa5017 in_seq=0 out_seq=0 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f417fa640 1 -- 172.21.15.116:0/4245092923 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2f54016040 con 0x561612228640 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f417fa640 10 client.25697 handle_command_reply: tid=0 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.346+0000 7f2f62a00740 1 client.25697 shutdown 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 msgr2=0x56161221c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161221c780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2f5401a7e0 tx=0x7f2f54012040 comp rx=0 tx=0).stop 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 shutdown_connections 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561612228640 0x561612291660 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5616122188f0 0x56161228da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f417fa640 0 client.25697 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56161220fd10 0x56161221ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 --2- 172.21.15.116:0/4245092923 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56161220f3f0 0x56161221c780 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 >> 172.21.15.116:0/4245092923 conn(0x561612209b20 msgr2=0x561612223170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 shutdown_connections 2024-03-24T09:22:43.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 1 -- 172.21.15.116:0/4245092923 wait complete. 2024-03-24T09:22:43.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f62a00740 20 client.25697 trim_cache size 0 max 16384 2024-03-24T09:22:43.353 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.347+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2f440a2340 msgr2=0x7f2f440a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2f440a2340 0x7f2f440a4800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2f50005fd0 tx=0x7f2f50009500 comp rx=0 tx=0).stop 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 msgr2=0x7f2f5819de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5819de60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2f4802c230 tx=0x7f2f48005d50 comp rx=0 tx=0).stop 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 shutdown_connections 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2f440a2340 0x7f2f440a4800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2f58107760 0x7f2f58197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f580692d0 0x7f2f5819de60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 --2- 172.21.15.116:0/3399424304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f58068910 0x7f2f5819d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 >> 172.21.15.116:0/3399424304 conn(0x7f2f58077710 msgr2=0x7f2f580fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 shutdown_connections 2024-03-24T09:22:43.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.348+0000 7f2f5fe23640 1 -- 172.21.15.116:0/3399424304 wait complete. 2024-03-24T09:22:43.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:22:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 -- 172.21.15.116:0/1898576275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa81032b0 msgr2=0x7feaa810b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1898576275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa81032b0 0x7feaa810b790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fea94009860 tx=0x7fea9402bc00 comp rx=0 tx=0).stop 2024-03-24T09:22:43.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 -- 172.21.15.116:0/1898576275 shutdown_connections 2024-03-24T09:22:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1898576275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa810bcd0 0x7feaa810e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1898576275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa81032b0 0x7feaa810b790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1898576275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa8102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 -- 172.21.15.116:0/1898576275 >> 172.21.15.116:0/1898576275 conn(0x7feaa8077770 msgr2=0x7feaa8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 -- 172.21.15.116:0/1898576275 shutdown_connections 2024-03-24T09:22:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 -- 172.21.15.116:0/1898576275 wait complete. 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.465+0000 7feaaea9d640 1 Processor -- start 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 -- start start 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa81032b0 0x7feaa819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa810bcd0 0x7feaa8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feaa8198100 con 0x7feaa8102990 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feaa8198270 con 0x7feaa810bcd0 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaaea9d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7feaa8198430 con 0x7feaa81032b0 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40388/0 (socket says 172.21.15.116:40388) 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 -- 172.21.15.116:0/1013634542 learned_addr learned my addr 172.21.15.116:0/1013634542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaa7fff640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa81032b0 0x7feaa819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaad03a640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa810bcd0 0x7feaa8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 -- 172.21.15.116:0/1013634542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa81032b0 msgr2=0x7feaa819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa81032b0 0x7feaa819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 -- 172.21.15.116:0/1013634542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa810bcd0 msgr2=0x7feaa8197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa810bcd0 0x7feaa8197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 -- 172.21.15.116:0/1013634542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa0009900 con 0x7feaa8102990 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaad03a640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa810bcd0 0x7feaa8197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaa7fff640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa81032b0 0x7feaa819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.466+0000 7feaac839640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa819d900 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e9f0 tx=0x7fea9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.467+0000 7feaa5ffb640 1 -- 172.21.15.116:0/1013634542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c009700 con 0x7feaa8102990 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.467+0000 7feaa5ffb640 1 -- 172.21.15.116:0/1013634542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea9c00cd80 con 0x7feaa8102990 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.467+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea94009490 con 0x7feaa8102990 2024-03-24T09:22:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.467+0000 7feaa5ffb640 1 -- 172.21.15.116:0/1013634542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c018840 con 0x7feaa8102990 2024-03-24T09:22:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.467+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa81a47c0 con 0x7feaa8102990 2024-03-24T09:22:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.470+0000 7feaa5ffb640 1 -- 172.21.15.116:0/1013634542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7fea9c0189a0 con 0x7feaa8102990 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.470+0000 7feab167e740 1 Processor -- start 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.470+0000 7feab167e740 1 -- start start 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feab167e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 0x55685f8aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feab167e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feab167e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feab167e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55685f818840 con 0x55685f8aad10 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feab167e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55685f815d70 con 0x55685f8b38f0 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feab167e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55685f811550 con 0x55685f8aa3f0 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaa5ffb640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fea980a2340 0x7fea980a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 0x55685f8aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaac839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 0x55685f8aa7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:51248/0 (socket says 172.21.15.116:51248) 2024-03-24T09:22:43.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 -- 172.21.15.116:0/3918358237 learned_addr learned my addr 172.21.15.116:0/3918358237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaa7fff640 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fea980a2340 0x7fea980a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaa5ffb640 1 -- 172.21.15.116:0/1013634542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fea9c014070 con 0x7feaa8102990 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 -- 172.21.15.116:0/3918358237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 msgr2=0x55685f8b5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8b5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 -- 172.21.15.116:0/3918358237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 msgr2=0x55685f8b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaad03a640 1 -- 172.21.15.116:0/3918358237 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55685f7a6de0 con 0x55685f8aa3f0 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaa7fff640 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.471+0000 7feaac839640 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.472+0000 7feaad03a640 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 0x55685f8aa7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feaa0005c40 tx=0x7feaa000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49e5202a1bb21c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.472+0000 7feaad03a640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fea980a2340 0x7fea980a4800 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feaa001d000 tx=0x7feaa001f000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.472+0000 7fea92ffd640 1 -- 172.21.15.116:0/3918358237 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0004410 con 0x55685f8aa3f0 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.472+0000 7fea92ffd640 1 -- 172.21.15.116:0/3918358237 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feaa0004570 con 0x55685f8aa3f0 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.472+0000 7fea92ffd640 1 -- 172.21.15.116:0/3918358237 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa000b680 con 0x55685f8aa3f0 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 -- 172.21.15.116:0/3918358237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 msgr2=0x55685f8aa7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 0x55685f8aa7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feaa0005c40 tx=0x7feaa000c800 comp rx=0 tx=0).stop 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 -- 172.21.15.116:0/3918358237 shutdown_connections 2024-03-24T09:22:43.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 --2- 172.21.15.116:0/3918358237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aa3f0 0x55685f8aa7d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 -- 172.21.15.116:0/3918358237 >> 172.21.15.116:0/3918358237 conn(0x55685f8a4b20 msgr2=0x55685f8a4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 -- 172.21.15.116:0/3918358237 shutdown_connections 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 -- 172.21.15.116:0/3918358237 wait complete. 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.473+0000 7feab167e740 1 -- start start 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 Processor -- start 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f8b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feaac839640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40398/0 (socket says 172.21.15.116:40398) 2024-03-24T09:22:43.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feaac839640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58334/0 (socket says 172.21.15.116:58334) 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feaa7fff640 1 -- 172.21.15.116:0/1787326637 learned_addr learned my addr 172.21.15.116:0/1787326637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55685f4e4430 con 0x55685f8aad10 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55685f4a0e00 con 0x55685f8aa3f0 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55685f796590 con 0x55685f8b38f0 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.474+0000 7feaad03a640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f8b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feab167e740 10 client.0 _ll_register_callbacks cb 0x55685f805fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaac839640 1 -- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 msgr2=0x55685f8b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaac839640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f8b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaac839640 1 -- 172.21.15.116:0/1787326637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 msgr2=0x55685f8b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaac839640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaac839640 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa000ebe0 con 0x55685f8aa3f0 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaa7fff640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55685f5107e0 con 0x55685f8aa3f0 2024-03-24T09:22:43.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feaac839640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b0a30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0cbb50 tx=0x7fea9c0a34e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cdecbee5ac3644d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7fea87fff640 10 client.25709 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7fea87fff640 1 -- 172.21.15.116:0/1787326637 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0ce660 con 0x55685f8aa3f0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7fea87fff640 1 -- 172.21.15.116:0/1787326637 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fea9c010c10 con 0x55685f8aa3f0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7fea87fff640 1 -- 172.21.15.116:0/1787326637 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea9c0cc010 con 0x55685f8aa3f0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feab167e740 10 client.25709 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.475+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55685f80fbd0 con 0x55685f8aa3f0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7fea87fff640 1 -- 172.21.15.116:0/1787326637 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fea9c09a240 con 0x55685f8aa3f0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feab167e740 10 client.25709 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feab167e740 10 client.25709 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feab167e740 20 client.25709 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feab167e740 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f8c38f0 0x55685f8c5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feab167e740 4 client.25709 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55685f54d720 con 0x55685f8c38f0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feaa7fff640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f8c38f0 0x55685f8c5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7fea87fff640 10 client.25709 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:22:43.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.476+0000 7feaa7fff640 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f8c38f0 0x55685f8c5d40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e2d018540cb284a server_cookie=276c37e745e40982 in_seq=0 out_seq=0 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.477+0000 7fea87fff640 1 -- 172.21.15.116:0/1787326637 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55685f54d720 con 0x55685f8c38f0 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7fea87fff640 10 client.25709 handle_command_reply: tid=0 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 client.25709 shutdown 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 msgr2=0x55685f8b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b0a30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0cbb50 tx=0x7fea9c0a34e0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 shutdown_connections 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f8b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f8c38f0 0x55685f8c5d40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 --2- 172.21.15.116:0/1787326637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b0a30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 >> 172.21.15.116:0/1787326637 conn(0x55685f8a4b20 msgr2=0x55685f8a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7fea87fff640 0 client.25709 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:22:43.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 shutdown_connections 2024-03-24T09:22:43.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 1 -- 172.21.15.116:0/1787326637 wait complete. 2024-03-24T09:22:43.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.478+0000 7feab167e740 20 client.25709 trim_cache size 0 max 16384 2024-03-24T09:22:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.503+0000 7feab167e740 1 Processor -- start 2024-03-24T09:22:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.503+0000 7feab167e740 1 -- start start 2024-03-24T09:22:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feab167e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aa3f0 0x55685f8bf120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feab167e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aad10 0x55685f8af300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feab167e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8cb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feab167e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55685f8bf690 con 0x55685f8aa3f0 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feab167e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55685f8af870 con 0x55685f8b38f0 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feab167e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55685f8aa0a0 con 0x55685f8aad10 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaac839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aa3f0 0x55685f8bf120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaac839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aa3f0 0x55685f8bf120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40414/0 (socket says 172.21.15.116:40414) 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaac839640 1 -- 172.21.15.116:0/1947637863 learned_addr learned my addr 172.21.15.116:0/1947637863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aad10 0x55685f8af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 -- 172.21.15.116:0/1947637863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aad10 msgr2=0x55685f8af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aad10 0x55685f8af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 -- 172.21.15.116:0/1947637863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aa3f0 msgr2=0x55685f8bf120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aa3f0 0x55685f8bf120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 -- 172.21.15.116:0/1947637863 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55685f7abc80 con 0x55685f8b38f0 2024-03-24T09:22:43.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.504+0000 7feaad03a640 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8cb600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7feaa0006010 tx=0x7feaa000eb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abd5d5609802ccd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.505+0000 7fea927fc640 1 -- 172.21.15.116:0/1947637863 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa00048e0 con 0x55685f8b38f0 2024-03-24T09:22:43.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.505+0000 7fea927fc640 1 -- 172.21.15.116:0/1947637863 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feaa001e930 con 0x55685f8b38f0 2024-03-24T09:22:43.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.505+0000 7fea927fc640 1 -- 172.21.15.116:0/1947637863 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0031830 con 0x55685f8b38f0 2024-03-24T09:22:43.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 -- 172.21.15.116:0/1947637863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 msgr2=0x55685f8cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8cb600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7feaa0006010 tx=0x7feaa000eb20 comp rx=0 tx=0).stop 2024-03-24T09:22:43.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 -- 172.21.15.116:0/1947637863 shutdown_connections 2024-03-24T09:22:43.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8b38f0 0x55685f8cb600 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8aad10 0x55685f8af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 --2- 172.21.15.116:0/1947637863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aa3f0 0x55685f8bf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 -- 172.21.15.116:0/1947637863 >> 172.21.15.116:0/1947637863 conn(0x55685f8a4b20 msgr2=0x55685f8a67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 -- 172.21.15.116:0/1947637863 shutdown_connections 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 -- 172.21.15.116:0/1947637863 wait complete. 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 -- start start 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.506+0000 7feab167e740 1 Processor -- start 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feab167e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feab167e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feab167e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f928c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feab167e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55685f8b8240 con 0x55685f8aad10 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feaac839640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40424/0 (socket says 172.21.15.116:40424) 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feaa7fff640 1 -- 172.21.15.116:0/3655282884 learned_addr learned my addr 172.21.15.116:0/3655282884 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feaad03a640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f928c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55685f929170 con 0x55685f8aa3f0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.507+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55685f9292b0 con 0x55685f8b38f0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feab167e740 10 client.0 _ll_register_callbacks cb 0x7fea9c09ac50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaac839640 1 -- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 msgr2=0x55685f928c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaac839640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f928c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaac839640 1 -- 172.21.15.116:0/3655282884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 msgr2=0x55685f8b7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaac839640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaac839640 1 -- 172.21.15.116:0/3655282884 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55685f7a6de0 con 0x55685f8aa3f0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaa7fff640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feaac839640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b7450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0977c0 tx=0x7fea9c0d0b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e02891d90e5febcb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7fea87fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7fea87fff640 1 -- 172.21.15.116:0/3655282884 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0cc010 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7fea87fff640 1 -- 172.21.15.116:0/3655282884 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fea9c010c10 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaa0004570 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7fea87fff640 1 -- 172.21.15.116:0/3655282884 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fea9c0cba90 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feab167e740 10 client.25715 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.508+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55685f7d1920 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7fea87fff640 1 -- 172.21.15.116:0/3655282884 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fea9c0b7060 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7fea87fff640 1 -- 172.21.15.116:0/3655282884 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fea9c09fdb0 con 0x55685f8aa3f0 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feab167e740 10 client.25715 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feab167e740 10 client.25715 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feab167e740 20 client.25715 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feab167e740 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f92a740 0x55685f92cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feab167e740 4 client.25715 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7feaa0004570 con 0x55685f92a740 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feaa7fff640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f92a740 0x55685f92cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7fea87fff640 10 client.25715 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.509+0000 7feaa7fff640 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f92a740 0x55685f92cb30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=58e43708fa22e561 server_cookie=b4ecff851345b238 in_seq=0 out_seq=0 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7fea87fff640 1 -- 172.21.15.116:0/3655282884 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7feaa0004570 con 0x55685f92a740 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7fea87fff640 10 client.25715 handle_command_reply: tid=0 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 client.25715 shutdown 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 msgr2=0x55685f8b7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b7450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0977c0 tx=0x7fea9c0d0b80 comp rx=0 tx=0).stop 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 shutdown_connections 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55685f8b38f0 0x55685f928c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55685f8aad10 0x55685f8b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55685f92a740 0x55685f92cb30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 --2- 172.21.15.116:0/3655282884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55685f8aa3f0 0x55685f8b7450 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 >> 172.21.15.116:0/3655282884 conn(0x55685f8a4b20 msgr2=0x55685f8a5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7fea87fff640 0 client.25715 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:22:43.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 shutdown_connections 2024-03-24T09:22:43.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 1 -- 172.21.15.116:0/3655282884 wait complete. 2024-03-24T09:22:43.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.510+0000 7feab167e740 20 client.25715 trim_cache size 0 max 16384 2024-03-24T09:22:43.516 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fea980a2340 msgr2=0x7fea980a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fea980a2340 0x7fea980a4800 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7feaa001d000 tx=0x7feaa001f000 comp rx=0 tx=0).stop 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 msgr2=0x7feaa819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa819d900 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e9f0 tx=0x7fea9c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 shutdown_connections 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feaa810bcd0 0x7feaa8197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fea980a2340 0x7fea980a4800 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feaa81032b0 0x7feaa819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 --2- 172.21.15.116:0/1013634542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaa8102990 0x7feaa819d900 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 >> 172.21.15.116:0/1013634542 conn(0x7feaa8077770 msgr2=0x7feaa80fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 shutdown_connections 2024-03-24T09:22:43.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.511+0000 7feaaea9d640 1 -- 172.21.15.116:0/1013634542 wait complete. 2024-03-24T09:22:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:43 smithi143 ceph-mon[35965]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-24T09:22:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:43 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1829171195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:43.526 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:22:43.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:43 smithi117 ceph-mon[35788]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-24T09:22:43.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:43 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1829171195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:22:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 -- 172.21.15.116:0/2693147541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 msgr2=0x7f897410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 --2- 172.21.15.116:0/2693147541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897410e570 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f896c009d00 tx=0x7f896c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 -- 172.21.15.116:0/2693147541 shutdown_connections 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 --2- 172.21.15.116:0/2693147541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897410e570 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 --2- 172.21.15.116:0/2693147541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f8974068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 --2- 172.21.15.116:0/2693147541 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 0x7f8974107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 -- 172.21.15.116:0/2693147541 >> 172.21.15.116:0/2693147541 conn(0x7f8974077710 msgr2=0x7f8974077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 -- 172.21.15.116:0/2693147541 shutdown_connections 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 -- 172.21.15.116:0/2693147541 wait complete. 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 Processor -- start 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.632+0000 7f897a874640 1 -- start start 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f897a874640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f897419d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f897a874640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f897a874640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 0x7f8974197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f897a874640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8974198070 con 0x7f8974068910 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f897a874640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f89741981e0 con 0x7f89740692d0 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f897a874640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f89741983a0 con 0x7f8974107760 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8973fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f897419d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8973fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f897419d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40448/0 (socket says 172.21.15.116:40448) 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8973fff640 1 -- 172.21.15.116:0/219014615 learned_addr learned my addr 172.21.15.116:0/219014615 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8978e11640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 0x7f8974197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8978e11640 1 -- 172.21.15.116:0/219014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 msgr2=0x7f897419ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f89737fe640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8978e11640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897419ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8978e11640 1 -- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 msgr2=0x7f897419d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8978e11640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f897419d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8978e11640 1 -- 172.21.15.116:0/219014615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8968009490 con 0x7f8974107760 2024-03-24T09:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f8973fff640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f897419d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.633+0000 7f89737fe640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897419ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.634+0000 7f8978e11640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 0x7f8974197a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f896c009bf0 tx=0x7f896c035ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.634+0000 7f89717fa640 1 -- 172.21.15.116:0/219014615 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c0329e0 con 0x7f8974107760 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.634+0000 7f89717fa640 1 -- 172.21.15.116:0/219014615 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f896c003e20 con 0x7f8974107760 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.634+0000 7f89717fa640 1 -- 172.21.15.116:0/219014615 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896c004610 con 0x7f8974107760 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.634+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c009900 con 0x7f8974107760 2024-03-24T09:22:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.634+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89741a4730 con 0x7f8974107760 2024-03-24T09:22:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.636+0000 7f89717fa640 1 -- 172.21.15.116:0/219014615 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f896c03b040 con 0x7f8974107760 2024-03-24T09:22:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.636+0000 7f89717fa640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f895c0a2340 0x7f895c0a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.637+0000 7f89717fa640 1 -- 172.21.15.116:0/219014615 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f896c0f0da0 con 0x7f8974107760 2024-03-24T09:22:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.637+0000 7f8973fff640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f895c0a2340 0x7f895c0a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.637+0000 7f8973fff640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f895c0a2340 0x7f895c0a4800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8974198b60 tx=0x7f8964005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.637+0000 7f897d44b740 1 Processor -- start 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.637+0000 7f897d44b740 1 -- start start 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f897d44b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2ba47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f897d44b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 0x55cef2bad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f897d44b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bafce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f897d44b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cef2b12840 con 0x55cef2bad8f0 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f897d44b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cef2b0fd70 con 0x55cef2ba4d10 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f897d44b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cef2b0b550 con 0x55cef2ba43f0 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 0x55cef2bad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f8978e11640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bafce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 0x55cef2bad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58390/0 (socket says 172.21.15.116:58390) 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f8978e11640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bafce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40458/0 (socket says 172.21.15.116:40458) 2024-03-24T09:22:43.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 -- 172.21.15.116:0/3662953686 learned_addr learned my addr 172.21.15.116:0/3662953686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f8973fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2ba47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 -- 172.21.15.116:0/3662953686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 msgr2=0x55cef2ba47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2ba47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 -- 172.21.15.116:0/3662953686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 msgr2=0x55cef2bafce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bafce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 -- 172.21.15.116:0/3662953686 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cef2aa0de0 con 0x55cef2ba4d10 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.638+0000 7f89737fe640 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 0x55cef2bad3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8968009d60 tx=0x7f8968010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e986c96d7ddd06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.639+0000 7f895a7fc640 1 -- 172.21.15.116:0/3662953686 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968005220 con 0x55cef2ba4d10 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.639+0000 7f895a7fc640 1 -- 172.21.15.116:0/3662953686 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8968010e50 con 0x55cef2ba4d10 2024-03-24T09:22:43.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.639+0000 7f895a7fc640 1 -- 172.21.15.116:0/3662953686 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968012690 con 0x55cef2ba4d10 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.639+0000 7f897d44b740 1 -- 172.21.15.116:0/3662953686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 msgr2=0x55cef2bad3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.639+0000 7f897d44b740 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 0x55cef2bad3b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8968009d60 tx=0x7f8968010740 comp rx=0 tx=0).stop 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 -- 172.21.15.116:0/3662953686 shutdown_connections 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bafce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba4d10 0x55cef2bad3b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 --2- 172.21.15.116:0/3662953686 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2ba47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 -- 172.21.15.116:0/3662953686 >> 172.21.15.116:0/3662953686 conn(0x55cef2b9eb20 msgr2=0x55cef2b9ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 -- 172.21.15.116:0/3662953686 shutdown_connections 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 -- 172.21.15.116:0/3662953686 wait complete. 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 -- start start 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.640+0000 7f897d44b740 1 Processor -- start 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2baa990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2baaed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f89737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2baaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f89737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2baaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40472/0 (socket says 172.21.15.116:40472) 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f89737fe640 1 -- 172.21.15.116:0/2256300937 learned_addr learned my addr 172.21.15.116:0/2256300937 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2baa990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bab410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cef27de430 con 0x55cef2ba4d10 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cef279ae00 con 0x55cef2bad8f0 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cef2a90590 con 0x55cef2ba43f0 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8978e11640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bab410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 -- 172.21.15.116:0/2256300937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 msgr2=0x55cef2bab410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bab410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 -- 172.21.15.116:0/2256300937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 msgr2=0x55cef2baaed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2baaed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89680099c0 con 0x55cef2ba43f0 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f89737fe640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2baaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f8973fff640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2baa990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8964009f40 tx=0x7f8964004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3380223c748f931d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f89537fe640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.641+0000 7f897d44b740 10 client.0 _ll_register_callbacks cb 0x55cef2afffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cef280a7e0 con 0x55cef2ba43f0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f89537fe640 1 -- 172.21.15.116:0/2256300937 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964023070 con 0x55cef2ba43f0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f89537fe640 1 -- 172.21.15.116:0/2256300937 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8964003e10 con 0x55cef2ba43f0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f89537fe640 1 -- 172.21.15.116:0/2256300937 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f896400fbc0 con 0x55cef2ba43f0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 10 client.25618 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55cef2b09bd0 con 0x55cef2ba43f0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f89537fe640 1 -- 172.21.15.116:0/2256300937 <== mon.1 v2:172.21.15.143:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f896400fd20 con 0x55cef2ba43f0 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 10 client.25618 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 10 client.25618 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 20 client.25618 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2bbd880 0x55cef2bbfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 4 client.25618 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:43.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.642+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cef2847720 con 0x55cef2bbd880 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.643+0000 7f89737fe640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2bbd880 0x55cef2bbfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.643+0000 7f89537fe640 10 client.25618 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.643+0000 7f89737fe640 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2bbd880 0x55cef2bbfd40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d0fa1a0e63b8198 server_cookie=c7b350497bee2f14 in_seq=0 out_seq=0 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f89537fe640 1 -- 172.21.15.116:0/2256300937 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cef2847720 con 0x55cef2bbd880 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f89537fe640 10 client.25618 handle_command_reply: tid=0 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 client.25618 shutdown 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 msgr2=0x55cef2baa990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2baa990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8964009f40 tx=0x7f8964004730 comp rx=0 tx=0).stop 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 shutdown_connections 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2bbd880 0x55cef2bbfd40 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bab410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f89537fe640 0 client.25618 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2baaed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 --2- 172.21.15.116:0/2256300937 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2baa990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 >> 172.21.15.116:0/2256300937 conn(0x55cef2b9eb20 msgr2=0x55cef2ba0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 shutdown_connections 2024-03-24T09:22:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 1 -- 172.21.15.116:0/2256300937 wait complete. 2024-03-24T09:22:43.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.644+0000 7f897d44b740 20 client.25618 trim_cache size 0 max 16384 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 Processor -- start 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 -- start start 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba43f0 0x55cef2bad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 0x55cef2b9d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cef2b9a250 con 0x55cef2bad8f0 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cef2b9d970 con 0x55cef2ba43f0 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.670+0000 7f897d44b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cef2ba4050 con 0x55cef2ba4d10 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 0x55cef2b9d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f8973fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba43f0 0x55cef2bad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 0x55cef2b9d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:51300/0 (socket says 172.21.15.116:51300) 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 -- 172.21.15.116:0/3258242816 learned_addr learned my addr 172.21.15.116:0/3258242816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f8978e11640 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 -- 172.21.15.116:0/3258242816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba43f0 msgr2=0x55cef2bad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba43f0 0x55cef2bad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 -- 172.21.15.116:0/3258242816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 msgr2=0x55cef2bab230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bab230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 -- 172.21.15.116:0/3258242816 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cef2aa5c80 con 0x55cef2ba4d10 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f8978e11640 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f89737fe640 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 0x55cef2b9d430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89680057a0 tx=0x7f8968009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b60d7c7a069b3132 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f8959ffb640 1 -- 172.21.15.116:0/3258242816 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968005c90 con 0x55cef2ba4d10 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f8959ffb640 1 -- 172.21.15.116:0/3258242816 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f896801f040 con 0x55cef2ba4d10 2024-03-24T09:22:43.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.671+0000 7f8959ffb640 1 -- 172.21.15.116:0/3258242816 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968004660 con 0x55cef2ba4d10 2024-03-24T09:22:43.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 -- 172.21.15.116:0/3258242816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 msgr2=0x55cef2b9d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 0x55cef2b9d430 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f89680057a0 tx=0x7f8968009760 comp rx=0 tx=0).stop 2024-03-24T09:22:43.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 -- 172.21.15.116:0/3258242816 shutdown_connections 2024-03-24T09:22:43.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2bad8f0 0x55cef2bab230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba4d10 0x55cef2b9d430 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 --2- 172.21.15.116:0/3258242816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2ba43f0 0x55cef2bad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 -- 172.21.15.116:0/3258242816 >> 172.21.15.116:0/3258242816 conn(0x55cef2b9eb20 msgr2=0x55cef2ba05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 -- 172.21.15.116:0/3258242816 shutdown_connections 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 -- 172.21.15.116:0/3258242816 wait complete. 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.672+0000 7f897d44b740 1 -- start start 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 Processor -- start 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2bb59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2bb2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bb2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cef2bb3280 con 0x55cef2ba4d10 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cef2bb33f0 con 0x55cef2bad8f0 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55cef2bb35b0 con 0x55cef2ba43f0 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f897d44b740 10 client.0 _ll_register_callbacks cb 0x7f8964004140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f89737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2bb2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8973fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2bb59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f89737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2bb2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40482/0 (socket says 172.21.15.116:40482) 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8973fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2bb59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:51316/0 (socket says 172.21.15.116:51316) 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f89737fe640 1 -- 172.21.15.116:0/2710385759 learned_addr learned my addr 172.21.15.116:0/2710385759 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8978e11640 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bb2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8973fff640 1 -- 172.21.15.116:0/2710385759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 msgr2=0x55cef2bb2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8973fff640 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bb2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8973fff640 1 -- 172.21.15.116:0/2710385759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 msgr2=0x55cef2bb2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.673+0000 7f8973fff640 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2bb2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f8973fff640 1 -- 172.21.15.116:0/2710385759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cef2aa0de0 con 0x55cef2ba43f0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f8973fff640 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2bb59a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8964011db0 tx=0x7f8964016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9e1b8501ccdac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f89537fe640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f89537fe640 1 -- 172.21.15.116:0/2710385759 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964032bd0 con 0x55cef2ba43f0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8968010e50 con 0x55cef2ba43f0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f89537fe640 1 -- 172.21.15.116:0/2710385759 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f896400fe80 con 0x55cef2ba43f0 2024-03-24T09:22:43.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f89537fe640 1 -- 172.21.15.116:0/2710385759 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8964032d30 con 0x55cef2ba43f0 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f897d44b740 10 client.25624 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cef2acb920 con 0x55cef2ba43f0 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f89537fe640 1 -- 172.21.15.116:0/2710385759 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8968010e50 con 0x55cef2ba43f0 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f89537fe640 1 -- 172.21.15.116:0/2710385759 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8964028420 con 0x55cef2ba43f0 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.674+0000 7f897d44b740 10 client.25624 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f897d44b740 10 client.25624 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f897d44b740 20 client.25624 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f897d44b740 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2c25260 0x55cef2c27650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f897d44b740 4 client.25624 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cef2896890 con 0x55cef2c25260 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f89737fe640 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2c25260 0x55cef2c27650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f89537fe640 10 client.25624 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f89737fe640 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2c25260 0x55cef2c27650 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=70ec50019143ae4 server_cookie=2ff65ccd6eb4b066 in_seq=0 out_seq=0 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f89537fe640 1 -- 172.21.15.116:0/2710385759 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cef2896890 con 0x55cef2c25260 2024-03-24T09:22:43.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f89537fe640 10 client.25624 handle_command_reply: tid=0 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.675+0000 7f897d44b740 1 client.25624 shutdown 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 msgr2=0x55cef2bb59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2bb59a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8964011db0 tx=0x7f8964016000 comp rx=0 tx=0).stop 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 shutdown_connections 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55cef2c25260 0x55cef2c27650 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cef2bad8f0 0x55cef2bb2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cef2ba4d10 0x55cef2bb2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 --2- 172.21.15.116:0/2710385759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55cef2ba43f0 0x55cef2bb59a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f89537fe640 0 client.25624 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 >> 172.21.15.116:0/2710385759 conn(0x55cef2b9eb20 msgr2=0x55cef2babff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 shutdown_connections 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 1 -- 172.21.15.116:0/2710385759 wait complete. 2024-03-24T09:22:43.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897d44b740 20 client.25624 trim_cache size 0 max 16384 2024-03-24T09:22:43.682 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:22:43.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f895c0a2340 msgr2=0x7f895c0a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f895c0a2340 0x7f895c0a4800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8974198b60 tx=0x7f8964005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:43.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 msgr2=0x7f8974197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 0x7f8974197a20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f896c009bf0 tx=0x7f896c035ca0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 shutdown_connections 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f895c0a2340 0x7f895c0a4800 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8974107760 0x7f8974197a20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89740692d0 0x7f897419ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 --2- 172.21.15.116:0/219014615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974068910 0x7f897419d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.676+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 >> 172.21.15.116:0/219014615 conn(0x7f8974077710 msgr2=0x7f89740fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.677+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 shutdown_connections 2024-03-24T09:22:43.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.677+0000 7f897a874640 1 -- 172.21.15.116:0/219014615 wait complete. 2024-03-24T09:22:43.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 -- 172.21.15.116:0/3301942950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9350100f60 msgr2=0x7f935010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 --2- 172.21.15.116:0/3301942950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9350100f60 0x7f935010d420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9348009c10 tx=0x7f934802bef0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 -- 172.21.15.116:0/3301942950 shutdown_connections 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 --2- 172.21.15.116:0/3301942950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9350100f60 0x7f935010d420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 --2- 172.21.15.116:0/3301942950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93501005a0 0x7f9350100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 --2- 172.21.15.116:0/3301942950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9350106660 0x7f9350106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 -- 172.21.15.116:0/3301942950 >> 172.21.15.116:0/3301942950 conn(0x7f9350077710 msgr2=0x7f9350077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 -- 172.21.15.116:0/3301942950 shutdown_connections 2024-03-24T09:22:43.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.797+0000 7f9356088640 1 -- 172.21.15.116:0/3301942950 wait complete. 2024-03-24T09:22:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.798+0000 7f9356088640 1 Processor -- start 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.798+0000 7f9356088640 1 -- start start 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f9356088640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 0x7f935010ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f9356088640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 0x7f935010e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 0x7f935010ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 0x7f935010e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 0x7f935010ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58430/0 (socket says 172.21.15.116:58430) 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 0x7f935010e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40494/0 (socket says 172.21.15.116:40494) 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f9356088640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9350106660 0x7f935010e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f9356088640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93501a5f60 con 0x7f9350100f60 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f9356088640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f93501a60d0 con 0x7f93501005a0 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f9356088640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f93501a6290 con 0x7f9350106660 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 -- 172.21.15.116:0/4221494781 learned_addr learned my addr 172.21.15.116:0/4221494781 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934ffff640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9350106660 0x7f935010e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 -- 172.21.15.116:0/4221494781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9350106660 msgr2=0x7f935010e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9350106660 0x7f935010e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 -- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 msgr2=0x7f935010e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 0x7f935010e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934f7fe640 1 -- 172.21.15.116:0/4221494781 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9348009840 con 0x7f93501005a0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.799+0000 7f934effd640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 0x7f935010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.800+0000 7f934f7fe640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 0x7f935010ddf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f934000e9f0 tx=0x7f934000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.800+0000 7f934cff9640 1 -- 172.21.15.116:0/4221494781 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934000ccc0 con 0x7f93501005a0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.800+0000 7f934cff9640 1 -- 172.21.15.116:0/4221494781 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f934000ce20 con 0x7f93501005a0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.800+0000 7f934cff9640 1 -- 172.21.15.116:0/4221494781 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340010430 con 0x7f93501005a0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.800+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93501a6570 con 0x7f93501005a0 2024-03-24T09:22:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.800+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93501a6a20 con 0x7f93501005a0 2024-03-24T09:22:43.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.802+0000 7f934cff9640 1 -- 172.21.15.116:0/4221494781 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f9340004f90 con 0x7f93501005a0 2024-03-24T09:22:43.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.803+0000 7f934cff9640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f93380a2340 0x7f93380a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.803+0000 7f934effd640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f93380a2340 0x7f93380a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.803+0000 7f934cff9640 1 -- 172.21.15.116:0/4221494781 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9340014070 con 0x7f93501005a0 2024-03-24T09:22:43.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.804+0000 7f934effd640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f93380a2340 0x7f93380a4800 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f935010f880 tx=0x7f9344005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.804+0000 7f9358c6e740 1 Processor -- start 2024-03-24T09:22:43.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.804+0000 7f9358c6e740 1 -- start start 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.804+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac596d10 0x55a9ac59f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40496/0 (socket says 172.21.15.116:40496) 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 -- 172.21.15.116:0/1535953427 learned_addr learned my addr 172.21.15.116:0/1535953427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac504840 con 0x55a9ac5963f0 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac501d70 con 0x55a9ac59f8f0 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac4fd550 con 0x55a9ac596d10 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934ffff640 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934effd640 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac596d10 0x55a9ac59f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 -- 172.21.15.116:0/1535953427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac596d10 msgr2=0x55a9ac59f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac596d10 0x55a9ac59f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 -- 172.21.15.116:0/1535953427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 msgr2=0x55a9ac5a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 -- 172.21.15.116:0/1535953427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9ac492de0 con 0x55a9ac5963f0 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934effd640 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac596d10 0x55a9ac59f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f934f7fe640 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5967d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f93400cbba0 tx=0x7f934009a720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e43394d25c19999 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.805+0000 7f9335ffb640 1 -- 172.21.15.116:0/1535953427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934009ee50 con 0x55a9ac5963f0 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9335ffb640 1 -- 172.21.15.116:0/1535953427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93400a34a0 con 0x55a9ac5963f0 2024-03-24T09:22:43.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9335ffb640 1 -- 172.21.15.116:0/1535953427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93400cbfc0 con 0x55a9ac5963f0 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 msgr2=0x55a9ac5967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5967d0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f93400cbba0 tx=0x7f934009a720 comp rx=0 tx=0).stop 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 shutdown_connections 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac596d10 0x55a9ac59f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1535953427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5967d0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.806+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 >> 172.21.15.116:0/1535953427 conn(0x55a9ac590b20 msgr2=0x55a9ac590f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.807+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 shutdown_connections 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.807+0000 7f9358c6e740 1 -- 172.21.15.116:0/1535953427 wait complete. 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.807+0000 7f9358c6e740 1 -- start start 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.807+0000 7f9358c6e740 1 Processor -- start 2024-03-24T09:22:43.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac59c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac596d10 0x55a9ac59ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac59d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac1d0430 con 0x55a9ac596d10 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac18ce00 con 0x55a9ac59f8f0 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac482590 con 0x55a9ac5963f0 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac59c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac596d10 0x55a9ac59ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac59c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:51346/0 (socket says 172.21.15.116:51346) 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 -- 172.21.15.116:0/1612374997 learned_addr learned my addr 172.21.15.116:0/1612374997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934ffff640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac59d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 10 client.0 _ll_register_callbacks cb 0x55a9ac4f1fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 -- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 msgr2=0x55a9ac59d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac59d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 -- 172.21.15.116:0/1612374997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac596d10 msgr2=0x55a9ac59ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac596d10 0x55a9ac59ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 -- 172.21.15.116:0/1612374997 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93501a6a20 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934effd640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac596d10 0x55a9ac59ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a9ac1fc7e0 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f934f7fe640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac59c990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f93400ce950 tx=0x7f934000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9c58452e5d21873 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.808+0000 7f932effd640 10 client.25633 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f932effd640 1 -- 172.21.15.116:0/1612374997 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93400a9070 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f932effd640 1 -- 172.21.15.116:0/1612374997 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9340099000 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f932effd640 1 -- 172.21.15.116:0/1612374997 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f93400b91f0 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 10 client.25633 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55a9ac4fbbd0 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f932effd640 1 -- 172.21.15.116:0/1612374997 <== mon.1 v2:172.21.15.143:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f934001f060 con 0x55a9ac5963f0 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 10 client.25633 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 20 client.25633 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac5b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 4 client.25633 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.809+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9ac239720 con 0x55a9ac5af510 2024-03-24T09:22:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.810+0000 7f934effd640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac5b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.810+0000 7f932effd640 10 client.25633 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.810+0000 7f934effd640 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac5b19b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=384cb4c486ad8b59 server_cookie=4c64fc70c25fdefa in_seq=0 out_seq=0 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f932effd640 1 -- 172.21.15.116:0/1612374997 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a9ac239720 con 0x55a9ac5af510 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f932effd640 10 client.25633 handle_command_reply: tid=0 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 client.25633 shutdown 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 msgr2=0x55a9ac59c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac59c990 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f93400ce950 tx=0x7f934000cf90 comp rx=0 tx=0).stop 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 shutdown_connections 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac5b19b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac59f8f0 0x55a9ac59d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac596d10 0x55a9ac59ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f932effd640 0 client.25633 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:43.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 --2- 172.21.15.116:0/1612374997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac59c990 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 >> 172.21.15.116:0/1612374997 conn(0x55a9ac590b20 msgr2=0x55a9ac592870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 shutdown_connections 2024-03-24T09:22:43.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 1 -- 172.21.15.116:0/1612374997 wait complete. 2024-03-24T09:22:43.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.811+0000 7f9358c6e740 20 client.25633 trim_cache size 0 max 16384 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.824+0000 7f9358c6e740 1 Processor -- start 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.824+0000 7f9358c6e740 1 -- start start 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5ab180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5b7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac59f8f0 0x55a9ac58fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f9358c6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac597790 con 0x55a9ac5963f0 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f9358c6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac59c780 con 0x55a9ac596d10 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58458/0 (socket says 172.21.15.116:58458) 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5ab180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40512/0 (socket says 172.21.15.116:40512) 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 -- 172.21.15.116:0/344259775 learned_addr learned my addr 172.21.15.116:0/344259775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f9358c6e740 1 -- 172.21.15.116:0/344259775 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac581e20 con 0x55a9ac59f8f0 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934ffff640 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac59f8f0 0x55a9ac58fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 -- 172.21.15.116:0/344259775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac59f8f0 msgr2=0x55a9ac58fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac59f8f0 0x55a9ac58fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 -- 172.21.15.116:0/344259775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 msgr2=0x55a9ac5ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.825+0000 7f934effd640 1 -- 172.21.15.116:0/344259775 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9ac497c80 con 0x55a9ac596d10 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f934effd640 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5b7280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55a9ac595ec0 tx=0x7f9344014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0d234a52db66942 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f93357fa640 1 -- 172.21.15.116:0/344259775 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344022040 con 0x55a9ac596d10 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f93357fa640 1 -- 172.21.15.116:0/344259775 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f934401dd50 con 0x55a9ac596d10 2024-03-24T09:22:43.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f93357fa640 1 -- 172.21.15.116:0/344259775 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344029480 con 0x55a9ac596d10 2024-03-24T09:22:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 -- 172.21.15.116:0/344259775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 msgr2=0x55a9ac5b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5b7280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x55a9ac595ec0 tx=0x7f9344014000 comp rx=0 tx=0).stop 2024-03-24T09:22:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 -- 172.21.15.116:0/344259775 shutdown_connections 2024-03-24T09:22:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac59f8f0 0x55a9ac58fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5b7280 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 --2- 172.21.15.116:0/344259775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac5963f0 0x55a9ac5ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.826+0000 7f9358c6e740 1 -- 172.21.15.116:0/344259775 >> 172.21.15.116:0/344259775 conn(0x55a9ac590b20 msgr2=0x55a9ac58f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 -- 172.21.15.116:0/344259775 shutdown_connections 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 -- 172.21.15.116:0/344259775 wait complete. 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 -- start start 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 Processor -- start 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac5a7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac5a8000 con 0x55a9ac59f8f0 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac5a4530 con 0x55a9ac596d10 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9ac5a46f0 con 0x55a9ac5963f0 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f9358c6e740 10 client.0 _ll_register_callbacks cb 0x7f9344005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:43.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f934effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f934effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58470/0 (socket says 172.21.15.116:58470) 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f934effd640 1 -- 172.21.15.116:0/3070124310 learned_addr learned my addr 172.21.15.116:0/3070124310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.827+0000 7f934ffff640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934f7fe640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac5a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934effd640 1 -- 172.21.15.116:0/3070124310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 msgr2=0x55a9ac5a7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934effd640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac5a7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934effd640 1 -- 172.21.15.116:0/3070124310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac59f8f0 msgr2=0x55a9ac5a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934effd640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934effd640 1 -- 172.21.15.116:0/3070124310 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9344017050 con 0x55a9ac596d10 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934f7fe640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac5a7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f934effd640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5a3ab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9344000c00 tx=0x7f9344003f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83be819d916762b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f932effd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f932effd640 1 -- 172.21.15.116:0/3070124310 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344022040 con 0x55a9ac596d10 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93400a34a0 con 0x55a9ac596d10 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f932effd640 1 -- 172.21.15.116:0/3070124310 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f934402a250 con 0x55a9ac596d10 2024-03-24T09:22:43.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f932effd640 1 -- 172.21.15.116:0/3070124310 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f934401d8a0 con 0x55a9ac596d10 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f9358c6e740 10 client.25748 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a9ac4bd920 con 0x55a9ac596d10 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.828+0000 7f932effd640 1 -- 172.21.15.116:0/3070124310 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f93400a34a0 con 0x55a9ac596d10 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f932effd640 1 -- 172.21.15.116:0/3070124310 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9344033040 con 0x55a9ac596d10 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f9358c6e740 10 client.25748 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f9358c6e740 10 client.25748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f9358c6e740 20 client.25748 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f9358c6e740 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac618cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f9358c6e740 4 client.25748 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9ac288890 con 0x55a9ac5af510 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.829+0000 7f934f7fe640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac618cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f932effd640 10 client.25748 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f934f7fe640 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac618cf0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b4057d40d8b9f5b server_cookie=3ac9c7ed3fa312c1 in_seq=0 out_seq=0 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f932effd640 1 -- 172.21.15.116:0/3070124310 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9ac288890 con 0x55a9ac5af510 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f932effd640 10 client.25748 handle_command_reply: tid=0 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 client.25748 shutdown 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 msgr2=0x55a9ac5a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5a3ab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9344000c00 tx=0x7f9344003f40 comp rx=0 tx=0).stop 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 shutdown_connections 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9ac59f8f0 0x55a9ac5a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a9ac5af510 0x55a9ac618cf0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f932effd640 0 client.25748 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9ac596d10 0x55a9ac5a3ab0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 --2- 172.21.15.116:0/3070124310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9ac5963f0 0x55a9ac5a7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 >> 172.21.15.116:0/3070124310 conn(0x55a9ac590b20 msgr2=0x55a9ac59d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 shutdown_connections 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 1 -- 172.21.15.116:0/3070124310 wait complete. 2024-03-24T09:22:43.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.830+0000 7f9358c6e740 20 client.25748 trim_cache size 0 max 16384 2024-03-24T09:22:43.836 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:22:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f93380a2340 msgr2=0x7f93380a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f93380a2340 0x7f93380a4800 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f935010f880 tx=0x7f9344005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:43.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 msgr2=0x7f935010ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 0x7f935010ddf0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f934000e9f0 tx=0x7f934000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 shutdown_connections 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f93380a2340 0x7f93380a4800 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9350106660 0x7f935010e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9350100f60 0x7f935010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 --2- 172.21.15.116:0/4221494781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93501005a0 0x7f935010ddf0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 >> 172.21.15.116:0/4221494781 conn(0x7f9350077710 msgr2=0x7f93500ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 shutdown_connections 2024-03-24T09:22:43.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:43.831+0000 7f9356088640 1 -- 172.21.15.116:0/4221494781 wait complete. 2024-03-24T09:22:44.844 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:22:44.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:22:44.844 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:22:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 -- 172.21.15.116:0/544451285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c0692d0 msgr2=0x7f9e5c10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/544451285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c10e570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9e54009c40 tx=0x7f9e5402bed0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 -- 172.21.15.116:0/544451285 shutdown_connections 2024-03-24T09:22:44.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/544451285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c10e570 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/544451285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c068910 0x7f9e5c068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/544451285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c107760 0x7f9e5c107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.951+0000 7f9e63a4d640 1 -- 172.21.15.116:0/544451285 >> 172.21.15.116:0/544451285 conn(0x7f9e5c077710 msgr2=0x7f9e5c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.952+0000 7f9e63a4d640 1 -- 172.21.15.116:0/544451285 shutdown_connections 2024-03-24T09:22:44.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.952+0000 7f9e63a4d640 1 -- 172.21.15.116:0/544451285 wait complete. 2024-03-24T09:22:44.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.952+0000 7f9e63a4d640 1 Processor -- start 2024-03-24T09:22:44.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.952+0000 7f9e63a4d640 1 -- start start 2024-03-24T09:22:44.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e63a4d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c068910 0x7f9e5c19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e63a4d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e63a4d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c107760 0x7f9e5c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e60fe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e60fe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c19ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40532/0 (socket says 172.21.15.116:40532) 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c068910 0x7f9e5c19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c068910 0x7f9e5c19d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:51384/0 (socket says 172.21.15.116:51384) 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e60fe8640 1 -- 172.21.15.116:0/237756972 learned_addr learned my addr 172.21.15.116:0/237756972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c198090 con 0x7f9e5c0692d0 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c198200 con 0x7f9e5c107760 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5c1983c0 con 0x7f9e5c068910 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c107760 0x7f9e5c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 -- 172.21.15.116:0/237756972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c068910 msgr2=0x7f9e5c19d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c068910 0x7f9e5c19d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 -- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c0692d0 msgr2=0x7f9e5c19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 -- 172.21.15.116:0/237756972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4c009560 con 0x7f9e5c107760 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.953+0000 7f9e61fea640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c107760 0x7f9e5c197a40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e5402c410 tx=0x7f9e54003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.954+0000 7f9e527fc640 1 -- 172.21.15.116:0/237756972 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e54033990 con 0x7f9e5c107760 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.954+0000 7f9e527fc640 1 -- 172.21.15.116:0/237756972 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e54004040 con 0x7f9e5c107760 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.954+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e54009840 con 0x7f9e5c107760 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.954+0000 7f9e527fc640 1 -- 172.21.15.116:0/237756972 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e54034690 con 0x7f9e5c107760 2024-03-24T09:22:44.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.954+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5c1a4750 con 0x7f9e5c107760 2024-03-24T09:22:44.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.956+0000 7f9e6661f740 1 Processor -- start 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e527fc640 1 -- 172.21.15.116:0/237756972 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7f9e5402dd50 con 0x7f9e5c107760 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 -- start start 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543c0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559854322530 con 0x5598543b5080 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55985431fa60 con 0x5598543bdc60 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e6661f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55985431b240 con 0x5598543b4760 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e60fe8640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e60fe8640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:51394/0 (socket says 172.21.15.116:51394) 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3419316350 learned_addr learned my addr 172.21.15.116:0/3419316350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:44.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543c0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e617e9640 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.957+0000 7f9e527fc640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e480a2340 0x7f9e480a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3419316350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 msgr2=0x5598543c0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543c0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e61fea640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e480a2340 0x7f9e480a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3419316350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 msgr2=0x5598543bd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e527fc640 1 -- 172.21.15.116:0/237756972 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9e540f0e80 con 0x7f9e5c107760 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bd720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3419316350 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598543198c0 con 0x5598543b4760 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e617e9640 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543b4b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c011a10 tx=0x7f9e4c009970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da762e7b9ff6546c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:44.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e61fea640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e480a2340 0x7f9e480a4800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ee2d0 tx=0x7f9e54009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e477fe640 1 -- 172.21.15.116:0/3419316350 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c011110 con 0x5598543b4760 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e477fe640 1 -- 172.21.15.116:0/3419316350 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e4c0044f0 con 0x5598543b4760 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.958+0000 7f9e477fe640 1 -- 172.21.15.116:0/3419316350 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0047a0 con 0x5598543b4760 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 -- 172.21.15.116:0/3419316350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 msgr2=0x5598543b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543b4b40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c011a10 tx=0x7f9e4c009970 comp rx=0 tx=0).stop 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 -- 172.21.15.116:0/3419316350 shutdown_connections 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543c0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3419316350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543b4b40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 -- 172.21.15.116:0/3419316350 >> 172.21.15.116:0/3419316350 conn(0x5598543aee90 msgr2=0x5598543af270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 -- 172.21.15.116:0/3419316350 shutdown_connections 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.959+0000 7f9e6661f740 1 -- 172.21.15.116:0/3419316350 wait complete. 2024-03-24T09:22:44.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e6661f740 1 -- start start 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e6661f740 1 Processor -- start 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 0x5598543bac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b5080 0x5598543bb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543bb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e60fe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 0x5598543bac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e60fe8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 0x5598543bac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40546/0 (socket says 172.21.15.116:40546) 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b5080 0x5598543bb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e60fe8640 1 -- 172.21.15.116:0/2706975443 learned_addr learned my addr 172.21.15.116:0/2706975443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.960+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598542bff80 con 0x5598543b4760 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559853fee4b0 con 0x5598543bdc60 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5598542a0ad0 con 0x5598543b5080 2024-03-24T09:22:44.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e6661f740 10 client.0 _ll_register_callbacks cb 0x5598542d3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543bb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e617e9640 1 -- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 msgr2=0x5598543bb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e617e9640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543bb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e617e9640 1 -- 172.21.15.116:0/2706975443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 msgr2=0x5598543bac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e617e9640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 0x5598543bac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e617e9640 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e4c019040 con 0x5598543b5080 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 0x5598543bac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e617e9640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b5080 0x5598543bb1d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e580126b0 tx=0x7f9e58012b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1908903e1e4a08e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e44ff9640 1 -- 172.21.15.116:0/2706975443 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58017070 con 0x5598543b5080 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55985433f7a0 con 0x5598543b5080 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.961+0000 7f9e44ff9640 1 -- 172.21.15.116:0/2706975443 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9e58004050 con 0x5598543b5080 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e44ff9640 1 -- 172.21.15.116:0/2706975443 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e58004210 con 0x5598543b5080 2024-03-24T09:22:44.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 10 client.25648 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5598542d16d0 con 0x5598543b5080 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e44ff9640 1 -- 172.21.15.116:0/2706975443 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e5800fdd0 con 0x5598543b5080 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e44ff9640 1 -- 172.21.15.116:0/2706975443 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9e58003af0 con 0x5598543b5080 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 10 client.25648 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 20 client.25648 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598543cdbf0 0x5598543d0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 4 client.25648 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55985433f7a0 con 0x5598543cdbf0 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.962+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598543cdbf0 0x5598543d0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.963+0000 7f9e44ff9640 10 client.25648 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.963+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598543cdbf0 0x5598543d0090 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac97fcb5aa42ab31 server_cookie=af47e8f6f7027d62 in_seq=0 out_seq=0 2024-03-24T09:22:44.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.963+0000 7f9e44ff9640 1 -- 172.21.15.116:0/2706975443 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55985433f7a0 con 0x5598543cdbf0 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.963+0000 7f9e44ff9640 10 client.25648 handle_command_reply: tid=0 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.963+0000 7f9e6661f740 1 client.25648 shutdown 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b5080 msgr2=0x5598543bb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b5080 0x5598543bb1d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e580126b0 tx=0x7f9e58012b70 comp rx=0 tx=0).stop 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 shutdown_connections 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598543cdbf0 0x5598543d0090 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e44ff9640 0 client.25648 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543bb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b5080 0x5598543bb1d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 --2- 172.21.15.116:0/2706975443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b4760 0x5598543bac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 >> 172.21.15.116:0/2706975443 conn(0x5598543aee90 msgr2=0x5598543b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 shutdown_connections 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 1 -- 172.21.15.116:0/2706975443 wait complete. 2024-03-24T09:22:44.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.964+0000 7f9e6661f740 20 client.25648 trim_cache size 0 max 16384 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 Processor -- start 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 -- start start 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543bd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543d59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598543b9200 con 0x5598543b5080 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5598543c0380 con 0x5598543bdc60 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e6661f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5598543cc530 con 0x5598543b4760 2024-03-24T09:22:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40548/0 (socket says 172.21.15.116:40548) 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e617e9640 1 -- 172.21.15.116:0/1106473423 learned_addr learned my addr 172.21.15.116:0/1106473423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543bd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.997+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543d59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e617e9640 1 -- 172.21.15.116:0/1106473423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 msgr2=0x5598543bd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e617e9640 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543bd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e617e9640 1 -- 172.21.15.116:0/1106473423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 msgr2=0x5598543d59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e617e9640 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543d59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e617e9640 1 -- 172.21.15.116:0/1106473423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598542b5c60 con 0x5598543b5080 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543bd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e617e9640 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bb530 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9e58000c00 tx=0x7f9e58010570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6eae1ab7163bd13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e46ffd640 1 -- 172.21.15.116:0/1106473423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58017040 con 0x5598543b5080 2024-03-24T09:22:45.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e46ffd640 1 -- 172.21.15.116:0/1106473423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e580050f0 con 0x5598543b5080 2024-03-24T09:22:45.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.998+0000 7f9e46ffd640 1 -- 172.21.15.116:0/1106473423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58003890 con 0x5598543b5080 2024-03-24T09:22:45.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 -- 172.21.15.116:0/1106473423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 msgr2=0x5598543bb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bb530 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9e58000c00 tx=0x7f9e58010570 comp rx=0 tx=0).stop 2024-03-24T09:22:45.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 -- 172.21.15.116:0/1106473423 shutdown_connections 2024-03-24T09:22:45.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543bdc60 0x5598543d59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543bb530 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 --2- 172.21.15.116:0/1106473423 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543b4760 0x5598543bd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 -- 172.21.15.116:0/1106473423 >> 172.21.15.116:0/1106473423 conn(0x5598543aee90 msgr2=0x5598543b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 -- 172.21.15.116:0/1106473423 shutdown_connections 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 -- 172.21.15.116:0/1106473423 wait complete. 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:44.999+0000 7f9e6661f740 1 -- start start 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 Processor -- start 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543b4760 0x5598543c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543bdc60 0x559854433fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598543c38f0 con 0x5598543b5080 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e617e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543c31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40560/0 (socket says 172.21.15.116:40560) 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e617e9640 1 -- 172.21.15.116:0/3763517398 learned_addr learned my addr 172.21.15.116:0/3763517398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559854434520 con 0x5598543b4760 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543b4760 0x5598543c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559854434660 con 0x5598543bdc60 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e6661f740 10 client.0 _ll_register_callbacks cb 0x7f9e58009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543bdc60 0x559854433fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3763517398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543bdc60 msgr2=0x559854433fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543bdc60 0x559854433fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.000+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3763517398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 msgr2=0x5598543c31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543c31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e60fe8640 1 -- 172.21.15.116:0/3763517398 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598543198c0 con 0x5598543b4760 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e617e9640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543c31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e60fe8640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543b4760 0x5598543c2cb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5598543cdfd0 tx=0x7f9e58009e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9954e1a4c29fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e44ff9640 1 -- 172.21.15.116:0/3763517398 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c031e90 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e44ff9640 1 -- 172.21.15.116:0/3763517398 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9e4c0302f0 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e4c0044f0 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e44ff9640 1 -- 172.21.15.116:0/3763517398 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e4c0304b0 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e6661f740 10 client.25766 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5598542d16d0 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e44ff9640 1 -- 172.21.15.116:0/3763517398 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e4c00fcf0 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e44ff9640 1 -- 172.21.15.116:0/3763517398 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9e4c036570 con 0x5598543b4760 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.001+0000 7f9e6661f740 10 client.25766 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e6661f740 10 client.25766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e6661f740 20 client.25766 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559854435dc0 0x5598544381d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e6661f740 4 client.25766 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f9e4c0044f0 con 0x559854435dc0 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e617e9640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559854435dc0 0x5598544381d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e617e9640 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559854435dc0 0x5598544381d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8466cb565fc61c36 server_cookie=545b94fe3fb8aa32 in_seq=0 out_seq=0 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.002+0000 7f9e44ff9640 10 client.25766 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.003+0000 7f9e44ff9640 1 -- 172.21.15.116:0/3763517398 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f9e4c0044f0 con 0x559854435dc0 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.003+0000 7f9e44ff9640 10 client.25766 handle_command_reply: tid=0 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.003+0000 7f9e6661f740 1 client.25766 shutdown 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.003+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543b4760 msgr2=0x5598543c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.003+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543b4760 0x5598543c2cb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5598543cdfd0 tx=0x7f9e58009e10 comp rx=0 tx=0).stop 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 shutdown_connections 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598543bdc60 0x559854433fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598543b5080 0x5598543c31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559854435dc0 0x5598544381d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e44ff9640 0 client.25766 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:45.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 --2- 172.21.15.116:0/3763517398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598543b4760 0x5598543c2cb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 >> 172.21.15.116:0/3763517398 conn(0x5598543aee90 msgr2=0x5598543b09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 shutdown_connections 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 1 -- 172.21.15.116:0/3763517398 wait complete. 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e6661f740 20 client.25766 trim_cache size 0 max 16384 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "3077530d-a6fb-413a-8ff4-c06e5b0be9f1", 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:22:45.009 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:22:45.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e480a2340 msgr2=0x7f9e480a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e480a2340 0x7f9e480a4800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9e540ee2d0 tx=0x7f9e54009500 comp rx=0 tx=0).stop 2024-03-24T09:22:45.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c107760 msgr2=0x7f9e5c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.004+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c107760 0x7f9e5c197a40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9e5402c410 tx=0x7f9e54003ba0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 shutdown_connections 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e5c107760 0x7f9e5c197a40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e5c0692d0 0x7f9e5c19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e480a2340 0x7f9e480a4800 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 --2- 172.21.15.116:0/237756972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e5c068910 0x7f9e5c19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 >> 172.21.15.116:0/237756972 conn(0x7f9e5c077710 msgr2=0x7f9e5c0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 shutdown_connections 2024-03-24T09:22:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.005+0000 7f9e63a4d640 1 -- 172.21.15.116:0/237756972 wait complete. 2024-03-24T09:22:45.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:22:45.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 -- 172.21.15.116:0/1526398404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc0692d0 msgr2=0x7ff0fc10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 --2- 172.21.15.116:0/1526398404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc10e570 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009d00 tx=0x7ff0f402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 -- 172.21.15.116:0/1526398404 shutdown_connections 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 --2- 172.21.15.116:0/1526398404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc10e570 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 --2- 172.21.15.116:0/1526398404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc068910 0x7ff0fc068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 --2- 172.21.15.116:0/1526398404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 0x7ff0fc107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 -- 172.21.15.116:0/1526398404 >> 172.21.15.116:0/1526398404 conn(0x7ff0fc077710 msgr2=0x7ff0fc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 -- 172.21.15.116:0/1526398404 shutdown_connections 2024-03-24T09:22:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.146+0000 7ff101156640 1 -- 172.21.15.116:0/1526398404 wait complete. 2024-03-24T09:22:45.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 Processor -- start 2024-03-24T09:22:45.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 -- start start 2024-03-24T09:22:45.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 0x7ff0fc19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 0x7ff0fc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc198090 con 0x7ff0fc0692d0 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc198200 con 0x7ff0fc068910 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff101156640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff0fc1983c0 con 0x7ff0fc107760 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 0x7ff0fc19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff0fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff0fb59e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 0x7ff0fc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 0x7ff0fc19d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58510/0 (socket says 172.21.15.116:58510) 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.147+0000 7ff0fad9d640 1 -- 172.21.15.116:0/2735743810 learned_addr learned my addr 172.21.15.116:0/2735743810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fb59e640 1 -- 172.21.15.116:0/2735743810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 msgr2=0x7ff0fc19d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 0x7ff0fc19d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fb59e640 1 -- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc0692d0 msgr2=0x7ff0fc19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fb59e640 1 -- 172.21.15.116:0/2735743810 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f0009490 con 0x7ff0fc107760 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 0x7ff0fc19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 0x7ff0fc197a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0f402c700 tx=0x7ff0f4032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0e3fff640 1 -- 172.21.15.116:0/2735743810 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40325d0 con 0x7ff0fc107760 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0e3fff640 1 -- 172.21.15.116:0/2735743810 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0f4003970 con 0x7ff0fc107760 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f4009900 con 0x7ff0fc107760 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.148+0000 7ff0e3fff640 1 -- 172.21.15.116:0/2735743810 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40045e0 con 0x7ff0fc107760 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.149+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0fc1a4750 con 0x7ff0fc107760 2024-03-24T09:22:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.151+0000 7ff0e3fff640 1 -- 172.21.15.116:0/2735743810 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 24) ==== 136933+0+0 (secure 0 0 0) 0x7ff0f403b040 con 0x7ff0fc107760 2024-03-24T09:22:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.152+0000 7ff0e3fff640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff0e40a22c0 0x7ff0e40a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.152+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff0e40a22c0 0x7ff0e40a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.152+0000 7ff0e3fff640 1 -- 172.21.15.116:0/2735743810 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff0f40f0750 con 0x7ff0fc107760 2024-03-24T09:22:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.152+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff0e40a22c0 0x7ff0e40a4780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc198b80 tx=0x7ff0ec005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.152+0000 7ff103e0d740 1 Processor -- start 2024-03-24T09:22:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.152+0000 7ff103e0d740 1 -- start start 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c34843f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348448190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c34844aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff103e0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c348363f30 con 0x55c34843faf0 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff103e0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c3483ad810 con 0x55c3484486d0 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff103e0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c3483aad40 con 0x55c34843f1d0 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348448190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c34843f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348448190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40594/0 (socket says 172.21.15.116:40594) 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fa59c640 1 -- 172.21.15.116:0/115125042 learned_addr learned my addr 172.21.15.116:0/115125042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c34844aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 -- 172.21.15.116:0/115125042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 msgr2=0x55c34844aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c34844aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 -- 172.21.15.116:0/115125042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 msgr2=0x55c348448190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348448190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 -- 172.21.15.116:0/115125042 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c348367ea0 con 0x55c34843f1d0 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348448190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c34844aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.153+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c34843f5b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009f80 tx=0x7ff0ec014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23874c99814f589d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff0e17fa640 1 -- 172.21.15.116:0/115125042 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec01c7d0 con 0x55c34843f1d0 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff0e17fa640 1 -- 172.21.15.116:0/115125042 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0ec01cde0 con 0x55c34843f1d0 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff0e17fa640 1 -- 172.21.15.116:0/115125042 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec026e80 con 0x55c34843f1d0 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 -- 172.21.15.116:0/115125042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 msgr2=0x55c34843f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c34843f5b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009f80 tx=0x7ff0ec014000 comp rx=0 tx=0).stop 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 -- 172.21.15.116:0/115125042 shutdown_connections 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c34844aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348448190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 --2- 172.21.15.116:0/115125042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c34843f5b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 -- 172.21.15.116:0/115125042 >> 172.21.15.116:0/115125042 conn(0x55c348439900 msgr2=0x55c348439ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 -- 172.21.15.116:0/115125042 shutdown_connections 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.154+0000 7ff103e0d740 1 -- 172.21.15.116:0/115125042 wait complete. 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 -- start start 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 Processor -- start 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c3484457a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843faf0 0x55c348445ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348446220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c3483a6520 con 0x55c34843f1d0 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c348079430 con 0x55c3484486d0 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c348035e00 con 0x55c34843faf0 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c3484457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c3484457a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40602/0 (socket says 172.21.15.116:40602) 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff0fad9d640 1 -- 172.21.15.116:0/2022749403 learned_addr learned my addr 172.21.15.116:0/2022749403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff103e0d740 10 client.0 _ll_register_callbacks cb 0x55c348344d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843faf0 0x55c348445ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.155+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348446220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fad9d640 1 -- 172.21.15.116:0/2022749403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843faf0 msgr2=0x55c348445ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843faf0 0x55c348445ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fad9d640 1 -- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 msgr2=0x55c348446220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348446220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fad9d640 1 -- 172.21.15.116:0/2022749403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ec017050 con 0x55c34843f1d0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348446220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843faf0 0x55c348445ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c3484457a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec029040 tx=0x7ff0ec027c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3255e9860355c9aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0deff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0deff5640 1 -- 172.21.15.116:0/2022749403 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec035070 con 0x55c34843f1d0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0deff5640 1 -- 172.21.15.116:0/2022749403 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff0ec027e20 con 0x55c34843f1d0 2024-03-24T09:22:45.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c3480a57e0 con 0x55c34843f1d0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0deff5640 1 -- 172.21.15.116:0/2022749403 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff0ec005040 con 0x55c34843f1d0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff103e0d740 10 client.15945 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c3483a4ba0 con 0x55c34843f1d0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.156+0000 7ff0deff5640 1 -- 172.21.15.116:0/2022749403 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c3480a57e0 con 0x55c34843f1d0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff0deff5640 1 -- 172.21.15.116:0/2022749403 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff0ec038020 con 0x55c34843f1d0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff103e0d740 10 client.15945 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff103e0d740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff103e0d740 20 client.15945 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff103e0d740 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c34845cf60 0x55c34845f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff103e0d740 4 client.15945 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3480e2720 con 0x55c34845cf60 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c34845cf60 0x55c34845f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c34845cf60 0x55c34845f3a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=791229f4caba4441 server_cookie=3cd6e0050ac01226 in_seq=0 out_seq=0 2024-03-24T09:22:45.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.157+0000 7ff0deff5640 10 client.15945 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:45.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff0deff5640 1 -- 172.21.15.116:0/2022749403 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c3480e2720 con 0x55c34845cf60 2024-03-24T09:22:45.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff0deff5640 10 client.15945 handle_command_reply: tid=0 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 client.15945 shutdown 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 msgr2=0x55c3484457a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c3484457a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec029040 tx=0x7ff0ec027c30 comp rx=0 tx=0).stop 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 shutdown_connections 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c34845cf60 0x55c34845f3a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348446220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843faf0 0x55c348445ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff0deff5640 0 client.15945 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 --2- 172.21.15.116:0/2022749403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c3484457a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 >> 172.21.15.116:0/2022749403 conn(0x55c348439900 msgr2=0x55c34843b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 shutdown_connections 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 1 -- 172.21.15.116:0/2022749403 wait complete. 2024-03-24T09:22:45.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.158+0000 7ff103e0d740 20 client.15945 trim_cache size 0 max 16384 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.190+0000 7ff103e0d740 1 Processor -- start 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.190+0000 7ff103e0d740 1 -- start start 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c348447ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348439ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348453f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff103e0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c34832b430 con 0x55c34843faf0 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff103e0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c348435240 con 0x55c3484486d0 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff103e0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c3484544a0 con 0x55c34843f1d0 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348439ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348439ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40610/0 (socket says 172.21.15.116:40610) 2024-03-24T09:22:45.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fa59c640 1 -- 172.21.15.116:0/181223698 learned_addr learned my addr 172.21.15.116:0/181223698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c348447ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348453f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 -- 172.21.15.116:0/181223698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 msgr2=0x55c348453f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348453f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 -- 172.21.15.116:0/181223698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 msgr2=0x55c348439ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348439ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 -- 172.21.15.116:0/181223698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c348340b20 con 0x55c34843f1d0 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348439ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.191+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c348447ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec004900 tx=0x7ff0ec027b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=842ae4992762b257 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.192+0000 7ff0e0ff9640 1 -- 172.21.15.116:0/181223698 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec035070 con 0x55c34843f1d0 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.192+0000 7ff0e0ff9640 1 -- 172.21.15.116:0/181223698 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0ec009e10 con 0x55c34843f1d0 2024-03-24T09:22:45.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.192+0000 7ff0e0ff9640 1 -- 172.21.15.116:0/181223698 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec026ea0 con 0x55c34843f1d0 2024-03-24T09:22:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 -- 172.21.15.116:0/181223698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 msgr2=0x55c348447ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c348447ed0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec004900 tx=0x7ff0ec027b50 comp rx=0 tx=0).stop 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 -- 172.21.15.116:0/181223698 shutdown_connections 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3484486d0 0x55c348453f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843faf0 0x55c348439ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 --2- 172.21.15.116:0/181223698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c34843f1d0 0x55c348447ed0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 -- 172.21.15.116:0/181223698 >> 172.21.15.116:0/181223698 conn(0x55c348439900 msgr2=0x55c3484382d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 -- 172.21.15.116:0/181223698 shutdown_connections 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 -- 172.21.15.116:0/181223698 wait complete. 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 -- start start 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 Processor -- start 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c34844bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.193+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c34843faf0 0x55c34844c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff103e0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3484486d0 0x55c34844c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff103e0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c348450db0 con 0x55c34843f1d0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff103e0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c3484c22e0 con 0x55c34843faf0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff103e0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c3484c2450 con 0x55c3484486d0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c34844bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c34844bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40618/0 (socket says 172.21.15.116:40618) 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 -- 172.21.15.116:0/4066896404 learned_addr learned my addr 172.21.15.116:0/4066896404 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fb59e640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3484486d0 0x55c34844c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff103e0d740 10 client.0 _ll_register_callbacks cb 0x7ff0ec029070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c34843faf0 0x55c34844c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 -- 172.21.15.116:0/4066896404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3484486d0 msgr2=0x55c34844c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3484486d0 0x55c34844c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 -- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c34843faf0 msgr2=0x55c34844c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c34843faf0 0x55c34844c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 -- 172.21.15.116:0/4066896404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f00128a0 con 0x55c34843f1d0 2024-03-24T09:22:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0fad9d640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c34844bde0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00f6a0 tx=0x7ff0ec004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecb552bf35e9125 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0deff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0deff5640 1 -- 172.21.15.116:0/4066896404 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ec035040 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff0deff5640 1 -- 172.21.15.116:0/4066896404 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff0ec01c310 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.194+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0ec01cde0 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff0deff5640 1 -- 172.21.15.116:0/4066896404 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff0ec026a50 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 10 client.15951 fetch_fsmap learned FSMap version 29 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c3483a4ba0 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff0deff5640 1 -- 172.21.15.116:0/4066896404 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff0ec01cde0 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff0deff5640 1 -- 172.21.15.116:0/4066896404 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff0ec038040 con 0x55c34843f1d0 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 10 client.15951 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 20 client.15951 populate_metadata read hostname 'smithi116' 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c3484c37b0 0x55c3484c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 4 client.15951 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:22:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c348131890 con 0x55c3484c37b0 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.195+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c3484c37b0 0x55c3484c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff0fa59c640 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c3484c37b0 0x55c3484c5ba0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8515af3210ed8e82 server_cookie=82ea6272c8235e6d in_seq=0 out_seq=0 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff0deff5640 10 client.15951 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff0deff5640 1 -- 172.21.15.116:0/4066896404 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c348131890 con 0x55c3484c37b0 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff0deff5640 10 client.15951 handle_command_reply: tid=0 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 client.15951 shutdown 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 msgr2=0x55c34844bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c34844bde0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec00f6a0 tx=0x7ff0ec004c30 comp rx=0 tx=0).stop 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 shutdown_connections 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3484486d0 0x55c34844c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c3484c37b0 0x55c3484c5ba0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff0deff5640 0 client.15951 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c34843faf0 0x55c34844c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 --2- 172.21.15.116:0/4066896404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c34843f1d0 0x55c34844bde0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 >> 172.21.15.116:0/4066896404 conn(0x55c348439900 msgr2=0x55c348438b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 shutdown_connections 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 1 -- 172.21.15.116:0/4066896404 wait complete. 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.196+0000 7ff103e0d740 20 client.15951 trim_cache size 0 max 16384 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout: "3077530d-a6fb-413a-8ff4-c06e5b0be9f1": { 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "3077530d-a6fb-413a-8ff4-c06e5b0be9f1", 2024-03-24T09:22:45.202 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:22:45.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:22:45.203 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:22:45.203 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:22:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff0e40a22c0 msgr2=0x7ff0e40a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff0e40a22c0 0x7ff0e40a4780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc198b80 tx=0x7ff0ec005f90 comp rx=0 tx=0).stop 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 msgr2=0x7ff0fc197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 0x7ff0fc197a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff0f402c700 tx=0x7ff0f4032f20 comp rx=0 tx=0).stop 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 shutdown_connections 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff0e40a22c0 0x7ff0e40a4780 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff0fc107760 0x7ff0fc197a40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0fc0692d0 0x7ff0fc19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 --2- 172.21.15.116:0/2735743810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0fc068910 0x7ff0fc19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 >> 172.21.15.116:0/2735743810 conn(0x7ff0fc077710 msgr2=0x7ff0fc0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 shutdown_connections 2024-03-24T09:22:45.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:22:45.197+0000 7ff101156640 1 -- 172.21.15.116:0/2735743810 wait complete. 2024-03-24T09:22:45.208 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:45 smithi116 ceph-mon[32207]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-24T09:22:45.209 INFO:tasks.cephfs.filesystem:scrub status for tag:3077530d-a6fb-413a-8ff4-c06e5b0be9f1 - {'path': '/', 'tag': '3077530d-a6fb-413a-8ff4-c06e5b0be9f1', 'options': 'recursive,force'} 2024-03-24T09:22:45.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:45 smithi143 ceph-mon[35965]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-24T09:22:45.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:45 smithi117 ceph-mon[35788]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 12 op/s 2024-03-24T09:22:46.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:46 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:22:46.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:46 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:46.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:46 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:22:46.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:46 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:46.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:46 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:22:46.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:46 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:46.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:46 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:22:46.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:46 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:46.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:46 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:22:46.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:46 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:46.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:46 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:22:46.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:46 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:22:47.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:47 smithi117 ceph-mon[35788]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:22:47.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:47 smithi143 ceph-mon[35965]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:22:47.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:47 smithi116 ceph-mon[32207]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:22:48.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:48 smithi143 ceph-mon[35965]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:22:48.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:48 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:22:48.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:48 smithi117 ceph-mon[35788]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:22:48.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:48 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:22:48.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:48 smithi116 ceph-mon[32207]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:22:48.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:48 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:22:50.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:50 smithi143 ceph-mon[35965]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:22:50.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:50 smithi117 ceph-mon[35788]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:22:50.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:50 smithi116 ceph-mon[32207]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:22:52.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:52 smithi143 ceph-mon[35965]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-03-24T09:22:52.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:52 smithi117 ceph-mon[35788]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-03-24T09:22:52.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:52 smithi116 ceph-mon[32207]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 11 op/s 2024-03-24T09:22:54.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:54 smithi143 ceph-mon[35965]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-24T09:22:54.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:54 smithi117 ceph-mon[35788]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-24T09:22:54.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:54 smithi116 ceph-mon[32207]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-24T09:22:56.515 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'. 2024-03-24T09:22:56.515 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.515 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T09:22:56.515 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.516 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-03-24T09:22:56.517 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.517 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T09:22:56.517 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:22:56.517 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at d39ab7bb1f1 qa/suites/fs/workload: enable snap_schedule early 2024-03-24T09:22:56.520 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T09:22:56.547 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T09:22:56.549 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T09:22:56.549 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T09:22:56.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:56 smithi143 ceph-mon[35965]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:22:56.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:56 smithi117 ceph-mon[35788]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:22:56.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:56 smithi116 ceph-mon[32207]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:22:57.337 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T09:22:57.406 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T09:22:57.447 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T09:22:57.449 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T09:22:57.449 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T09:22:57.487 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T09:22:57.491 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:22:57.491 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T09:22:57.547 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-24T09:22:57.548 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-24T09:22:57.549 DEBUG:teuthology.orchestra.run.smithi116:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-24T09:22:57.611 INFO:tasks.workunit.client.0.smithi116.stderr:+ set -e 2024-03-24T09:22:57.611 INFO:tasks.workunit.client.0.smithi116.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-24T09:22:57.614 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into 'xfstests-dev'... 2024-03-24T09:22:58.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:22:58 smithi143 ceph-mon[35965]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:22:58.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:22:58 smithi117 ceph-mon[35788]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:22:58.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:22:58 smithi116 ceph-mon[32207]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:23:00.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:23:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:23:00.054+0000 7fc24ad9b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:23:00.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:23:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:23:00.054+0000 7fc24ad9b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:23:00.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:23:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:23:00.054+0000 7fc24ad9b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:23:00.318 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:23:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:23:00.054+0000 7fc24ad9b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:23:00.319 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:23:00 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x[32440]: 2024-03-24T09:23:00.054+0000 7fc24ad9b700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T09:23:00.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:00 smithi143 ceph-mon[35965]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:23:00.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:00 smithi117 ceph-mon[35788]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:23:00.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:00 smithi116 ceph-mon[32207]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:23:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:02 smithi143 ceph-mon[35965]: pgmap v31: 129 pgs: 129 active+clean; 12 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 944 KiB/s wr, 41 op/s 2024-03-24T09:23:02.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:02 smithi117 ceph-mon[35788]: pgmap v31: 129 pgs: 129 active+clean; 12 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 944 KiB/s wr, 41 op/s 2024-03-24T09:23:02.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:02 smithi116 ceph-mon[32207]: pgmap v31: 129 pgs: 129 active+clean; 12 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 944 KiB/s wr, 41 op/s 2024-03-24T09:23:03.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:03 smithi143 ceph-mon[35965]: mgrmap e25: x(active, since 57s), standbys: z, y 2024-03-24T09:23:03.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:03 smithi117 ceph-mon[35788]: mgrmap e25: x(active, since 57s), standbys: z, y 2024-03-24T09:23:03.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:03 smithi116 ceph-mon[32207]: mgrmap e25: x(active, since 57s), standbys: z, y 2024-03-24T09:23:04.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:04 smithi143 ceph-mon[35965]: pgmap v32: 129 pgs: 129 active+clean; 16 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-24T09:23:04.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:04 smithi117 ceph-mon[35788]: pgmap v32: 129 pgs: 129 active+clean; 16 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-24T09:23:04.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:04 smithi116 ceph-mon[32207]: pgmap v32: 129 pgs: 129 active+clean; 16 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.2 MiB/s wr, 48 op/s 2024-03-24T09:23:06.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:06 smithi143 ceph-mon[35965]: pgmap v33: 129 pgs: 129 active+clean; 16 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-03-24T09:23:06.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:06 smithi117 ceph-mon[35788]: pgmap v33: 129 pgs: 129 active+clean; 16 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-03-24T09:23:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:06 smithi116 ceph-mon[32207]: pgmap v33: 129 pgs: 129 active+clean; 16 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 1.2 MiB/s wr, 46 op/s 2024-03-24T09:23:08.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:08 smithi143 ceph-mon[35965]: pgmap v34: 129 pgs: 129 active+clean; 18 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-24T09:23:08.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:08 smithi117 ceph-mon[35788]: pgmap v34: 129 pgs: 129 active+clean; 18 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-24T09:23:08.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:08 smithi116 ceph-mon[32207]: pgmap v34: 129 pgs: 129 active+clean; 18 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-03-24T09:23:10.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:10 smithi143 ceph-mon[35965]: pgmap v35: 129 pgs: 129 active+clean; 18 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-03-24T09:23:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:10 smithi117 ceph-mon[35788]: pgmap v35: 129 pgs: 129 active+clean; 18 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-03-24T09:23:10.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:10 smithi116 ceph-mon[32207]: pgmap v35: 129 pgs: 129 active+clean; 18 MiB data, 447 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-03-24T09:23:11.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:23:11.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:11 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:23:11.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:11 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:23:11.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:11.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:11 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:12.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:12 smithi143 ceph-mon[35965]: pgmap v36: 129 pgs: 129 active+clean; 22 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-03-24T09:23:12.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:12 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:23:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:12 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:23:12.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:12 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:12.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:12 smithi117 ceph-mon[35788]: pgmap v36: 129 pgs: 129 active+clean; 22 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-03-24T09:23:12.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:12 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:23:12.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:12 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:23:12.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:12 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:12.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:12 smithi116 ceph-mon[32207]: pgmap v36: 129 pgs: 129 active+clean; 22 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1.9 MiB/s wr, 148 op/s 2024-03-24T09:23:12.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:12 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:23:12.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:12 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:23:12.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:12 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:23:14.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:14 smithi143 ceph-mon[35965]: pgmap v37: 129 pgs: 129 active+clean; 22 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 989 KiB/s wr, 128 op/s 2024-03-24T09:23:14.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:14 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:14.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:14 smithi117 ceph-mon[35788]: pgmap v37: 129 pgs: 129 active+clean; 22 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 989 KiB/s wr, 128 op/s 2024-03-24T09:23:14.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:14 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:14.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:14 smithi116 ceph-mon[32207]: pgmap v37: 129 pgs: 129 active+clean; 22 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 989 KiB/s wr, 128 op/s 2024-03-24T09:23:14.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:14 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:15.211 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:23:15.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 -- 172.21.15.116:0/1025606404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b81005b0 msgr2=0x7fd5b810e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 --2- 172.21.15.116:0/1025606404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b81005b0 0x7fd5b810e590 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009cd0 tx=0x7fd5b002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 -- 172.21.15.116:0/1025606404 shutdown_connections 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 --2- 172.21.15.116:0/1025606404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b81005b0 0x7fd5b810e590 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 --2- 172.21.15.116:0/1025606404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 --2- 172.21.15.116:0/1025606404 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5b81076d0 0x7fd5b8107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 -- 172.21.15.116:0/1025606404 >> 172.21.15.116:0/1025606404 conn(0x7fd5b8076bd0 msgr2=0x7fd5b80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 -- 172.21.15.116:0/1025606404 shutdown_connections 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 -- 172.21.15.116:0/1025606404 wait complete. 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.317+0000 7fd5be22a640 1 Processor -- start 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 -- start start 2024-03-24T09:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b81005b0 0x7fd5b8101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5b81076d0 0x7fd5b8101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81a5fb0 con 0x7fd5b80ffc10 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81a6120 con 0x7fd5b81005b0 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5be22a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b81a62e0 con 0x7fd5b81076d0 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53968/0 (socket says 172.21.15.116:53968) 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 -- 172.21.15.116:0/4291836065 learned_addr learned my addr 172.21.15.116:0/4291836065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b81005b0 0x7fd5b8101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5b81076d0 0x7fd5b8101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 -- 172.21.15.116:0/4291836065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5b81076d0 msgr2=0x7fd5b8101870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5b81076d0 0x7fd5b8101870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 -- 172.21.15.116:0/4291836065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b81005b0 msgr2=0x7fd5b8101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b81005b0 0x7fd5b8101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 -- 172.21.15.116:0/4291836065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b0009900 con 0x7fd5b80ffc10 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b81005b0 0x7fd5b8101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.318+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100df0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800e9f0 tx=0x7fd5a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.319+0000 7fd5b4ff9640 1 -- 172.21.15.116:0/4291836065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8009720 con 0x7fd5b80ffc10 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.319+0000 7fd5b4ff9640 1 -- 172.21.15.116:0/4291836065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5a800cd80 con 0x7fd5b80ffc10 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.319+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b81a65c0 con 0x7fd5b80ffc10 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.319+0000 7fd5b4ff9640 1 -- 172.21.15.116:0/4291836065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8010430 con 0x7fd5b80ffc10 2024-03-24T09:23:15.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.319+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b81a6a80 con 0x7fd5b80ffc10 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b4ff9640 1 -- 172.21.15.116:0/4291836065 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7fd5a8005020 con 0x7fd5b80ffc10 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 Processor -- start 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 -- start start 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b4ff9640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5a00a2380 0x7fd5a00a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804985b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804a1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b803bcf30 con 0x555b80498af0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b80406810 con 0x555b804a16d0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555b80403d40 con 0x555b804981d0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b4ff9640 1 -- 172.21.15.116:0/4291836065 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd5a8014070 con 0x7fd5b80ffc10 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5a00a2380 0x7fd5a00a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45814/0 (socket says 172.21.15.116:45814) 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.321+0000 7fd5b77fe640 1 -- 172.21.15.116:0/2218609961 learned_addr learned my addr 172.21.15.116:0/2218609961 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/2218609961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 msgr2=0x555b804985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/2218609961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 msgr2=0x555b804a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/2218609961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b803c0ea0 con 0x555b80498af0 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5a00a2380 0x7fd5a00a4840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0005cc0 tx=0x7fd5b0032fc0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804a1190 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009fa0 tx=0x7fd5ac0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbb5648f550b970 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd59dffb640 1 -- 172.21.15.116:0/2218609961 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac004030 con 0x555b80498af0 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd59dffb640 1 -- 172.21.15.116:0/2218609961 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5ac005020 con 0x555b80498af0 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.322+0000 7fd59dffb640 1 -- 172.21.15.116:0/2218609961 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac004b10 con 0x555b80498af0 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/2218609961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 msgr2=0x555b804a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804a1190 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009fa0 tx=0x7fd5ac0118d0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/2218609961 shutdown_connections 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804a1190 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/2218609961 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/2218609961 >> 172.21.15.116:0/2218609961 conn(0x555b80492900 msgr2=0x555b80492ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/2218609961 shutdown_connections 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/2218609961 wait complete. 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 -- start start 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.323+0000 7fd5c0e0a740 1 Processor -- start 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804981d0 0x555b8049e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b8049ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804a16d0 0x555b8049f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b803ff520 con 0x555b80498af0 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b800d2430 con 0x555b804981d0 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555b8008ee00 con 0x555b804a16d0 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 10 client.0 _ll_register_callbacks cb 0x555b8039dd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b8049ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b8049ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53980/0 (socket says 172.21.15.116:53980) 2024-03-24T09:23:15.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1922651689 learned_addr learned my addr 172.21.15.116:0/1922651689 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804981d0 0x555b8049e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804a16d0 0x555b8049f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1922651689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804a16d0 msgr2=0x555b8049f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804a16d0 0x555b8049f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804981d0 msgr2=0x555b8049e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804981d0 0x555b8049e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1922651689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ac009c00 con 0x555b80498af0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b8049ec10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009560 tx=0x7fd5ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82cfdffe2fec9d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd596ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd596ffd640 1 -- 172.21.15.116:0/1922651689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac005590 con 0x555b80498af0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd596ffd640 1 -- 172.21.15.116:0/1922651689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd5ac004460 con 0x555b80498af0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b800fe7e0 con 0x555b80498af0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd596ffd640 1 -- 172.21.15.116:0/1922651689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd5ac004a20 con 0x555b80498af0 2024-03-24T09:23:15.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 10 client.15969 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.324+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555b803fdba0 con 0x555b80498af0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd596ffd640 1 -- 172.21.15.116:0/1922651689 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555b800fe7e0 con 0x555b80498af0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd596ffd640 1 -- 172.21.15.116:0/1922651689 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7fd5ac00f5a0 con 0x555b80498af0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5c0e0a740 10 client.15969 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5c0e0a740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5c0e0a740 20 client.15969 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b804b19d0 0x555b804b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5c0e0a740 4 client.15969 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b8013b720 con 0x555b804b19d0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b804b19d0 0x555b804b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd596ffd640 10 client.15969 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.325+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b804b19d0 0x555b804b3e10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8762bb3aa883660 server_cookie=737a8a4f16342688 in_seq=0 out_seq=0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.327+0000 7fd596ffd640 1 -- 172.21.15.116:0/1922651689 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555b8013b720 con 0x555b804b19d0 2024-03-24T09:23:15.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.327+0000 7fd596ffd640 10 client.15969 handle_command_reply: tid=0 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.327+0000 7fd5c0e0a740 1 client.15969 shutdown 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.327+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 msgr2=0x555b8049ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.327+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b8049ec10 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac009560 tx=0x7fd5ac0042f0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 shutdown_connections 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804a16d0 0x555b8049f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b8049ec10 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b804b19d0 0x555b804b3e10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1922651689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804981d0 0x555b8049e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd596ffd640 0 client.15969 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 >> 172.21.15.116:0/1922651689 conn(0x555b80492900 msgr2=0x555b80494650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 shutdown_connections 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1922651689 wait complete. 2024-03-24T09:23:15.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.328+0000 7fd5c0e0a740 20 client.15969 trim_cache size 0 max 16384 2024-03-24T09:23:15.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.344+0000 7fd5c0e0a740 1 Processor -- start 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.344+0000 7fd5c0e0a740 1 -- start start 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804acf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804b9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b80384430 con 0x555b80498af0 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b80497970 con 0x555b804a16d0 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555b80497c10 con 0x555b804981d0 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804acf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804acf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53988/0 (socket says 172.21.15.116:53988) 2024-03-24T09:23:15.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1412545421 learned_addr learned my addr 172.21.15.116:0/1412545421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804b9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1412545421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 msgr2=0x555b804a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1412545421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 msgr2=0x555b804b9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804b9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 -- 172.21.15.116:0/1412545421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b80399b20 con 0x555b80498af0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804acf60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00fb20 tx=0x7fd5ac005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6945b3f31cb7af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd59d7fa640 1 -- 172.21.15.116:0/1412545421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac00f4e0 con 0x555b80498af0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd59d7fa640 1 -- 172.21.15.116:0/1412545421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5ac00f640 con 0x555b80498af0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.345+0000 7fd59d7fa640 1 -- 172.21.15.116:0/1412545421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac004e10 con 0x555b80498af0 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1412545421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 msgr2=0x555b804acf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804acf60 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00fb20 tx=0x7fd5ac005850 comp rx=0 tx=0).stop 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1412545421 shutdown_connections 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804b9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b80498af0 0x555b804acf60 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/1412545421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b804981d0 0x555b804a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1412545421 >> 172.21.15.116:0/1412545421 conn(0x555b80492900 msgr2=0x555b80491b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1412545421 shutdown_connections 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/1412545421 wait complete. 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.346+0000 7fd5c0e0a740 1 -- start start 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 Processor -- start 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 0x555b804a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b80498af0 0x555b804a5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555b80516e20 con 0x555b804981d0 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555b80516f90 con 0x555b804a16d0 2024-03-24T09:23:15.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555b80517150 con 0x555b80498af0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 10 client.0 _ll_register_callbacks cb 0x7fd5a80049c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 0x555b804a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 0x555b804a50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53990/0 (socket says 172.21.15.116:53990) 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 -- 172.21.15.116:0/3041249002 learned_addr learned my addr 172.21.15.116:0/3041249002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b7fff640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b80498af0 0x555b804a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 -- 172.21.15.116:0/3041249002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b80498af0 msgr2=0x555b804a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b80498af0 0x555b804a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 -- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 msgr2=0x555b804a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 -- 172.21.15.116:0/3041249002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b803fdba0 con 0x555b804981d0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5b77fe640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 0x555b804a50c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800c6a0 tx=0x7fd5a80047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da991f938b6eca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd596ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd596ffd640 1 -- 172.21.15.116:0/3041249002 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5a809ef40 con 0x555b804981d0 2024-03-24T09:23:15.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd596ffd640 1 -- 172.21.15.116:0/3041249002 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd5a80aee40 con 0x555b804981d0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.347+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5ac005020 con 0x555b804981d0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd596ffd640 1 -- 172.21.15.116:0/3041249002 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fd5a80a3590 con 0x555b804981d0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 10 client.15975 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555b803bf700 con 0x555b804981d0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd596ffd640 1 -- 172.21.15.116:0/3041249002 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5a8054030 con 0x555b804981d0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd596ffd640 1 -- 172.21.15.116:0/3041249002 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7fd5a801f060 con 0x555b804981d0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 10 client.15975 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 10 client.15975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 20 client.15975 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b80518990 0x555b8051adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 4 client.15975 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555b8018a890 con 0x555b80518990 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b80518990 0x555b8051adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd5b6ffd640 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b80518990 0x555b8051adb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fedf008cc9c22754 server_cookie=3f521df3d31148c8 in_seq=0 out_seq=0 2024-03-24T09:23:15.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.348+0000 7fd596ffd640 10 client.15975 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.350+0000 7fd596ffd640 1 -- 172.21.15.116:0/3041249002 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555b8018a890 con 0x555b80518990 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.350+0000 7fd596ffd640 10 client.15975 handle_command_reply: tid=0 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.350+0000 7fd5c0e0a740 1 client.15975 shutdown 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.350+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 msgr2=0x555b804a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.350+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 0x555b804a50c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800c6a0 tx=0x7fd5a80047f0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 shutdown_connections 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555b80518990 0x555b8051adb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b804a16d0 0x555b804a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555b80498af0 0x555b804a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd596ffd640 0 client.15975 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 --2- 172.21.15.116:0/3041249002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555b804981d0 0x555b804a50c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 >> 172.21.15.116:0/3041249002 conn(0x555b80492900 msgr2=0x555b8049e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 shutdown_connections 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 1 -- 172.21.15.116:0/3041249002 wait complete. 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5c0e0a740 20 client.15975 trim_cache size 0 max 16384 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:23:15.357 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:23:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5a00a2380 msgr2=0x7fd5a00a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5a00a2380 0x7fd5a00a4840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0005cc0 tx=0x7fd5b0032fc0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 msgr2=0x7fd5b8100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100df0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd5a800e9f0 tx=0x7fd5a800edf0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 shutdown_connections 2024-03-24T09:23:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5a00a2380 0x7fd5a00a4840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5b81076d0 0x7fd5b8101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5b81005b0 0x7fd5b8101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 --2- 172.21.15.116:0/4291836065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5b80ffc10 0x7fd5b8100df0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 >> 172.21.15.116:0/4291836065 conn(0x7fd5b8076bd0 msgr2=0x7fd5b80fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 shutdown_connections 2024-03-24T09:23:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.351+0000 7fd5be22a640 1 -- 172.21.15.116:0/4291836065 wait complete. 2024-03-24T09:23:15.362 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:23:15.362 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 -- 172.21.15.116:0/517497684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4102060 msgr2=0x7f32e410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 --2- 172.21.15.116:0/517497684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4102060 0x7f32e410e570 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f32dc009cd0 tx=0x7f32dc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 -- 172.21.15.116:0/517497684 shutdown_connections 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 --2- 172.21.15.116:0/517497684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e4102060 0x7f32e410e570 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 --2- 172.21.15.116:0/517497684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41016a0 0x7f32e4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 --2- 172.21.15.116:0/517497684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e41076b0 0x7f32e4107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 -- 172.21.15.116:0/517497684 >> 172.21.15.116:0/517497684 conn(0x7f32e4078b10 msgr2=0x7f32e4078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 -- 172.21.15.116:0/517497684 shutdown_connections 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 -- 172.21.15.116:0/517497684 wait complete. 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.479+0000 7f32e944e640 1 Processor -- start 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 -- start start 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 0x7f32e419d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4102060 0x7f32e419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41076b0 0x7f32e4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32e41980a0 con 0x7f32e41016a0 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f32e4198210 con 0x7f32e4102060 2024-03-24T09:23:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e944e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f32e41983d0 con 0x7f32e41076b0 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 0x7f32e419d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 0x7f32e419d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54004/0 (socket says 172.21.15.116:54004) 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 -- 172.21.15.116:0/3121486867 learned_addr learned my addr 172.21.15.116:0/3121486867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32da7fc640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4102060 0x7f32e419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e37fe640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41076b0 0x7f32e4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 -- 172.21.15.116:0/3121486867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41076b0 msgr2=0x7f32e4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41076b0 0x7f32e4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 -- 172.21.15.116:0/3121486867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4102060 msgr2=0x7f32e419dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4102060 0x7f32e419dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 -- 172.21.15.116:0/3121486867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32dc009900 con 0x7f32e41016a0 2024-03-24T09:23:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e37fe640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41076b0 0x7f32e4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.480+0000 7f32e2ffd640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 0x7f32e419d8a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f32d400e9f0 tx=0x7f32d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.481+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4009700 con 0x7f32e41016a0 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.481+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e41986b0 con 0x7f32e41016a0 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.481+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32d400cd80 con 0x7f32e41016a0 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.481+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4018840 con 0x7f32e41016a0 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.481+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e41a4940 con 0x7f32e41016a0 2024-03-24T09:23:15.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.482+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e4067c70 con 0x7f32e41016a0 2024-03-24T09:23:15.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.483+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f32d40189a0 con 0x7f32e41016a0 2024-03-24T09:23:15.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.484+0000 7f32e0ff9640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32d00a2350 0x7f32d00a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.484+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f32d4014070 con 0x7f32e41016a0 2024-03-24T09:23:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.484+0000 7f32da7fc640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32d00a2350 0x7f32d00a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.484+0000 7f32da7fc640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32d00a2350 0x7f32d00a4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32e4103940 tx=0x7f32cc005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.489+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f32d4097530 con 0x7f32e41016a0 2024-03-24T09:23:15.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.737+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32e4198f80 con 0x7f32e41016a0 2024-03-24T09:23:15.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.738+0000 7f32e0ff9640 1 -- 172.21.15.116:0/3121486867 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11954 (secure 0 0 0) 0x7f32d409abf0 con 0x7f32e41016a0 2024-03-24T09:23:15.739 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:23:15.739 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:12.964404+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24515,24484,24451,24422,24443]},"id":1}]} 2024-03-24T09:23:15.739 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-03-24T09:23:15.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32d00a2350 msgr2=0x7f32d00a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32d00a2350 0x7f32d00a4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f32e4103940 tx=0x7f32cc005f90 comp rx=0 tx=0).stop 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 msgr2=0x7f32e419d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 0x7f32e419d8a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f32d400e9f0 tx=0x7f32d400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 shutdown_connections 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32d00a2350 0x7f32d00a4810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32e41076b0 0x7f32e4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4102060 0x7f32e419dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 --2- 172.21.15.116:0/3121486867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32e41016a0 0x7f32e419d8a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 >> 172.21.15.116:0/3121486867 conn(0x7f32e4078b10 msgr2=0x7f32e41040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 shutdown_connections 2024-03-24T09:23:15.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.741+0000 7f32e944e640 1 -- 172.21.15.116:0/3121486867 wait complete. 2024-03-24T09:23:15.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:23:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 -- 172.21.15.116:0/3178572200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 msgr2=0x7f092c10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 --2- 172.21.15.116:0/3178572200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c10d420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0924009ca0 tx=0x7f092402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 -- 172.21.15.116:0/3178572200 shutdown_connections 2024-03-24T09:23:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 --2- 172.21.15.116:0/3178572200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c10d420 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 --2- 172.21.15.116:0/3178572200 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c1005a0 0x7f092c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 --2- 172.21.15.116:0/3178572200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c106660 0x7f092c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 -- 172.21.15.116:0/3178572200 >> 172.21.15.116:0/3178572200 conn(0x7f092c077710 msgr2=0x7f092c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 -- 172.21.15.116:0/3178572200 shutdown_connections 2024-03-24T09:23:15.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.879+0000 7f093104e640 1 -- 172.21.15.116:0/3178572200 wait complete. 2024-03-24T09:23:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 Processor -- start 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 -- start start 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 0x7f092c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c106660 0x7f092c1a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f092c199730 con 0x7f092c100f60 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f092c1a3f50 con 0x7f092c1005a0 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f093104e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f092c1a4090 con 0x7f092c106660 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f092a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f092ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 0x7f092c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f092ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 0x7f092c198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:54084/0 (socket says 172.21.15.116:54084) 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f092ad9d640 1 -- 172.21.15.116:0/454172261 learned_addr learned my addr 172.21.15.116:0/454172261 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f092a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c198ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54014/0 (socket says 172.21.15.116:54014) 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.880+0000 7f092b59e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c106660 0x7f092c1a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092ad9d640 1 -- 172.21.15.116:0/454172261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c106660 msgr2=0x7f092c1a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092ad9d640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c106660 0x7f092c1a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092ad9d640 1 -- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 msgr2=0x7f092c198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092ad9d640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092ad9d640 1 -- 172.21.15.116:0/454172261 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0924009900 con 0x7f092c1005a0 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092a59c640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092b59e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c106660 0x7f092c1a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f092ad9d640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 0x7f092c198990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f091c00e9f0 tx=0x7f091c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f0913fff640 1 -- 172.21.15.116:0/454172261 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c009700 con 0x7f092c1005a0 2024-03-24T09:23:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f092c1a42f0 con 0x7f092c1005a0 2024-03-24T09:23:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f0913fff640 1 -- 172.21.15.116:0/454172261 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f091c00cd80 con 0x7f092c1005a0 2024-03-24T09:23:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.881+0000 7f0913fff640 1 -- 172.21.15.116:0/454172261 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091c018840 con 0x7f092c1005a0 2024-03-24T09:23:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.882+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f092c1a4850 con 0x7f092c1005a0 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0913fff640 1 -- 172.21.15.116:0/454172261 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f091c0189a0 con 0x7f092c1005a0 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 Processor -- start 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 -- start start 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e657d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e70ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4dd3840 con 0x55e7f4e653f0 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4dd0d70 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.884+0000 7f0933c2e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4dcc550 con 0x55e7f4e65d10 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e657d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e657d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54016/0 (socket says 172.21.15.116:54016) 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092b59e640 1 -- 172.21.15.116:0/3244956332 learned_addr learned my addr 172.21.15.116:0/3244956332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e70ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f0913fff640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f09140a2350 0x7f09140a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 -- 172.21.15.116:0/3244956332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 msgr2=0x55e7f4e6e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 -- 172.21.15.116:0/3244956332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 msgr2=0x55e7f4e657d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e657d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 -- 172.21.15.116:0/3244956332 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7f4d61de0 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092b59e640 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e657d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092b59e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f09140a2350 0x7f09140a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f0913fff640 1 -- 172.21.15.116:0/454172261 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f091c014070 con 0x7f092c1005a0 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f092a59c640 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e70ce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0920009fa0 tx=0x7f09200118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bae6748df44e635 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f09117fa640 1 -- 172.21.15.116:0/3244956332 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920004310 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f09117fa640 1 -- 172.21.15.116:0/3244956332 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0920004470 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.885+0000 7f09117fa640 1 -- 172.21.15.116:0/3244956332 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920004e80 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 -- 172.21.15.116:0/3244956332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 msgr2=0x55e7f4e70ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f092b59e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f09140a2350 0x7f09140a4810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0924037040 tx=0x7f09240095c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e70ce0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0920009fa0 tx=0x7f09200118d0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 -- 172.21.15.116:0/3244956332 shutdown_connections 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e70ce0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3244956332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e657d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 -- 172.21.15.116:0/3244956332 >> 172.21.15.116:0/3244956332 conn(0x55e7f4e5fb20 msgr2=0x55e7f4e5ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 -- 172.21.15.116:0/3244956332 shutdown_connections 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.886+0000 7f0933c2e740 1 -- 172.21.15.116:0/3244956332 wait complete. 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 -- start start 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 Processor -- start 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e6c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4a9f430 con 0x55e7f4e653f0 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4a5be00 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4d51590 con 0x55e7f4e65d10 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f0933c2e740 10 client.0 _ll_register_callbacks cb 0x55e7f4dc0fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:15.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f092a59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f092a59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45872/0 (socket says 172.21.15.116:45872) 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f092a59c640 1 -- 172.21.15.116:0/3564797205 learned_addr learned my addr 172.21.15.116:0/3564797205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f092b59e640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e6c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.887+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092a59c640 1 -- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 msgr2=0x55e7f4e6c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092a59c640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e6c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092a59c640 1 -- 172.21.15.116:0/3564797205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 msgr2=0x55e7f4e6b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092a59c640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092a59c640 1 -- 172.21.15.116:0/3564797205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0920009c00 con 0x55e7f4e65d10 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092b59e640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e6c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f092a59c640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6be00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f092001b040 tx=0x7f0920004880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1823338efdbbcbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f090eff5640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f090eff5640 1 -- 172.21.15.116:0/3564797205 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920003a00 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e7f4acb7e0 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f090eff5640 1 -- 172.21.15.116:0/3564797205 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0920003b60 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f090eff5640 1 -- 172.21.15.116:0/3564797205 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0920003d20 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f0933c2e740 10 client.25714 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.888+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e7f4dcabd0 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f090eff5640 1 -- 172.21.15.116:0/3564797205 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e7f4acb7e0 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f090eff5640 1 -- 172.21.15.116:0/3564797205 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f0920023560 con 0x55e7f4e65d10 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f0933c2e740 10 client.25714 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f0933c2e740 10 client.25714 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f0933c2e740 20 client.25714 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4e7e990 0x55e7f4e80df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f0933c2e740 4 client.25714 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7f4b08720 con 0x55e7f4e7e990 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.889+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4e7e990 0x55e7f4e80df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.890+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4e7e990 0x55e7f4e80df0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5a0d5deddeef53c4 server_cookie=fb897c1690413627 in_seq=0 out_seq=0 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.890+0000 7f090eff5640 10 client.25714 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.890+0000 7f090eff5640 1 -- 172.21.15.116:0/3564797205 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7f4b08720 con 0x55e7f4e7e990 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.890+0000 7f090eff5640 10 client.25714 handle_command_reply: tid=0 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 client.25714 shutdown 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 msgr2=0x55e7f4e6be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6be00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f092001b040 tx=0x7f0920004880 comp rx=0 tx=0).stop 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 shutdown_connections 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e6c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e6be00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4e7e990 0x55e7f4e80df0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3564797205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f090eff5640 0 client.25714 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 >> 172.21.15.116:0/3564797205 conn(0x55e7f4e5fb20 msgr2=0x55e7f4e61870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 shutdown_connections 2024-03-24T09:23:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 1 -- 172.21.15.116:0/3564797205 wait complete. 2024-03-24T09:23:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.891+0000 7f0933c2e740 20 client.25714 trim_cache size 0 max 16384 2024-03-24T09:23:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 Processor -- start 2024-03-24T09:23:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 -- start start 2024-03-24T09:23:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e7a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e86700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4e715c0 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4e7b470 con 0x55e7f4e653f0 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f0933c2e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4e70ce0 con 0x55e7f4e65d10 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e86700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092a59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e7a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e86700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54042/0 (socket says 172.21.15.116:54042) 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 -- 172.21.15.116:0/3482268483 learned_addr learned my addr 172.21.15.116:0/3482268483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 -- 172.21.15.116:0/3482268483 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 msgr2=0x55e7f4e7a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e7a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 -- 172.21.15.116:0/3482268483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 msgr2=0x55e7f4e6dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.906+0000 7f092b59e640 1 -- 172.21.15.116:0/3482268483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7f4d66c80 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.907+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.907+0000 7f092b59e640 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e86700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0924036000 tx=0x7f0924043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862cfff14ffca3d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.907+0000 7f0910ff9640 1 -- 172.21.15.116:0/3482268483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0924003970 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.907+0000 7f0910ff9640 1 -- 172.21.15.116:0/3482268483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0924004040 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.907+0000 7f0910ff9640 1 -- 172.21.15.116:0/3482268483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f092404d680 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- 172.21.15.116:0/3482268483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 msgr2=0x55e7f4e86700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e86700 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f0924036000 tx=0x7f0924043000 comp rx=0 tx=0).stop 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- 172.21.15.116:0/3482268483 shutdown_connections 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4e86700 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e65d10 0x55e7f4e7a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3482268483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e6dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- 172.21.15.116:0/3482268483 >> 172.21.15.116:0/3482268483 conn(0x55e7f4e5fb20 msgr2=0x55e7f4e5ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- 172.21.15.116:0/3482268483 shutdown_connections 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- 172.21.15.116:0/3482268483 wait complete. 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- start start 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 Processor -- start 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e72870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 0x55e7f4e72db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4ee3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4ee4270 con 0x55e7f4e65d10 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4ee43b0 con 0x55e7f4e653f0 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e7f4ee44f0 con 0x55e7f4e6e8f0 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f0933c2e740 10 client.0 _ll_register_callbacks cb 0x7f091c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:15.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f092a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 0x55e7f4e72db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f092ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e72870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f092a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 0x55e7f4e72db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54052/0 (socket says 172.21.15.116:54052) 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f092ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e72870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:54110/0 (socket says 172.21.15.116:54110) 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f092b59e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4ee3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.908+0000 7f092a59c640 1 -- 172.21.15.116:0/3328257962 learned_addr learned my addr 172.21.15.116:0/3328257962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f092a59c640 1 -- 172.21.15.116:0/3328257962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e6e8f0 msgr2=0x55e7f4ee3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f092a59c640 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4ee3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f092a59c640 1 -- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 msgr2=0x55e7f4e72870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f092a59c640 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e72870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f092a59c640 1 -- 172.21.15.116:0/3328257962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f092c1a4850 con 0x55e7f4e65d10 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f092a59c640 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 0x55e7f4e72db0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x55e7f4e6ce90 tx=0x7f0920009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0bd1d387ca2fa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:15.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f090eff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f090eff5640 1 -- 172.21.15.116:0/3328257962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0920011dd0 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f090eff5640 1 -- 172.21.15.116:0/3328257962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0920000e40 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0920004470 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f090eff5640 1 -- 172.21.15.116:0/3328257962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0920023440 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f0933c2e740 10 client.15999 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x7f0924030070 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.909+0000 7f090eff5640 1 -- 172.21.15.116:0/3328257962 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0920004470 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f090eff5640 1 -- 172.21.15.116:0/3328257962 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f092000f4e0 con 0x55e7f4e65d10 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 10 client.15999 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 10 client.15999 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 20 client.15999 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4ee59f0 0x55e7f4ee7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:15.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 4 client.15999 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7f4b57890 con 0x55e7f4ee59f0 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4ee59f0 0x55e7f4ee7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f090eff5640 10 client.15999 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f092ad9d640 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4ee59f0 0x55e7f4ee7e10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8cdf4a187b7850a0 server_cookie=4d01130a4f0f361f in_seq=0 out_seq=0 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f090eff5640 1 -- 172.21.15.116:0/3328257962 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7f4b57890 con 0x55e7f4ee59f0 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f090eff5640 10 client.15999 handle_command_reply: tid=0 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.910+0000 7f0933c2e740 1 client.15999 shutdown 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 msgr2=0x55e7f4e72db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 0x55e7f4e72db0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x55e7f4e6ce90 tx=0x7f0920009eb0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 shutdown_connections 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7f4e6e8f0 0x55e7f4ee3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e7f4ee59f0 0x55e7f4ee7e10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7f4e65d10 0x55e7f4e72db0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 --2- 172.21.15.116:0/3328257962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7f4e653f0 0x55e7f4e72870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f090eff5640 0 client.15999 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 >> 172.21.15.116:0/3328257962 conn(0x55e7f4e5fb20 msgr2=0x55e7f4e6b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 shutdown_connections 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 1 -- 172.21.15.116:0/3328257962 wait complete. 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f0933c2e740 20 client.15999 trim_cache size 0 max 16384 2024-03-24T09:23:15.919 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:15.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f09140a2350 msgr2=0x7f09140a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f09140a2350 0x7f09140a4810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0924037040 tx=0x7f09240095c0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 msgr2=0x7f092c198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:15.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 0x7f092c198990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f091c00e9f0 tx=0x7f091c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 shutdown_connections 2024-03-24T09:23:15.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f09140a2350 0x7f09140a4810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f092c106660 0x7f092c1a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f092c100f60 0x7f092c198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 --2- 172.21.15.116:0/454172261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092c1005a0 0x7f092c198990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:15.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 >> 172.21.15.116:0/454172261 conn(0x7f092c077710 msgr2=0x7f092c0ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:15.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 shutdown_connections 2024-03-24T09:23:15.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:15.911+0000 7f093104e640 1 -- 172.21.15.116:0/454172261 wait complete. 2024-03-24T09:23:15.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 -- 172.21.15.116:0/2348590805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec1005a0 msgr2=0x7f3fec100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 --2- 172.21.15.116:0/2348590805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec1005a0 0x7f3fec100a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009890 tx=0x7f3fe002bb70 comp rx=0 tx=0).stop 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 -- 172.21.15.116:0/2348590805 shutdown_connections 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 --2- 172.21.15.116:0/2348590805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 --2- 172.21.15.116:0/2348590805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec1005a0 0x7f3fec100a20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 --2- 172.21.15.116:0/2348590805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec106660 0x7f3fec106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 -- 172.21.15.116:0/2348590805 >> 172.21.15.116:0/2348590805 conn(0x7f3fec077710 msgr2=0x7f3fec077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 -- 172.21.15.116:0/2348590805 shutdown_connections 2024-03-24T09:23:16.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.093+0000 7f3ff1247640 1 -- 172.21.15.116:0/2348590805 wait complete. 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.094+0000 7f3ff1247640 1 Processor -- start 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.094+0000 7f3ff1247640 1 -- start start 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.094+0000 7f3ff1247640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec1005a0 0x7f3fec199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3ff1247640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3ff1247640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec106660 0x7f3fec19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3ff1247640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec19ecd0 con 0x7f3fec100f60 2024-03-24T09:23:16.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3ff1247640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec19ee40 con 0x7f3fec1005a0 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3ff1247640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3fec19f000 con 0x7f3fec106660 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec1005a0 0x7f3fec199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54066/0 (socket says 172.21.15.116:54066) 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec106660 0x7f3fec19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fea59c640 1 -- 172.21.15.116:0/4135920698 learned_addr learned my addr 172.21.15.116:0/4135920698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 -- 172.21.15.116:0/4135920698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec1005a0 msgr2=0x7f3fec199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec1005a0 0x7f3fec199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 -- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 msgr2=0x7f3fec19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 -- 172.21.15.116:0/4135920698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe4009900 con 0x7f3fec106660 2024-03-24T09:23:16.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fead9d640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec1005a0 0x7f3fec199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fea59c640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3feb59e640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec106660 0x7f3fec19e6d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400efc0 tx=0x7f3fe400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3fd3fff640 1 -- 172.21.15.116:0/4135920698 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4003970 con 0x7f3fec106660 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.095+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fdc009490 con 0x7f3fec106660 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.096+0000 7f3fd3fff640 1 -- 172.21.15.116:0/4135920698 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fe4004020 con 0x7f3fec106660 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.096+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe0009490 con 0x7f3fec106660 2024-03-24T09:23:16.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.096+0000 7f3fd3fff640 1 -- 172.21.15.116:0/4135920698 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe400b890 con 0x7f3fec106660 2024-03-24T09:23:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.098+0000 7f3fd3fff640 1 -- 172.21.15.116:0/4135920698 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f3fe400a040 con 0x7f3fec106660 2024-03-24T09:23:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 Processor -- start 2024-03-24T09:23:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 -- start start 2024-03-24T09:23:16.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fd3fff640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3fd40a23d0 0x7f3fd40a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bacd10 0x5628a9bb53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fea59c640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3fd40a23d0 0x7f3fd40a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fd3fff640 1 -- 172.21.15.116:0/4135920698 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3fe40ce750 con 0x7f3fec106660 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5628a9b1a840 con 0x5628a9bac3f0 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628a9b17d70 con 0x5628a9bb58f0 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628a9b13550 con 0x5628a9bacd10 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bac7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54072/0 (socket says 172.21.15.116:54072) 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fead9d640 1 -- 172.21.15.116:0/3743219668 learned_addr learned my addr 172.21.15.116:0/3743219668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3fea59c640 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bacd10 0x5628a9bb53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.099+0000 7f3feb59e640 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3feb59e640 1 -- 172.21.15.116:0/3743219668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bacd10 msgr2=0x5628a9bb53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3feb59e640 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bacd10 0x5628a9bb53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3feb59e640 1 -- 172.21.15.116:0/3743219668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 msgr2=0x5628a9bac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3fea59c640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3fd40a23d0 0x7f3fd40a4890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005e00 tx=0x7f3fe0004290 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3feb59e640 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3feb59e640 1 -- 172.21.15.116:0/3743219668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628a9aa8de0 con 0x5628a9bb58f0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3fead9d640 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3feb59e640 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb7ce0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40cb970 tx=0x7f3fe409a7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=679adca0b1ab9a69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3fd17fa640 1 -- 172.21.15.116:0/3743219668 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe409ec90 con 0x5628a9bb58f0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3fd17fa640 1 -- 172.21.15.116:0/3743219668 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fe40a32e0 con 0x5628a9bb58f0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.100+0000 7f3fd17fa640 1 -- 172.21.15.116:0/3743219668 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe40ce520 con 0x5628a9bb58f0 2024-03-24T09:23:16.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 -- 172.21.15.116:0/3743219668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 msgr2=0x5628a9bb7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb7ce0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40cb970 tx=0x7f3fe409a7b0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 -- 172.21.15.116:0/3743219668 shutdown_connections 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb7ce0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bacd10 0x5628a9bb53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/3743219668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 -- 172.21.15.116:0/3743219668 >> 172.21.15.116:0/3743219668 conn(0x5628a9ba6b20 msgr2=0x5628a9ba6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 -- 172.21.15.116:0/3743219668 shutdown_connections 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 -- 172.21.15.116:0/3743219668 wait complete. 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 -- start start 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.101+0000 7f3ff3e26740 1 Processor -- start 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5628a97e6430 con 0x5628a9bacd10 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628a97a2e00 con 0x5628a9bb58f0 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628a9a98590 con 0x5628a9bac3f0 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3ff3e26740 10 client.0 _ll_register_callbacks cb 0x5628a9b07fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54076/0 (socket says 172.21.15.116:54076) 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fea59c640 1 -- 172.21.15.116:0/2759772792 learned_addr learned my addr 172.21.15.116:0/2759772792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fead9d640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3feb59e640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fead9d640 1 -- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 msgr2=0x5628a9bb3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fead9d640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fead9d640 1 -- 172.21.15.116:0/2759772792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 msgr2=0x5628a9bb2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fead9d640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fead9d640 1 -- 172.21.15.116:0/2759772792 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe0009490 con 0x5628a9bac3f0 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.102+0000 7f3fea59c640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3fead9d640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb2990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0058c0 tx=0x7f3fdc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79676f5985e2c125 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3fceff5640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:23:16.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3fceff5640 1 -- 172.21.15.116:0/2759772792 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc017070 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5628a98127e0 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3fceff5640 1 -- 172.21.15.116:0/2759772792 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3fdc004030 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3fceff5640 1 -- 172.21.15.116:0/2759772792 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3fdc0122e0 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3ff3e26740 10 client.25732 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.103+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5628a9b11bd0 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3fceff5640 1 -- 172.21.15.116:0/2759772792 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fdc009e50 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3fceff5640 1 -- 172.21.15.116:0/2759772792 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f3fdc01c460 con 0x5628a9bac3f0 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3ff3e26740 10 client.25732 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3ff3e26740 10 client.25732 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3ff3e26740 20 client.25732 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9bc5a70 0x5628a9bc7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3ff3e26740 4 client.25732 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:16.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628a98127e0 con 0x5628a9bc5a70 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3fea59c640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9bc5a70 0x5628a9bc7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3fea59c640 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9bc5a70 0x5628a9bc7ef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=15653d3d7d9a83e0 server_cookie=d5aae0d2b47461a7 in_seq=0 out_seq=0 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.104+0000 7f3fceff5640 10 client.25732 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3fceff5640 1 -- 172.21.15.116:0/2759772792 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628a98127e0 con 0x5628a9bc5a70 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3fceff5640 10 client.25732 handle_command_reply: tid=0 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 client.25732 shutdown 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 msgr2=0x5628a9bb2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb2990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc0058c0 tx=0x7f3fdc00f790 comp rx=0 tx=0).stop 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 shutdown_connections 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9bc5a70 0x5628a9bc7ef0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3fceff5640 0 client.25732 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:16.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/2759772792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb2990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.105+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 >> 172.21.15.116:0/2759772792 conn(0x5628a9ba6b20 msgr2=0x5628a9ba8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.106+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 shutdown_connections 2024-03-24T09:23:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.106+0000 7f3ff3e26740 1 -- 172.21.15.116:0/2759772792 wait complete. 2024-03-24T09:23:16.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.106+0000 7f3ff3e26740 20 client.25732 trim_cache size 0 max 16384 2024-03-24T09:23:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.122+0000 7f3ff3e26740 1 Processor -- start 2024-03-24T09:23:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.122+0000 7f3ff3e26740 1 -- start start 2024-03-24T09:23:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bc1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bacd10 0x5628a9ba8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bb58f0 0x5628a9bcd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5628a9ba6650 con 0x5628a9bac3f0 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628a9bc16c0 con 0x5628a9bacd10 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628a9ba23e0 con 0x5628a9bb58f0 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bc1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bacd10 0x5628a9ba8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bc1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54078/0 (socket says 172.21.15.116:54078) 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 -- 172.21.15.116:0/1421258651 learned_addr learned my addr 172.21.15.116:0/1421258651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3feb59e640 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bb58f0 0x5628a9bcd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 -- 172.21.15.116:0/1421258651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bb58f0 msgr2=0x5628a9bcd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bb58f0 0x5628a9bcd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 -- 172.21.15.116:0/1421258651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bacd10 msgr2=0x5628a9ba8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bacd10 0x5628a9ba8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 -- 172.21.15.116:0/1421258651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628a9aadc80 con 0x5628a9bac3f0 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fea59c640 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bacd10 0x5628a9ba8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3feb59e640 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bb58f0 0x5628a9bcd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fead9d640 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bc1180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00fed0 tx=0x7f3fdc012ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7460789401985666 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.123+0000 7f3fd0ff9640 1 -- 172.21.15.116:0/1421258651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc017040 con 0x5628a9bac3f0 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3fd0ff9640 1 -- 172.21.15.116:0/1421258651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fdc009e50 con 0x5628a9bac3f0 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3fd0ff9640 1 -- 172.21.15.116:0/1421258651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc003850 con 0x5628a9bac3f0 2024-03-24T09:23:16.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 -- 172.21.15.116:0/1421258651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 msgr2=0x5628a9bc1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bc1180 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00fed0 tx=0x7f3fdc012ca0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 -- 172.21.15.116:0/1421258651 shutdown_connections 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bb58f0 0x5628a9bcd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bacd10 0x5628a9ba8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/1421258651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bac3f0 0x5628a9bc1180 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 -- 172.21.15.116:0/1421258651 >> 172.21.15.116:0/1421258651 conn(0x5628a9ba6b20 msgr2=0x5628a9ba7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 -- 172.21.15.116:0/1421258651 shutdown_connections 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.124+0000 7f3ff3e26740 1 -- 172.21.15.116:0/1421258651 wait complete. 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 -- start start 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 Processor -- start 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5628a9c33600 con 0x5628a9bacd10 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628a9c33770 con 0x5628a9bb58f0 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628a9c33930 con 0x5628a9bac3f0 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3ff3e26740 10 client.0 _ll_register_callbacks cb 0x7f3fdc009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3feb59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3fead9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb91d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45928/0 (socket says 172.21.15.116:45928) 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54090/0 (socket says 172.21.15.116:54090) 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3fead9d640 1 -- 172.21.15.116:0/238124554 learned_addr learned my addr 172.21.15.116:0/238124554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.125+0000 7f3feb59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:54152/0 (socket says 172.21.15.116:54152) 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3feb59e640 1 -- 172.21.15.116:0/238124554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 msgr2=0x5628a9bb91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3feb59e640 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3feb59e640 1 -- 172.21.15.116:0/238124554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 msgr2=0x5628a9bb9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3feb59e640 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3feb59e640 1 -- 172.21.15.116:0/238124554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe0049050 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3fead9d640 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3feb59e640 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb9c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4097510 tx=0x7f3fe4004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=676099f0d0b4434d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3fceff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3fceff5640 1 -- 172.21.15.116:0/238124554 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe407d070 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3fceff5640 1 -- 172.21.15.116:0/238124554 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3fe409ab40 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fe40a32e0 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.126+0000 7f3fceff5640 1 -- 172.21.15.116:0/238124554 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3fe40043a0 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 10 client.25850 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5628a9b11bd0 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3fceff5640 1 -- 172.21.15.116:0/238124554 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3fe40a7020 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3fceff5640 1 -- 172.21.15.116:0/238124554 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f3fe4080030 con 0x5628a9bb58f0 2024-03-24T09:23:16.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 10 client.25850 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 10 client.25850 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 20 client.25850 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9c34d60 0x5628a9c37150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 4 client.25850 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3fe40a32e0 con 0x5628a9c34d60 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.127+0000 7f3fead9d640 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9c34d60 0x5628a9c37150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3fceff5640 10 client.25850 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:16.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3fead9d640 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9c34d60 0x5628a9c37150 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b510b45f6323c906 server_cookie=c8c47a8a13836851 in_seq=0 out_seq=0 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3fceff5640 1 -- 172.21.15.116:0/238124554 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3fe40a32e0 con 0x5628a9c34d60 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3fceff5640 10 client.25850 handle_command_reply: tid=0 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 client.25850 shutdown 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 msgr2=0x5628a9bb9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb9c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4097510 tx=0x7f3fe4004ab0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 shutdown_connections 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5628a9c34d60 0x5628a9c37150 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a9bb58f0 0x5628a9bb9c50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3fceff5640 0 client.25850 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628a9bacd10 0x5628a9bb9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 --2- 172.21.15.116:0/238124554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628a9bac3f0 0x5628a9bb91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 >> 172.21.15.116:0/238124554 conn(0x5628a9ba6b20 msgr2=0x5628a9ba5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 shutdown_connections 2024-03-24T09:23:16.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.128+0000 7f3ff3e26740 1 -- 172.21.15.116:0/238124554 wait complete. 2024-03-24T09:23:16.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff3e26740 20 client.25850 trim_cache size 0 max 16384 2024-03-24T09:23:16.136 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3fd40a23d0 msgr2=0x7f3fd40a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3fd40a23d0 0x7f3fd40a4890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005e00 tx=0x7f3fe0004290 comp rx=0 tx=0).stop 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec106660 msgr2=0x7f3fec19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec106660 0x7f3fec19e6d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3fe400efc0 tx=0x7f3fe400c800 comp rx=0 tx=0).stop 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 shutdown_connections 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3fd40a23d0 0x7f3fd40a4890 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3fec106660 0x7f3fec19e6d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fec100f60 0x7f3fec19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 --2- 172.21.15.116:0/4135920698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fec1005a0 0x7f3fec199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.129+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 >> 172.21.15.116:0/4135920698 conn(0x7f3fec077710 msgr2=0x7f3fec068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.130+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 shutdown_connections 2024-03-24T09:23:16.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.130+0000 7f3ff1247640 1 -- 172.21.15.116:0/4135920698 wait complete. 2024-03-24T09:23:16.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 -- 172.21.15.116:0/2277590049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 msgr2=0x7f1308106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 --2- 172.21.15.116:0/2277590049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f1308106a40 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f1300009ca0 tx=0x7f130002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 -- 172.21.15.116:0/2277590049 shutdown_connections 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 --2- 172.21.15.116:0/2277590049 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1308100f60 0x7f130810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 --2- 172.21.15.116:0/2277590049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13081005a0 0x7f1308100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 --2- 172.21.15.116:0/2277590049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f1308106a40 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 -- 172.21.15.116:0/2277590049 >> 172.21.15.116:0/2277590049 conn(0x7f1308077710 msgr2=0x7f1308077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 -- 172.21.15.116:0/2277590049 shutdown_connections 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.288+0000 7f1310642640 1 -- 172.21.15.116:0/2277590049 wait complete. 2024-03-24T09:23:16.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 Processor -- start 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 -- start start 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13081005a0 0x7f130819d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1308100f60 0x7f130819dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f13081979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1308198010 con 0x7f1308106660 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1308198180 con 0x7f1308100f60 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f1310642640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1308198340 con 0x7f13081005a0 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130ebdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f13081979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130ebdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f13081979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54106/0 (socket says 172.21.15.116:54106) 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130ebdf640 1 -- 172.21.15.116:0/2773766913 learned_addr learned my addr 172.21.15.116:0/2773766913 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130dbdd640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1308100f60 0x7f130819dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130e3de640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13081005a0 0x7f130819d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130dbdd640 1 -- 172.21.15.116:0/2773766913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13081005a0 msgr2=0x7f130819d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130dbdd640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13081005a0 0x7f130819d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130dbdd640 1 -- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 msgr2=0x7f13081979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130dbdd640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f13081979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130dbdd640 1 -- 172.21.15.116:0/2773766913 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1300009900 con 0x7f1308100f60 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.289+0000 7f130ebdf640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f13081979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.290+0000 7f130dbdd640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1308100f60 0x7f130819dd50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00e9f0 tx=0x7f12fc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.290+0000 7f13077fe640 1 -- 172.21.15.116:0/2773766913 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00ccc0 con 0x7f1308100f60 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.290+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1308198620 con 0x7f1308100f60 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.290+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13081a48b0 con 0x7f1308100f60 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.290+0000 7f13077fe640 1 -- 172.21.15.116:0/2773766913 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f12fc00ce20 con 0x7f1308100f60 2024-03-24T09:23:16.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.290+0000 7f13077fe640 1 -- 172.21.15.116:0/2773766913 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc018640 con 0x7f1308100f60 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.292+0000 7f13077fe640 1 -- 172.21.15.116:0/2773766913 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f12fc010430 con 0x7f1308100f60 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.292+0000 7f1313219740 1 Processor -- start 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 -- start start 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad79e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599ad70c840 con 0x5599ad79ed10 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599ad709d70 con 0x5599ad7a78f0 2024-03-24T09:23:16.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f1313219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5599ad705550 con 0x5599ad79e3f0 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f13077fe640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f12f00a2350 0x7f12f00a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130ebdf640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f12f00a2350 0x7f12f00a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54118/0 (socket says 172.21.15.116:54118) 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 -- 172.21.15.116:0/111958443 learned_addr learned my addr 172.21.15.116:0/111958443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:54182/0 (socket says 172.21.15.116:54182) 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130ebdf640 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad79e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f13077fe640 1 -- 172.21.15.116:0/2773766913 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f12fc014070 con 0x7f1308100f60 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 -- 172.21.15.116:0/111958443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 msgr2=0x5599ad79e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad79e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 -- 172.21.15.116:0/111958443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 msgr2=0x5599ad7a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130e3de640 1 -- 172.21.15.116:0/111958443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ad69ade0 con 0x5599ad79ed10 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.293+0000 7f130dbdd640 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f130ebdf640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f12f00a2350 0x7f12f00a4810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12f8011d80 tx=0x7f12f8009f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f130e3de640 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a73b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f130002c700 tx=0x7f1300005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d62be9f0616d6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f1304bfe640 1 -- 172.21.15.116:0/111958443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300034440 con 0x5599ad79ed10 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f1304bfe640 1 -- 172.21.15.116:0/111958443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13000345a0 con 0x5599ad79ed10 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f1304bfe640 1 -- 172.21.15.116:0/111958443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300003970 con 0x5599ad79ed10 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f1313219740 1 -- 172.21.15.116:0/111958443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 msgr2=0x5599ad7a73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.294+0000 7f1313219740 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a73b0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f130002c700 tx=0x7f1300005f70 comp rx=0 tx=0).stop 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- 172.21.15.116:0/111958443 shutdown_connections 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a73b0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 --2- 172.21.15.116:0/111958443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad79e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- 172.21.15.116:0/111958443 >> 172.21.15.116:0/111958443 conn(0x5599ad798b20 msgr2=0x5599ad798f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- 172.21.15.116:0/111958443 shutdown_connections 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- 172.21.15.116:0/111958443 wait complete. 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- start start 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 Processor -- start 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad7a4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599ad3d8430 con 0x5599ad79ed10 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599ad394e00 con 0x5599ad7a78f0 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5599ad68a590 con 0x5599ad79e3f0 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.295+0000 7f1313219740 10 client.0 _ll_register_callbacks cb 0x5599ad6f9fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad7a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad7a4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45964/0 (socket says 172.21.15.116:45964) 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130ebdf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 -- 172.21.15.116:0/3998265230 learned_addr learned my addr 172.21.15.116:0/3998265230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54134/0 (socket says 172.21.15.116:54134) 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 -- 172.21.15.116:0/3998265230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 msgr2=0x5599ad7a5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 -- 172.21.15.116:0/3998265230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 msgr2=0x5599ad7a4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 -- 172.21.15.116:0/3998265230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130810dd20 con 0x5599ad79e3f0 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130ebdf640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130dbdd640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f130e3de640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad7a4890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1300037040 tx=0x7f1300032bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fba4ec222285bdbc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f12edffb640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.296+0000 7f12edffb640 1 -- 172.21.15.116:0/3998265230 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300034c60 con 0x5599ad79e3f0 2024-03-24T09:23:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5599ad4047e0 con 0x5599ad79e3f0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f12edffb640 1 -- 172.21.15.116:0/3998265230 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1300050070 con 0x5599ad79e3f0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f12edffb640 1 -- 172.21.15.116:0/3998265230 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1300034dc0 con 0x5599ad79e3f0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 10 client.25750 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5599ad703bd0 con 0x5599ad79e3f0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f12edffb640 1 -- 172.21.15.116:0/3998265230 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5599ad4047e0 con 0x5599ad79e3f0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f12edffb640 1 -- 172.21.15.116:0/3998265230 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f1300004550 con 0x5599ad79e3f0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 10 client.25750 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 10 client.25750 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 20 client.25750 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad7b7680 0x5599ad7b9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 4 client.25750 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599ad441720 con 0x5599ad7b7680 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.297+0000 7f130dbdd640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad7b7680 0x5599ad7b9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.298+0000 7f130dbdd640 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad7b7680 0x5599ad7b9ac0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c0d4e5bf2f1e7c7f server_cookie=2c3905edf6853175 in_seq=0 out_seq=0 2024-03-24T09:23:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.298+0000 7f12edffb640 10 client.25750 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f12edffb640 1 -- 172.21.15.116:0/3998265230 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5599ad441720 con 0x5599ad7b7680 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f12edffb640 10 client.25750 handle_command_reply: tid=0 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 client.25750 shutdown 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 msgr2=0x5599ad7a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad7a4890 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1300037040 tx=0x7f1300032bd0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 shutdown_connections 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad7b7680 0x5599ad7b9ac0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad7a78f0 0x5599ad7a5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f12edffb640 0 client.25750 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7a4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 --2- 172.21.15.116:0/3998265230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad79e3f0 0x5599ad7a4890 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 >> 172.21.15.116:0/3998265230 conn(0x5599ad798b20 msgr2=0x5599ad79a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 shutdown_connections 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 1 -- 172.21.15.116:0/3998265230 wait complete. 2024-03-24T09:23:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.299+0000 7f1313219740 20 client.25750 trim_cache size 0 max 16384 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.315+0000 7f1313219740 1 Processor -- start 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.315+0000 7f1313219740 1 -- start start 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f1313219740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 0x5599ad7a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f1313219740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79ed10 0x5599ad7b3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f1313219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad797b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f1313219740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599ad7b43d0 con 0x5599ad79e3f0 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f1313219740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599ad7a9ce0 con 0x5599ad79ed10 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f1313219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5599ad7941e0 con 0x5599ad7a78f0 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 0x5599ad7a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130ebdf640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad797b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79ed10 0x5599ad7b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 0x5599ad7a6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54144/0 (socket says 172.21.15.116:54144) 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 -- 172.21.15.116:0/575707658 learned_addr learned my addr 172.21.15.116:0/575707658 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 -- 172.21.15.116:0/575707658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 msgr2=0x5599ad797b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad797b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 -- 172.21.15.116:0/575707658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79ed10 msgr2=0x5599ad7b3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79ed10 0x5599ad7b3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 -- 172.21.15.116:0/575707658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599ad69fc80 con 0x5599ad79e3f0 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f130e3de640 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 0x5599ad7a6fc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f130002c630 tx=0x7f1300032930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fc1cc12597c127 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.316+0000 7f12effff640 1 -- 172.21.15.116:0/575707658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300034440 con 0x5599ad79e3f0 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f12effff640 1 -- 172.21.15.116:0/575707658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f130004e070 con 0x5599ad79e3f0 2024-03-24T09:23:16.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f12effff640 1 -- 172.21.15.116:0/575707658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130004a450 con 0x5599ad79e3f0 2024-03-24T09:23:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 -- 172.21.15.116:0/575707658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 msgr2=0x5599ad7a6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 0x5599ad7a6fc0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f130002c630 tx=0x7f1300032930 comp rx=0 tx=0).stop 2024-03-24T09:23:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 -- 172.21.15.116:0/575707658 shutdown_connections 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad797b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79ed10 0x5599ad7b3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 --2- 172.21.15.116:0/575707658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79e3f0 0x5599ad7a6fc0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 -- 172.21.15.116:0/575707658 >> 172.21.15.116:0/575707658 conn(0x5599ad798b20 msgr2=0x5599ad797430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 -- 172.21.15.116:0/575707658 shutdown_connections 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.317+0000 7f1313219740 1 -- 172.21.15.116:0/575707658 wait complete. 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 -- start start 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 Processor -- start 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79e3f0 0x5599ad7ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad7abc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5599ad81ca00 con 0x5599ad79ed10 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599ad81cb40 con 0x5599ad79e3f0 2024-03-24T09:23:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5599ad81cd00 con 0x5599ad7a78f0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f1313219740 10 client.0 _ll_register_callbacks cb 0x7f12fc096e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7ab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54148/0 (socket says 172.21.15.116:54148) 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 -- 172.21.15.116:0/3988273951 learned_addr learned my addr 172.21.15.116:0/3988273951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130e3de640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79e3f0 0x5599ad7ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130ebdf640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad7abc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 -- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 msgr2=0x5599ad7abc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad7abc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 -- 172.21.15.116:0/3988273951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79e3f0 msgr2=0x5599ad7ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79e3f0 0x5599ad7ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.318+0000 7f130dbdd640 1 -- 172.21.15.116:0/3988273951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599ad703bd0 con 0x5599ad79ed10 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f130dbdd640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7ab6e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0cc290 tx=0x7f12fc06cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f08405a3185183 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f12edffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f12edffb640 1 -- 172.21.15.116:0/3988273951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc0cbff0 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f12edffb640 1 -- 172.21.15.116:0/3988273951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f12fc0b3660 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13000345a0 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f12edffb640 1 -- 172.21.15.116:0/3988273951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f12fc0b3820 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f1313219740 10 client.16035 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5599ad6c5920 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f12edffb640 1 -- 172.21.15.116:0/3988273951 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13000345a0 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f12edffb640 1 -- 172.21.15.116:0/3988273951 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f12fc09fd50 con 0x5599ad79ed10 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f1313219740 10 client.16035 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.319+0000 7f1313219740 10 client.16035 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f1313219740 20 client.16035 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f1313219740 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad81e570 0x5599ad820990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f1313219740 4 client.16035 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599ad490890 con 0x5599ad81e570 2024-03-24T09:23:16.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f130e3de640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad81e570 0x5599ad820990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f130e3de640 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad81e570 0x5599ad820990 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eb311167981ac1cd server_cookie=ae8154ba9fc0ad99 in_seq=0 out_seq=0 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f12edffb640 10 client.16035 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f12edffb640 1 -- 172.21.15.116:0/3988273951 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599ad490890 con 0x5599ad81e570 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f12edffb640 10 client.16035 handle_command_reply: tid=0 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f1313219740 1 client.16035 shutdown 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.320+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 msgr2=0x5599ad7ab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7ab6e0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0cc290 tx=0x7f12fc06cd50 comp rx=0 tx=0).stop 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 shutdown_connections 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5599ad7a78f0 0x5599ad7abc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5599ad79ed10 0x5599ad7ab6e0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5599ad81e570 0x5599ad820990 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 --2- 172.21.15.116:0/3988273951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599ad79e3f0 0x5599ad7ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f12edffb640 0 client.16035 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 >> 172.21.15.116:0/3988273951 conn(0x5599ad798b20 msgr2=0x5599ad797cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 shutdown_connections 2024-03-24T09:23:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 1 -- 172.21.15.116:0/3988273951 wait complete. 2024-03-24T09:23:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1313219740 20 client.16035 trim_cache size 0 max 16384 2024-03-24T09:23:16.327 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f12f00a2350 msgr2=0x7f12f00a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1310642640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f12f00a2350 0x7f12f00a4810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12f8011d80 tx=0x7f12f8009f90 comp rx=0 tx=0).stop 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1308100f60 msgr2=0x7f130819dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.321+0000 7f1310642640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1308100f60 0x7f130819dd50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00e9f0 tx=0x7f12fc00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 shutdown_connections 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1308106660 0x7f13081979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1308100f60 0x7f130819dd50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f12f00a2350 0x7f12f00a4810 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 --2- 172.21.15.116:0/2773766913 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13081005a0 0x7f130819d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 >> 172.21.15.116:0/2773766913 conn(0x7f1308077710 msgr2=0x7f1308104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 shutdown_connections 2024-03-24T09:23:16.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.322+0000 7f1310642640 1 -- 172.21.15.116:0/2773766913 wait complete. 2024-03-24T09:23:16.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:23:16.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 -- 172.21.15.116:0/745197609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f201076a0 msgr2=0x7f6f20107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/745197609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f201076a0 0x7f6f20107a80 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6f10009930 tx=0x7f6f1002bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 -- 172.21.15.116:0/745197609 shutdown_connections 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/745197609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f20102050 0x7f6f2010e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/745197609 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f20101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/745197609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f201076a0 0x7f6f20107a80 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 -- 172.21.15.116:0/745197609 >> 172.21.15.116:0/745197609 conn(0x7f6f200fd1a0 msgr2=0x7f6f200ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 -- 172.21.15.116:0/745197609 shutdown_connections 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 -- 172.21.15.116:0/745197609 wait complete. 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 Processor -- start 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.463+0000 7f6f24bf7640 1 -- start start 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f24bf7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f2019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f24bf7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 0x7f6f2019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f24bf7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f201076a0 0x7f6f20197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f24bf7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201982a0 con 0x7f6f20102050 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f24bf7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20198410 con 0x7f6f201076a0 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f24bf7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6f201985d0 con 0x7f6f20101690 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 0x7f6f2019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f2019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f201076a0 0x7f6f20197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f2019da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41860/0 (socket says 172.21.15.116:41860) 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f201076a0 0x7f6f20197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:51892/0 (socket says 172.21.15.116:51892) 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 -- 172.21.15.116:0/1866032041 learned_addr learned my addr 172.21.15.116:0/1866032041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 0x7f6f2019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54416/0 (socket says 172.21.15.116:54416) 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 -- 172.21.15.116:0/1866032041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f201076a0 msgr2=0x7f6f20197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f201076a0 0x7f6f20197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 -- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 msgr2=0x7f6f2019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 0x7f6f2019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 -- 172.21.15.116:0/1866032041 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f10009560 con 0x7f6f20101690 2024-03-24T09:23:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 0x7f6f2019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.464+0000 7f6f1e59c640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f2019da50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f10005ab0 tx=0x7f6f10005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.465+0000 7f6f0f7fe640 1 -- 172.21.15.116:0/1866032041 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f10004350 con 0x7f6f20101690 2024-03-24T09:23:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.465+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f20198850 con 0x7f6f20101690 2024-03-24T09:23:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.465+0000 7f6f0f7fe640 1 -- 172.21.15.116:0/1866032041 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f100044b0 con 0x7f6f20101690 2024-03-24T09:23:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.465+0000 7f6f0f7fe640 1 -- 172.21.15.116:0/1866032041 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f100053d0 con 0x7f6f20101690 2024-03-24T09:23:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.465+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f201a49d0 con 0x7f6f20101690 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.468+0000 7f6f277d8740 1 Processor -- start 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.468+0000 7f6f277d8740 1 -- start start 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.468+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 0x55e20f6d3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f0f7fe640 1 -- 172.21.15.116:0/1866032041 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f6f10005530 con 0x7f6f20101690 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d4120 0x55e20f6dc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6df0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f277d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e20f63d840 con 0x55e20f6d4120 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f277d8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e20f63ad70 con 0x55e20f6d3800 2024-03-24T09:23:16.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f277d8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e20f636550 con 0x55e20f6dcd00 2024-03-24T09:23:16.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 0x55e20f6d3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 0x55e20f6d3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:51906/0 (socket says 172.21.15.116:51906) 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3037003365 learned_addr learned my addr 172.21.15.116:0/3037003365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6df0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1e59c640 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d4120 0x55e20f6dc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3037003365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 msgr2=0x55e20f6df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6df0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3037003365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d4120 msgr2=0x55e20f6dc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d4120 0x55e20f6dc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3037003365 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e20f5cbde0 con 0x55e20f6d3800 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6df0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f0f7fe640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6f080a2350 0x7f6f080a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 0x55e20f6d3be0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009f50 tx=0x7f6f180118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc4abcea6c4a0dbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.469+0000 7f6f1e59c640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6f080a2350 0x7f6f080a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f0f7fe640 1 -- 172.21.15.116:0/1866032041 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6f1002de40 con 0x7f6f20101690 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f0cff9640 1 -- 172.21.15.116:0/3037003365 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18004d20 con 0x55e20f6d3800 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f0cff9640 1 -- 172.21.15.116:0/3037003365 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f18004e80 con 0x55e20f6d3800 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f0cff9640 1 -- 172.21.15.116:0/3037003365 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18005280 con 0x55e20f6d3800 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f277d8740 1 -- 172.21.15.116:0/3037003365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 msgr2=0x55e20f6d3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f1e59c640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6f080a2350 0x7f6f080a4810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f100ee030 tx=0x7f6f100f45b0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.470+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 0x55e20f6d3be0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009f50 tx=0x7f6f180118d0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- 172.21.15.116:0/3037003365 shutdown_connections 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6df0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d4120 0x55e20f6dc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3037003365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d3800 0x55e20f6d3be0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- 172.21.15.116:0/3037003365 >> 172.21.15.116:0/3037003365 conn(0x55e20f6cdf30 msgr2=0x55e20f6ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- 172.21.15.116:0/3037003365 shutdown_connections 2024-03-24T09:23:16.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- 172.21.15.116:0/3037003365 wait complete. 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- start start 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 Processor -- start 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 0x55e20f6d9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6da320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6da860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e20f309430 con 0x55e20f6d3800 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e20f2c5e00 con 0x55e20f6d4120 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e20f5bb590 con 0x55e20f6dcd00 2024-03-24T09:23:16.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.471+0000 7f6f277d8740 10 client.0 _ll_register_callbacks cb 0x55e20f62afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 0x55e20f6d9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1dd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6da320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 0x55e20f6d9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54436/0 (socket says 172.21.15.116:54436) 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1dd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6da320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:51910/0 (socket says 172.21.15.116:51910) 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3306589213 learned_addr learned my addr 172.21.15.116:0/3306589213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 msgr2=0x55e20f6da860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6da860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3306589213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 msgr2=0x55e20f6da320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6da320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3306589213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f18009c00 con 0x55e20f6d3800 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6da320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 0x55e20f6d9de0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009250 tx=0x7f6f18004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20b3789b4adb80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.472+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3306589213 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18003e40 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3306589213 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6f18005000 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e20f372720 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3306589213 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6f180051c0 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 10 client.16047 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e20f634bd0 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3306589213 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e20f372720 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3306589213 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f6f1802b020 con 0x55e20f6d3800 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 10 client.16047 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 10 client.16047 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 20 client.16047 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f6ece20 0x55e20f6ef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 4 client.16047 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e20f3c1890 con 0x55e20f6ece20 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.473+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f6ece20 0x55e20f6ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.474+0000 7f6f05ffb640 10 client.16047 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.474+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f6ece20 0x55e20f6ef210 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=208981bf4524e2e2 server_cookie=ee86c0b5e7949d6a in_seq=0 out_seq=0 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3306589213 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e20f3c1890 con 0x55e20f6ece20 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f05ffb640 10 client.16047 handle_command_reply: tid=0 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 client.16047 shutdown 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 msgr2=0x55e20f6d9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 0x55e20f6d9de0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6f18009250 tx=0x7f6f18004c70 comp rx=0 tx=0).stop 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 shutdown_connections 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f6ece20 0x55e20f6ef210 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6dcd00 0x55e20f6da860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6da320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3306589213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6d3800 0x55e20f6d9de0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 >> 172.21.15.116:0/3306589213 conn(0x55e20f6cdf30 msgr2=0x55e20f6cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.475+0000 7f6f05ffb640 0 client.16047 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.476+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 shutdown_connections 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.476+0000 7f6f277d8740 1 -- 172.21.15.116:0/3306589213 wait complete. 2024-03-24T09:23:16.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.476+0000 7f6f277d8740 20 client.16047 trim_cache size 0 max 16384 2024-03-24T09:23:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.492+0000 7f6f277d8740 1 Processor -- start 2024-03-24T09:23:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.492+0000 7f6f277d8740 1 -- start start 2024-03-24T09:23:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f277d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e20f6e9930 con 0x55e20f6dcd00 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f277d8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e20f6cef30 con 0x55e20f6d4120 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f277d8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e20f6e8ad0 con 0x55e20f6d3800 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6f4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54446/0 (socket says 172.21.15.116:54446) 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3458831168 learned_addr learned my addr 172.21.15.116:0/3458831168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3458831168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 msgr2=0x55e20f6e8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3458831168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 msgr2=0x55e20f6c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3458831168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e20f5d0c80 con 0x55e20f6dcd00 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6f4aa0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6f18012fd0 tx=0x7f6f1800f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c07bcbcafd2bb0b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.493+0000 7f6f07fff640 1 -- 172.21.15.116:0/3458831168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f18004ce0 con 0x55e20f6dcd00 2024-03-24T09:23:16.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f07fff640 1 -- 172.21.15.116:0/3458831168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6f1800f530 con 0x55e20f6dcd00 2024-03-24T09:23:16.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f07fff640 1 -- 172.21.15.116:0/3458831168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1800fd90 con 0x55e20f6dcd00 2024-03-24T09:23:16.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 -- 172.21.15.116:0/3458831168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 msgr2=0x55e20f6f4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6f4aa0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6f18012fd0 tx=0x7f6f1800f480 comp rx=0 tx=0).stop 2024-03-24T09:23:16.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 -- 172.21.15.116:0/3458831168 shutdown_connections 2024-03-24T09:23:16.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6f4aa0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3458831168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.494+0000 7f6f277d8740 1 -- 172.21.15.116:0/3458831168 >> 172.21.15.116:0/3458831168 conn(0x55e20f6cdf30 msgr2=0x55e20f6c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 -- 172.21.15.116:0/3458831168 shutdown_connections 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 -- 172.21.15.116:0/3458831168 wait complete. 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 -- start start 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 Processor -- start 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6d6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6d6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e20f6d7bc0 con 0x55e20f6dcd00 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e20f6d7d30 con 0x55e20f6d4120 2024-03-24T09:23:16.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e20f6d7ef0 con 0x55e20f6d3800 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f277d8740 10 client.0 _ll_register_callbacks cb 0x7f6f100f4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f1ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41906/0 (socket says 172.21.15.116:41906) 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3968651012 learned_addr learned my addr 172.21.15.116:0/3968651012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6d6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.495+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6d6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3968651012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 msgr2=0x55e20f6d6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6d6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3968651012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 msgr2=0x55e20f6d6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6d6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1ed9d640 1 -- 172.21.15.116:0/3968651012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f14012a80 con 0x55e20f6d3800 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6d6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f1ed9d640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e4bc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f18004020 tx=0x7f6f18009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60a89c063895d720 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3968651012 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f180043e0 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.496+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3968651012 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6f1800f6a0 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e20f3357e0 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3968651012 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6f18003c40 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f277d8740 10 client.25771 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55e20f634bd0 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3968651012 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e20f3357e0 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3968651012 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f6f1802b040 con 0x55e20f6d3800 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.497+0000 7f6f277d8740 10 client.25771 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f277d8740 10 client.25771 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f277d8740 20 client.25771 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f752c00 0x55e20f754ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f277d8740 4 client.25771 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e20f2e1700 con 0x55e20f752c00 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f752c00 0x55e20f754ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f05ffb640 10 client.25771 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.498+0000 7f6f1dd9b640 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f752c00 0x55e20f754ff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e9a96c10bd497015 server_cookie=3fb303e304bf1a8 in_seq=0 out_seq=0 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f05ffb640 1 -- 172.21.15.116:0/3968651012 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e20f2e1700 con 0x55e20f752c00 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f05ffb640 10 client.25771 handle_command_reply: tid=0 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 client.25771 shutdown 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 msgr2=0x55e20f6e4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e4bc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f18004020 tx=0x7f6f18009230 comp rx=0 tx=0).stop 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 shutdown_connections 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e20f752c00 0x55e20f754ff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e20f6dcd00 0x55e20f6d6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e20f6d4120 0x55e20f6d6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f05ffb640 0 client.25771 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 --2- 172.21.15.116:0/3968651012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e20f6d3800 0x55e20f6e4bc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 >> 172.21.15.116:0/3968651012 conn(0x55e20f6cdf30 msgr2=0x55e20f6d9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 shutdown_connections 2024-03-24T09:23:16.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 1 -- 172.21.15.116:0/3968651012 wait complete. 2024-03-24T09:23:16.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f277d8740 20 client.25771 trim_cache size 0 max 16384 2024-03-24T09:23:16.505 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6f080a2350 msgr2=0x7f6f080a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6f080a2350 0x7f6f080a4810 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6f100ee030 tx=0x7f6f100f45b0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 msgr2=0x7f6f2019da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.499+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f2019da50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6f10005ab0 tx=0x7f6f10005e00 comp rx=0 tx=0).stop 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 shutdown_connections 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f201076a0 0x7f6f20197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f20102050 0x7f6f2019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6f080a2350 0x7f6f080a4810 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 --2- 172.21.15.116:0/1866032041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6f20101690 0x7f6f2019da50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 >> 172.21.15.116:0/1866032041 conn(0x7f6f200fd1a0 msgr2=0x7f6f200fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 shutdown_connections 2024-03-24T09:23:16.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.500+0000 7f6f24bf7640 1 -- 172.21.15.116:0/1866032041 wait complete. 2024-03-24T09:23:16.511 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:16 smithi116 ceph-mon[32207]: pgmap v38: 129 pgs: 129 active+clean; 22 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 657 KiB/s wr, 119 op/s 2024-03-24T09:23:16.511 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:16 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3121486867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:23:16.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 -- 172.21.15.116:0/527010141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 msgr2=0x7f61e0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 --2- 172.21.15.116:0/527010141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0106a40 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009930 tx=0x7f61d002bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 -- 172.21.15.116:0/527010141 shutdown_connections 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 --2- 172.21.15.116:0/527010141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e0100f80 0x7f61e010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 --2- 172.21.15.116:0/527010141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 0x7f61e0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 --2- 172.21.15.116:0/527010141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0106a40 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 -- 172.21.15.116:0/527010141 >> 172.21.15.116:0/527010141 conn(0x7f61e0076990 msgr2=0x7f61e0076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 -- 172.21.15.116:0/527010141 shutdown_connections 2024-03-24T09:23:16.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.658+0000 7f61e49b5640 1 -- 172.21.15.116:0/527010141 wait complete. 2024-03-24T09:23:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 Processor -- start 2024-03-24T09:23:16.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 -- start start 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 0x7f61e019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e0100f80 0x7f61e019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61e0196fd0 con 0x7f61e0106660 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f61e0197140 con 0x7f61e0100f80 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61e49b5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f61e0197300 con 0x7f61e01005c0 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0196980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54474/0 (socket says 172.21.15.116:54474) 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61de59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 0x7f61e019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61ded9d640 1 -- 172.21.15.116:0/1839798543 learned_addr learned my addr 172.21.15.116:0/1839798543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61d7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e0100f80 0x7f61e019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61ded9d640 1 -- 172.21.15.116:0/1839798543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 msgr2=0x7f61e019c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.659+0000 7f61ded9d640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 0x7f61e019c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61ded9d640 1 -- 172.21.15.116:0/1839798543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e0100f80 msgr2=0x7f61e019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61ded9d640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e0100f80 0x7f61e019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61ded9d640 1 -- 172.21.15.116:0/1839798543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61d0009560 con 0x7f61e0106660 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61de59c640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 0x7f61e019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61ded9d640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0196980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f61d800eca0 tx=0x7f61d800b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61d77fe640 1 -- 172.21.15.116:0/1839798543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d8013070 con 0x7f61e0106660 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61d77fe640 1 -- 172.21.15.116:0/1839798543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d8005020 con 0x7f61e0106660 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e01975e0 con 0x7f61e0106660 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61d77fe640 1 -- 172.21.15.116:0/1839798543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d80099b0 con 0x7f61e0106660 2024-03-24T09:23:16.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.660+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e0069ee0 con 0x7f61e0106660 2024-03-24T09:23:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.662+0000 7f61d77fe640 1 -- 172.21.15.116:0/1839798543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f61d8005190 con 0x7f61e0106660 2024-03-24T09:23:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.662+0000 7f61d77fe640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61cc0a2350 0x7f61cc0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.662+0000 7f61e758b740 1 Processor -- start 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61de59c640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61cc0a2350 0x7f61cc0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61d77fe640 1 -- 172.21.15.116:0/1839798543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61d80cffd0 con 0x7f61e0106660 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 -- start start 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac1f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac283b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac2ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b4ab8d840 con 0x557b4ac1f3f0 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b4ab8ad70 con 0x557b4ac288f0 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61e758b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b4ab86550 con 0x557b4ac1fd10 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac1f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac2ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61d7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac1f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54482/0 (socket says 172.21.15.116:54482) 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61de59c640 1 -- 172.21.15.116:0/1689459103 learned_addr learned my addr 172.21.15.116:0/1689459103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac2ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:51960/0 (socket says 172.21.15.116:51960) 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61d7fff640 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac283b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41940/0 (socket says 172.21.15.116:41940) 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61de59c640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61cc0a2350 0x7f61cc0a4810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f61d002c170 tx=0x7f61d0000bc0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 -- 172.21.15.116:0/1689459103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 msgr2=0x557b4ac283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac283b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 -- 172.21.15.116:0/1689459103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 msgr2=0x557b4ac1f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac1f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61ded9d640 1 -- 172.21.15.116:0/1689459103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b4ab1bde0 con 0x557b4ac288f0 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.663+0000 7f61de59c640 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac1f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61ded9d640 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac2ace0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61d80cd1f0 tx=0x7f61d809bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bfedfa8a42090a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61d4ff9640 1 -- 172.21.15.116:0/1689459103 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d80a0510 con 0x557b4ac288f0 2024-03-24T09:23:16.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61d4ff9640 1 -- 172.21.15.116:0/1689459103 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d80a4b60 con 0x557b4ac288f0 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61d4ff9640 1 -- 172.21.15.116:0/1689459103 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d80cd730 con 0x557b4ac288f0 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 -- 172.21.15.116:0/1689459103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 msgr2=0x557b4ac2ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac2ace0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61d80cd1f0 tx=0x7f61d809bfb0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 -- 172.21.15.116:0/1689459103 shutdown_connections 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac2ace0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 --2- 172.21.15.116:0/1689459103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac1f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.664+0000 7f61e758b740 1 -- 172.21.15.116:0/1689459103 >> 172.21.15.116:0/1689459103 conn(0x557b4ac19b20 msgr2=0x557b4ac19f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 -- 172.21.15.116:0/1689459103 shutdown_connections 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 -- 172.21.15.116:0/1689459103 wait complete. 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 -- start start 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 Processor -- start 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1f3f0 0x557b4ac25980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 0x557b4ac25ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac26400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b4a859430 con 0x557b4ac1fd10 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b4a815e00 con 0x557b4ac1f3f0 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b4ab0b590 con 0x557b4ac288f0 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61e758b740 10 client.0 _ll_register_callbacks cb 0x557b4ab7afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 0x557b4ac25ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 0x557b4ac25ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54486/0 (socket says 172.21.15.116:54486) 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 -- 172.21.15.116:0/2747001642 learned_addr learned my addr 172.21.15.116:0/2747001642 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61de59c640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1f3f0 0x557b4ac25980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61ded9d640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac26400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 -- 172.21.15.116:0/2747001642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 msgr2=0x557b4ac26400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac26400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 -- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1f3f0 msgr2=0x557b4ac25980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1f3f0 0x557b4ac25980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.665+0000 7f61d7fff640 1 -- 172.21.15.116:0/2747001642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e0069ee0 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61d7fff640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 0x557b4ac25ec0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f61c80126e0 tx=0x7f61c8012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3a411504253daf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2747001642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c8017070 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2747001642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61c8012190 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b4a8857e0 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2747001642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f61c800f920 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 10 client.16065 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b4ab84bd0 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2747001642 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557b4a8857e0 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2747001642 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f61c8010af0 con 0x557b4ac1fd10 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 10 client.16065 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 20 client.16065 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac38b70 0x557b4ac3af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 4 client.16065 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.666+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b4a8c2720 con 0x557b4ac38b70 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.667+0000 7f61de59c640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac38b70 0x557b4ac3af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.667+0000 7f61de59c640 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac38b70 0x557b4ac3af90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3839d3fa583132e2 server_cookie=92b77408c96a87a4 in_seq=0 out_seq=0 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.667+0000 7f61c5ffb640 10 client.16065 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2747001642 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b4a8c2720 con 0x557b4ac38b70 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61c5ffb640 10 client.16065 handle_command_reply: tid=0 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 client.16065 shutdown 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 msgr2=0x557b4ac25ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 0x557b4ac25ec0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f61c80126e0 tx=0x7f61c8012ba0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 shutdown_connections 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac26400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1fd10 0x557b4ac25ec0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac38b70 0x557b4ac3af90 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 --2- 172.21.15.116:0/2747001642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1f3f0 0x557b4ac25980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61c5ffb640 0 client.16065 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 >> 172.21.15.116:0/2747001642 conn(0x557b4ac19b20 msgr2=0x557b4ac1b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 shutdown_connections 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 1 -- 172.21.15.116:0/2747001642 wait complete. 2024-03-24T09:23:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.668+0000 7f61e758b740 20 client.16065 trim_cache size 0 max 16384 2024-03-24T09:23:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 Processor -- start 2024-03-24T09:23:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 -- start start 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac34180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1fd10 0x557b4ac26220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac18b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b4ac37260 con 0x557b4ac1f3f0 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b4ac0ae20 con 0x557b4ac1fd10 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61e758b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b4ac153e0 con 0x557b4ac288f0 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac34180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac34180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54498/0 (socket says 172.21.15.116:54498) 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.681+0000 7f61de59c640 1 -- 172.21.15.116:0/3139700072 learned_addr learned my addr 172.21.15.116:0/3139700072 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61ded9d640 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac18b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61d7fff640 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1fd10 0x557b4ac26220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61de59c640 1 -- 172.21.15.116:0/3139700072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 msgr2=0x557b4ac18b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61de59c640 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac18b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61de59c640 1 -- 172.21.15.116:0/3139700072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1fd10 msgr2=0x557b4ac26220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61de59c640 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1fd10 0x557b4ac26220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61de59c640 1 -- 172.21.15.116:0/3139700072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b4ab20c80 con 0x557b4ac1f3f0 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61de59c640 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac34180 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009230 tx=0x7f61d003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f9214c8c0f235f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61c7fff640 1 -- 172.21.15.116:0/3139700072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d00043b0 con 0x557b4ac1f3f0 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61c7fff640 1 -- 172.21.15.116:0/3139700072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d0004510 con 0x557b4ac1f3f0 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.682+0000 7f61c7fff640 1 -- 172.21.15.116:0/3139700072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d00051e0 con 0x557b4ac1f3f0 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- 172.21.15.116:0/3139700072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 msgr2=0x557b4ac34180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac34180 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009230 tx=0x7f61d003f000 comp rx=0 tx=0).stop 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- 172.21.15.116:0/3139700072 shutdown_connections 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac288f0 0x557b4ac18b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac1fd10 0x557b4ac26220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- 172.21.15.116:0/3139700072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac34180 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- 172.21.15.116:0/3139700072 >> 172.21.15.116:0/3139700072 conn(0x557b4ac19b20 msgr2=0x557b4ac18430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- 172.21.15.116:0/3139700072 shutdown_connections 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- 172.21.15.116:0/3139700072 wait complete. 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- start start 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 Processor -- start 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac2c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac2cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac9ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b4ac2d440 con 0x557b4ac1f3f0 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b4ac9e410 con 0x557b4ac288f0 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b4ac9e5d0 con 0x557b4ac1fd10 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61e758b740 10 client.0 _ll_register_callbacks cb 0x7f61c8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac2c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61d7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac2cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac2c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54500/0 (socket says 172.21.15.116:54500) 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61de59c640 1 -- 172.21.15.116:0/2073818720 learned_addr learned my addr 172.21.15.116:0/2073818720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.683+0000 7f61ded9d640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac9ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61de59c640 1 -- 172.21.15.116:0/2073818720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 msgr2=0x557b4ac2cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61de59c640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac2cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61de59c640 1 -- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 msgr2=0x557b4ac9ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61de59c640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac9ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61de59c640 1 -- 172.21.15.116:0/2073818720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61d0030070 con 0x557b4ac1f3f0 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61d7fff640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac2cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61ded9d640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac9ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61de59c640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac2c7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f61d0036040 tx=0x7f61d0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b542bd168c0ff31f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61c5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2073818720 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d0005230 con 0x557b4ac1f3f0 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2073818720 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61d0047040 con 0x557b4ac1f3f0 2024-03-24T09:23:16.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61d80a4b60 con 0x557b4ac1f3f0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2073818720 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f61d0005390 con 0x557b4ac1f3f0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 10 client.16071 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557b4ab46920 con 0x557b4ac1f3f0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2073818720 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61d80a4b60 con 0x557b4ac1f3f0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2073818720 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f61d004e4e0 con 0x557b4ac1f3f0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 10 client.16071 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 10 client.16071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 20 client.16071 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac9fa70 0x557b4aca1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 4 client.16071 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.684+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b4a911890 con 0x557b4ac9fa70 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61d7fff640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac9fa70 0x557b4aca1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61c5ffb640 10 client.16071 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61d7fff640 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac9fa70 0x557b4aca1e90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45da96c90c94ba8 server_cookie=33456408543ffbd9 in_seq=0 out_seq=0 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61c5ffb640 1 -- 172.21.15.116:0/2073818720 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b4a911890 con 0x557b4ac9fa70 2024-03-24T09:23:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61c5ffb640 10 client.16071 handle_command_reply: tid=0 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 client.16071 shutdown 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 msgr2=0x557b4ac2c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac2c7a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f61d0036040 tx=0x7f61d0009260 comp rx=0 tx=0).stop 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 shutdown_connections 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557b4ac9fa70 0x557b4aca1e90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b4ac288f0 0x557b4ac9ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b4ac1fd10 0x557b4ac2cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61c5ffb640 0 client.16071 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 --2- 172.21.15.116:0/2073818720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b4ac1f3f0 0x557b4ac2c7a0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 >> 172.21.15.116:0/2073818720 conn(0x557b4ac19b20 msgr2=0x557b4ac1b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 shutdown_connections 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 1 -- 172.21.15.116:0/2073818720 wait complete. 2024-03-24T09:23:16.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.685+0000 7f61e758b740 20 client.16071 trim_cache size 0 max 16384 2024-03-24T09:23:16.694 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:16.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61cc0a2350 msgr2=0x7f61cc0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61cc0a2350 0x7f61cc0a4810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f61d002c170 tx=0x7f61d0000bc0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 msgr2=0x7f61e0196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0196980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f61d800eca0 tx=0x7f61d800b690 comp rx=0 tx=0).stop 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 shutdown_connections 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61e0106660 0x7f61e0196980 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61e0100f80 0x7f61e019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61cc0a2350 0x7f61cc0a4810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 --2- 172.21.15.116:0/1839798543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61e01005c0 0x7f61e019c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 >> 172.21.15.116:0/1839798543 conn(0x7f61e0076990 msgr2=0x7f61e00fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 shutdown_connections 2024-03-24T09:23:16.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:16.686+0000 7f61e49b5640 1 -- 172.21.15.116:0/1839798543 wait complete. 2024-03-24T09:23:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:16 smithi143 ceph-mon[35965]: pgmap v38: 129 pgs: 129 active+clean; 22 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 657 KiB/s wr, 119 op/s 2024-03-24T09:23:16.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:16 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3121486867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:23:16.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:16 smithi117 ceph-mon[35788]: pgmap v38: 129 pgs: 129 active+clean; 22 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 657 KiB/s wr, 119 op/s 2024-03-24T09:23:16.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:16 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3121486867' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:23:17.697 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:23:17.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:23:17.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 -- 172.21.15.116:0/3749885650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 msgr2=0x7f44ac10e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 --2- 172.21.15.116:0/3749885650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac10e570 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009cd0 tx=0x7f44a402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 -- 172.21.15.116:0/3749885650 shutdown_connections 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 --2- 172.21.15.116:0/3749885650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac10e570 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 --2- 172.21.15.116:0/3749885650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac1016a0 0x7f44ac101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 --2- 172.21.15.116:0/3749885650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac1076b0 0x7f44ac107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 -- 172.21.15.116:0/3749885650 >> 172.21.15.116:0/3749885650 conn(0x7f44ac078b10 msgr2=0x7f44ac078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 -- 172.21.15.116:0/3749885650 shutdown_connections 2024-03-24T09:23:17.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.814+0000 7f44b0c42640 1 -- 172.21.15.116:0/3749885650 wait complete. 2024-03-24T09:23:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 Processor -- start 2024-03-24T09:23:17.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 -- start start 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac1016a0 0x7f44ac19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac1076b0 0x7f44ac197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac198080 con 0x7f44ac102060 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac1981f0 con 0x7f44ac1076b0 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44b0c42640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac1983b0 con 0x7f44ac1016a0 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.815+0000 7f44a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54522/0 (socket says 172.21.15.116:54522) 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 -- 172.21.15.116:0/1541762324 learned_addr learned my addr 172.21.15.116:0/1541762324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44aa59c640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac1016a0 0x7f44ac19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44aad9d640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac1076b0 0x7f44ac197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 -- 172.21.15.116:0/1541762324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac1016a0 msgr2=0x7f44ac19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac1016a0 0x7f44ac19d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 -- 172.21.15.116:0/1541762324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac1076b0 msgr2=0x7f44ac197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac1076b0 0x7f44ac197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 -- 172.21.15.116:0/1541762324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a4009900 con 0x7f44ac102060 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a3fff640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac19ddc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f449400e9f0 tx=0x7f449400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a37fe640 1 -- 172.21.15.116:0/1541762324 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494009700 con 0x7f44ac102060 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac198690 con 0x7f44ac102060 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a37fe640 1 -- 172.21.15.116:0/1541762324 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f449400cd80 con 0x7f44ac102060 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44a37fe640 1 -- 172.21.15.116:0/1541762324 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4494018840 con 0x7f44ac102060 2024-03-24T09:23:17.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.816+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac1a4920 con 0x7f44ac102060 2024-03-24T09:23:17.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44a37fe640 1 -- 172.21.15.116:0/1541762324 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f44940189a0 con 0x7f44ac102060 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 Processor -- start 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 -- start start 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72019b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d72022720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72025050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555d71f87530 con 0x555d7201a080 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555d71f84a60 con 0x555d72019760 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.819+0000 7f44b381c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555d71f80240 con 0x555d72022c60 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72019b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d72022720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72019b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:52020/0 (socket says 172.21.15.116:52020) 2024-03-24T09:23:17.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aad9d640 1 -- 172.21.15.116:0/196401999 learned_addr learned my addr 172.21.15.116:0/196401999 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44a3fff640 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72025050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44a37fe640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44980a2350 0x7f44980a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 -- 172.21.15.116:0/196401999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 msgr2=0x555d72025050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72025050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aad9d640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44980a2350 0x7f44980a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 -- 172.21.15.116:0/196401999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 msgr2=0x555d72019b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72019b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 -- 172.21.15.116:0/196401999 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d71f7e8c0 con 0x555d7201a080 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44a37fe640 1 -- 172.21.15.116:0/1541762324 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4494014070 con 0x7f44ac102060 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aa59c640 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d72022720 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f449c009bd0 tx=0x7f449c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4153958027ab009b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44a0ff9640 1 -- 172.21.15.116:0/196401999 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c004050 con 0x555d7201a080 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44aad9d640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44980a2350 0x7f44980a4810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f44a4032570 tx=0x7f44a402c0f0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44a0ff9640 1 -- 172.21.15.116:0/196401999 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f449c005020 con 0x555d7201a080 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.820+0000 7f44a0ff9640 1 -- 172.21.15.116:0/196401999 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c004bc0 con 0x555d7201a080 2024-03-24T09:23:17.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 -- 172.21.15.116:0/196401999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 msgr2=0x555d72022720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d72022720 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f449c009bd0 tx=0x7f449c011810 comp rx=0 tx=0).stop 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 -- 172.21.15.116:0/196401999 shutdown_connections 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72025050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d72022720 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 --2- 172.21.15.116:0/196401999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72019b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 -- 172.21.15.116:0/196401999 >> 172.21.15.116:0/196401999 conn(0x555d72013e90 msgr2=0x555d72014270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 -- 172.21.15.116:0/196401999 shutdown_connections 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 -- 172.21.15.116:0/196401999 wait complete. 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.821+0000 7f44b381c740 1 -- start start 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 Processor -- start 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 0x555d7201fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72020180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72022c60 0x555d720206c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555d71f24f80 con 0x555d72019760 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555d71c534b0 con 0x555d72022c60 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555d71f05ad0 con 0x555d7201a080 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44b381c740 10 client.0 _ll_register_callbacks cb 0x555d71f38b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 0x555d7201fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 0x555d7201fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54540/0 (socket says 172.21.15.116:54540) 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44a3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72020180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 -- 172.21.15.116:0/4104430534 learned_addr learned my addr 172.21.15.116:0/4104430534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aad9d640 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72022c60 0x555d720206c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 -- 172.21.15.116:0/4104430534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 msgr2=0x555d72020180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72020180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 -- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72022c60 msgr2=0x555d720206c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72022c60 0x555d720206c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 -- 172.21.15.116:0/4104430534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f449c009830 con 0x555d72019760 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f44aa59c640 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 0x555d7201fc40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f449c0048a0 tx=0x7f449c004b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3501e16248e147a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f4491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.822+0000 7f4491ffb640 1 -- 172.21.15.116:0/4104430534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c004050 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f4491ffb640 1 -- 172.21.15.116:0/4104430534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f449c0041b0 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555d71fa47a0 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f4491ffb640 1 -- 172.21.15.116:0/4104430534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f449c004370 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 10 client.16083 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555d71f366d0 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f4491ffb640 1 -- 172.21.15.116:0/4104430534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555d71fa47a0 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f4491ffb640 1 -- 172.21.15.116:0/4104430534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f449c021040 con 0x555d72019760 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 10 client.16083 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 20 client.16083 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d72032cb0 0x555d720350f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 4 client.16083 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:17.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d71f761f0 con 0x555d72032cb0 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.823+0000 7f44a3fff640 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d72032cb0 0x555d720350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f4491ffb640 10 client.16083 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44a3fff640 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d72032cb0 0x555d720350f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=989cf66def7bede2 server_cookie=aae1de847a2adc46 in_seq=0 out_seq=0 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f4491ffb640 1 -- 172.21.15.116:0/4104430534 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d71f761f0 con 0x555d72032cb0 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f4491ffb640 10 client.16083 handle_command_reply: tid=0 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 client.16083 shutdown 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 msgr2=0x555d7201fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 0x555d7201fc40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f449c0048a0 tx=0x7f449c004b70 comp rx=0 tx=0).stop 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 shutdown_connections 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d72032cb0 0x555d720350f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72022c60 0x555d720206c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72020180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 --2- 172.21.15.116:0/4104430534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72019760 0x555d7201fc40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f4491ffb640 0 client.16083 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:17.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 >> 172.21.15.116:0/4104430534 conn(0x555d72013e90 msgr2=0x555d72015be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 shutdown_connections 2024-03-24T09:23:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.824+0000 7f44b381c740 1 -- 172.21.15.116:0/4104430534 wait complete. 2024-03-24T09:23:17.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.825+0000 7f44b381c740 20 client.16083 trim_cache size 0 max 16384 2024-03-24T09:23:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.841+0000 7f44b381c740 1 Processor -- start 2024-03-24T09:23:17.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.841+0000 7f44b381c740 1 -- start start 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44b381c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72022370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44b381c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d7202e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44b381c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72012d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44b381c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555d72012350 con 0x555d7201a080 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44b381c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555d72025620 con 0x555d72019760 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44b381c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555d7200f620 con 0x555d72022c60 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d7202e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44aa59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72022370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d7202e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54556/0 (socket says 172.21.15.116:54556) 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 -- 172.21.15.116:0/2250092552 learned_addr learned my addr 172.21.15.116:0/2250092552 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44aad9d640 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72012d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 -- 172.21.15.116:0/2250092552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 msgr2=0x555d72012d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72012d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 -- 172.21.15.116:0/2250092552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 msgr2=0x555d72022370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72022370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 -- 172.21.15.116:0/2250092552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d71f1ac60 con 0x555d7201a080 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44aa59c640 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72022370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f44a3fff640 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d7202e550 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f449400c6a0 tx=0x7f44940d0ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7690b1c6d0cdfa63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f4493fff640 1 -- 172.21.15.116:0/2250092552 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449409aea0 con 0x555d7201a080 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f4493fff640 1 -- 172.21.15.116:0/2250092552 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f449409ef40 con 0x555d7201a080 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.842+0000 7f4493fff640 1 -- 172.21.15.116:0/2250092552 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449409a5e0 con 0x555d7201a080 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- 172.21.15.116:0/2250092552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 msgr2=0x555d7202e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d7202e550 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f449400c6a0 tx=0x7f44940d0ce0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- 172.21.15.116:0/2250092552 shutdown_connections 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d72022c60 0x555d72012d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d7201a080 0x555d7202e550 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- 172.21.15.116:0/2250092552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72022370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- 172.21.15.116:0/2250092552 >> 172.21.15.116:0/2250092552 conn(0x555d72013e90 msgr2=0x555d72012660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- 172.21.15.116:0/2250092552 shutdown_connections 2024-03-24T09:23:17.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- 172.21.15.116:0/2250092552 wait complete. 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- start start 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 Processor -- start 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72027b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72028090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 0x555d720285d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555d72099040 con 0x555d72022c60 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555d72099180 con 0x555d72019760 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555d720992c0 con 0x555d7201a080 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44b381c740 10 client.0 _ll_register_callbacks cb 0x7f449c009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.843+0000 7f44aad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 0x555d720285d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44a3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72028090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aa59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72027b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 0x555d720285d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54562/0 (socket says 172.21.15.116:54562) 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 -- 172.21.15.116:0/405815622 learned_addr learned my addr 172.21.15.116:0/405815622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 -- 172.21.15.116:0/405815622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 msgr2=0x555d72028090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72028090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 -- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 msgr2=0x555d72027b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72027b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 -- 172.21.15.116:0/405815622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d71f366d0 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aa59c640 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72027b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44aad9d640 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 0x555d720285d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f44a402c700 tx=0x7f44a40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b95a7db4e40301 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f4491ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f4491ffb640 1 -- 172.21.15.116:0/405815622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4032a50 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f4491ffb640 1 -- 172.21.15.116:0/405815622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f44a4003970 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f449c005020 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f4491ffb640 1 -- 172.21.15.116:0/405815622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f44a4003b30 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44b381c740 10 client.16089 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x7f44ac1a4920 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.844+0000 7f4491ffb640 1 -- 172.21.15.116:0/405815622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f449c005020 con 0x555d72022c60 2024-03-24T09:23:17.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f4491ffb640 1 -- 172.21.15.116:0/405815622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f44a4042420 con 0x555d72022c60 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44b381c740 10 client.16089 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44b381c740 10 client.16089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44b381c740 20 client.16089 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44b381c740 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d7209ab60 0x555d7209cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44b381c740 4 client.16089 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d71c7f860 con 0x555d7209ab60 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44aa59c640 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d7209ab60 0x555d7209cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f4491ffb640 10 client.16089 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.845+0000 7f44aa59c640 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d7209ab60 0x555d7209cf80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5551fadd9d133aa4 server_cookie=dfab7442641118a4 in_seq=0 out_seq=0 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.846+0000 7f4491ffb640 1 -- 172.21.15.116:0/405815622 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d71c7f860 con 0x555d7209ab60 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.846+0000 7f4491ffb640 10 client.16089 handle_command_reply: tid=0 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.846+0000 7f44b381c740 1 client.16089 shutdown 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 msgr2=0x555d720285d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 0x555d720285d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f44a402c700 tx=0x7f44a40095c0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 shutdown_connections 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555d72022c60 0x555d720285d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555d7201a080 0x555d72028090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555d7209ab60 0x555d7209cf80 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 --2- 172.21.15.116:0/405815622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555d72019760 0x555d72027b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 >> 172.21.15.116:0/405815622 conn(0x555d72013e90 msgr2=0x555d72012ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f4491ffb640 0 client.16089 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 shutdown_connections 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 1 -- 172.21.15.116:0/405815622 wait complete. 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b381c740 20 client.16089 trim_cache size 0 max 16384 2024-03-24T09:23:17.853 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:23:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:23:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "64ba0957-78f8-41e7-8b29-d085c06b3937", 2024-03-24T09:23:17.854 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:23:17.854 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:23:17.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44980a2350 msgr2=0x7f44980a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44980a2350 0x7f44980a4810 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f44a4032570 tx=0x7f44a402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 msgr2=0x7f44ac19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac19ddc0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f449400e9f0 tx=0x7f449400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 shutdown_connections 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac1076b0 0x7f44ac197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac102060 0x7f44ac19ddc0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44980a2350 0x7f44980a4810 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 --2- 172.21.15.116:0/1541762324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac1016a0 0x7f44ac19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 >> 172.21.15.116:0/1541762324 conn(0x7f44ac078b10 msgr2=0x7f44ac1040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 shutdown_connections 2024-03-24T09:23:17.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.847+0000 7f44b0c42640 1 -- 172.21.15.116:0/1541762324 wait complete. 2024-03-24T09:23:17.858 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:23:17.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 -- 172.21.15.116:0/2654185631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec100f80 msgr2=0x7f29ec10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 --2- 172.21.15.116:0/2654185631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec100f80 0x7f29ec10d490 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009cd0 tx=0x7f29e402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 -- 172.21.15.116:0/2654185631 shutdown_connections 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 --2- 172.21.15.116:0/2654185631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec100f80 0x7f29ec10d490 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 --2- 172.21.15.116:0/2654185631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec1005c0 0x7f29ec100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 --2- 172.21.15.116:0/2654185631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec106660 0x7f29ec106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 -- 172.21.15.116:0/2654185631 >> 172.21.15.116:0/2654185631 conn(0x7f29ec076990 msgr2=0x7f29ec076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 -- 172.21.15.116:0/2654185631 shutdown_connections 2024-03-24T09:23:17.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.972+0000 7f29f0f97640 1 -- 172.21.15.116:0/2654185631 wait complete. 2024-03-24T09:23:17.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 Processor -- start 2024-03-24T09:23:17.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 -- start start 2024-03-24T09:23:17.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec1005c0 0x7f29ec19c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec100f80 0x7f29ec19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 0x7f29ec1968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec196f30 con 0x7f29ec106660 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec1970a0 con 0x7f29ec1005c0 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29f0f97640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec197260 con 0x7f29ec100f80 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 0x7f29ec1968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec1005c0 0x7f29ec19c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 0x7f29ec1968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54588/0 (socket says 172.21.15.116:54588) 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 -- 172.21.15.116:0/3983069538 learned_addr learned my addr 172.21.15.116:0/3983069538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29e3fff640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec100f80 0x7f29ec19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 -- 172.21.15.116:0/3983069538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec100f80 msgr2=0x7f29ec19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec100f80 0x7f29ec19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 -- 172.21.15.116:0/3983069538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec1005c0 msgr2=0x7f29ec19c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec1005c0 0x7f29ec19c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ead9d640 1 -- 172.21.15.116:0/3983069538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e4009900 con 0x7f29ec106660 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.973+0000 7f29ea59c640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec1005c0 0x7f29ec19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.974+0000 7f29ead9d640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 0x7f29ec1968e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f29e4004af0 tx=0x7f29e4032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.974+0000 7f29e37fe640 1 -- 172.21.15.116:0/3983069538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e40325d0 con 0x7f29ec106660 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.974+0000 7f29e37fe640 1 -- 172.21.15.116:0/3983069538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29e4003970 con 0x7f29ec106660 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.974+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec1974e0 con 0x7f29ec106660 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.974+0000 7f29e37fe640 1 -- 172.21.15.116:0/3983069538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e40046b0 con 0x7f29ec106660 2024-03-24T09:23:17.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.974+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec1a37d0 con 0x7f29ec106660 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.976+0000 7f29e37fe640 1 -- 172.21.15.116:0/3983069538 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 25) ==== 136990+0+0 (secure 0 0 0) 0x7f29e403b040 con 0x7f29ec106660 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.976+0000 7f29f3b70740 1 Processor -- start 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 -- start start 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29e37fe640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29d80a2350 0x7f29d80a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29e3fff640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29d80a2350 0x7f29d80a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 0x557dd5c4f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29e37fe640 1 -- 172.21.15.116:0/3983069538 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29e40f0f50 con 0x7f29ec106660 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c58190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c5aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557dd5b73f30 con 0x557dd5c586d0 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557dd5bbd810 con 0x557dd5c4f1d0 2024-03-24T09:23:17.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29f3b70740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557dd5bbad40 con 0x557dd5c4faf0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 0x557dd5c4f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c5aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 0x557dd5c4f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:52086/0 (socket says 172.21.15.116:52086) 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 -- 172.21.15.116:0/1826546120 learned_addr learned my addr 172.21.15.116:0/1826546120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29e3fff640 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c58190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 -- 172.21.15.116:0/1826546120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 msgr2=0x557dd5c58190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c58190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 -- 172.21.15.116:0/1826546120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 msgr2=0x557dd5c5aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c5aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29e3fff640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29d80a2350 0x7f29d80a4810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f29ec105cd0 tx=0x7f29d4011920 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 -- 172.21.15.116:0/1826546120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dd5b77ea0 con 0x557dd5c4f1d0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ead9d640 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c5aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.977+0000 7f29ea59c640 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 0x557dd5c4f5b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f29dc009fd0 tx=0x7f29dc011a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44121cd7f2002adc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29e0ff9640 1 -- 172.21.15.116:0/1826546120 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc016070 con 0x557dd5c4f1d0 2024-03-24T09:23:17.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29e0ff9640 1 -- 172.21.15.116:0/1826546120 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29dc00e650 con 0x557dd5c4f1d0 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29e0ff9640 1 -- 172.21.15.116:0/1826546120 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc003850 con 0x557dd5c4f1d0 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 -- 172.21.15.116:0/1826546120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 msgr2=0x557dd5c4f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 0x557dd5c4f5b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f29dc009fd0 tx=0x7f29dc011a60 comp rx=0 tx=0).stop 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 -- 172.21.15.116:0/1826546120 shutdown_connections 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c5aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c58190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1826546120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4f1d0 0x557dd5c4f5b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 -- 172.21.15.116:0/1826546120 >> 172.21.15.116:0/1826546120 conn(0x557dd5c49900 msgr2=0x557dd5c49ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 -- 172.21.15.116:0/1826546120 shutdown_connections 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.978+0000 7f29f3b70740 1 -- 172.21.15.116:0/1826546120 wait complete. 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 -- start start 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 Processor -- start 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c55c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c586d0 0x557dd5c56160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557dd5bb6520 con 0x557dd5c4f1d0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557dd5889430 con 0x557dd5c4faf0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557dd5845e00 con 0x557dd5c586d0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29f3b70740 10 client.0 _ll_register_callbacks cb 0x557dd5b54d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54602/0 (socket says 172.21.15.116:54602) 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29ea59c640 1 -- 172.21.15.116:0/4237105273 learned_addr learned my addr 172.21.15.116:0/4237105273 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29e3fff640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c55c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29ead9d640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c586d0 0x557dd5c56160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29ea59c640 1 -- 172.21.15.116:0/4237105273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c586d0 msgr2=0x557dd5c56160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.979+0000 7f29ea59c640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c586d0 0x557dd5c56160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29ea59c640 1 -- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 msgr2=0x557dd5c55c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29ea59c640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c55c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29ea59c640 1 -- 172.21.15.116:0/4237105273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29dc009c80 con 0x557dd5c4f1d0 2024-03-24T09:23:17.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29ead9d640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c586d0 0x557dd5c56160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29e3fff640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c55c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29ea59c640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c556e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f29dc004370 tx=0x7f29dc00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e634435fc999165f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29d1ffb640 1 -- 172.21.15.116:0/4237105273 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc016030 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29d1ffb640 1 -- 172.21.15.116:0/4237105273 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29dc005260 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557dd58b57e0 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29d1ffb640 1 -- 172.21.15.116:0/4237105273 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f29dc00ee10 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 10 client.16101 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557dd5bb4ba0 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29d1ffb640 1 -- 172.21.15.116:0/4237105273 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557dd58b57e0 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29d1ffb640 1 -- 172.21.15.116:0/4237105273 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f29dc0035f0 con 0x557dd5c4f1d0 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 10 client.16101 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 10 client.16101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 20 client.16101 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:17.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5c6cfe0 0x557dd5c6f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:17.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 4 client.16101 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:17.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.980+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557dd58f2720 con 0x557dd5c6cfe0 2024-03-24T09:23:17.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.981+0000 7f29e3fff640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5c6cfe0 0x557dd5c6f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:17.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.981+0000 7f29d1ffb640 10 client.16101 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:17.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.981+0000 7f29e3fff640 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5c6cfe0 0x557dd5c6f440 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a55a79561840cba9 server_cookie=e293a6f50d215c5e in_seq=0 out_seq=0 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29d1ffb640 1 -- 172.21.15.116:0/4237105273 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557dd58f2720 con 0x557dd5c6cfe0 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29d1ffb640 10 client.16101 handle_command_reply: tid=0 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 client.16101 shutdown 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 msgr2=0x557dd5c556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c556e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f29dc004370 tx=0x7f29dc00ec60 comp rx=0 tx=0).stop 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 shutdown_connections 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c586d0 0x557dd5c56160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5c6cfe0 0x557dd5c6f440 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c55c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 --2- 172.21.15.116:0/4237105273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c556e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:17.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29d1ffb640 0 client.16101 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:17.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 >> 172.21.15.116:0/4237105273 conn(0x557dd5c49900 msgr2=0x557dd5c4b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:17.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 shutdown_connections 2024-03-24T09:23:17.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 1 -- 172.21.15.116:0/4237105273 wait complete. 2024-03-24T09:23:17.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:17.983+0000 7f29f3b70740 20 client.16101 trim_cache size 0 max 16384 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.000+0000 7f29f3b70740 1 Processor -- start 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.000+0000 7f29f3b70740 1 -- start start 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c57e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c586d0 0x557dd5c489e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29f3b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557dd5b3b430 con 0x557dd5c4f1d0 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29f3b70740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557dd5c4a700 con 0x557dd5c586d0 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29f3b70740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557dd5c566b0 con 0x557dd5c4faf0 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c57e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c57e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54604/0 (socket says 172.21.15.116:54604) 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 -- 172.21.15.116:0/1958881668 learned_addr learned my addr 172.21.15.116:0/1958881668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ead9d640 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c586d0 0x557dd5c489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29e3fff640 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 -- 172.21.15.116:0/1958881668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 msgr2=0x557dd5c683f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c683f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 -- 172.21.15.116:0/1958881668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c586d0 msgr2=0x557dd5c489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c586d0 0x557dd5c489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 -- 172.21.15.116:0/1958881668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557dd5b50b20 con 0x557dd5c4f1d0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29ea59c640 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c57e10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0058c0 tx=0x7f29dc005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85a56ab8c76f9776 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29d3fff640 1 -- 172.21.15.116:0/1958881668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc016040 con 0x557dd5c4f1d0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29d3fff640 1 -- 172.21.15.116:0/1958881668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29dc017040 con 0x557dd5c4f1d0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.001+0000 7f29d3fff640 1 -- 172.21.15.116:0/1958881668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29dc0037b0 con 0x557dd5c4f1d0 2024-03-24T09:23:18.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 -- 172.21.15.116:0/1958881668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 msgr2=0x557dd5c57e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c57e10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f29dc0058c0 tx=0x7f29dc005db0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 -- 172.21.15.116:0/1958881668 shutdown_connections 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c586d0 0x557dd5c489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4faf0 0x557dd5c683f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 --2- 172.21.15.116:0/1958881668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c4f1d0 0x557dd5c57e10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 -- 172.21.15.116:0/1958881668 >> 172.21.15.116:0/1958881668 conn(0x557dd5c49900 msgr2=0x557dd5c482d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 -- 172.21.15.116:0/1958881668 shutdown_connections 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 -- 172.21.15.116:0/1958881668 wait complete. 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 -- start start 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 Processor -- start 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.002+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4f1d0 0x557dd5c522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c52d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557dd5cd2380 con 0x557dd5c586d0 2024-03-24T09:23:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557dd5cd24c0 con 0x557dd5c4faf0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557dd5cd2600 con 0x557dd5c4f1d0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 10 client.0 _ll_register_callbacks cb 0x7f29e40b90c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c52d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c52d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54610/0 (socket says 172.21.15.116:54610) 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 -- 172.21.15.116:0/719473354 learned_addr learned my addr 172.21.15.116:0/719473354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ea59c640 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4f1d0 0x557dd5c522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 -- 172.21.15.116:0/719473354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4f1d0 msgr2=0x557dd5c522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4f1d0 0x557dd5c522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 -- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 msgr2=0x557dd5c527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 -- 172.21.15.116:0/719473354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557dd5bb4ba0 con 0x557dd5c586d0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29ead9d640 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c52d30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c6d0 tx=0x7f29e40bcf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c83fbfda83bf99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:18.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29d1ffb640 1 -- 172.21.15.116:0/719473354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e40c1630 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29dc00e650 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29d1ffb640 1 -- 172.21.15.116:0/719473354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f29e40c5bf0 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29d1ffb640 1 -- 172.21.15.116:0/719473354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f29e4004810 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 10 client.16107 fetch_fsmap learned FSMap version 30 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.003+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557dd5b76700 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29d1ffb640 1 -- 172.21.15.116:0/719473354 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29dc00e650 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29d1ffb640 1 -- 172.21.15.116:0/719473354 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8366+0+0 (secure 0 0 0) 0x7f29e40ee450 con 0x557dd5c586d0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29f3b70740 10 client.16107 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29f3b70740 10 client.16107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29f3b70740 20 client.16107 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29f3b70740 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5cd3ca0 0x557dd5cd60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29f3b70740 4 client.16107 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557dd5941890 con 0x557dd5cd3ca0 2024-03-24T09:23:18.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29ea59c640 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5cd3ca0 0x557dd5cd60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29ea59c640 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5cd3ca0 0x557dd5cd60c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33cad267424c80d8 server_cookie=bb9e369d31b12fab in_seq=0 out_seq=0 2024-03-24T09:23:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.004+0000 7f29d1ffb640 10 client.16107 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29d1ffb640 1 -- 172.21.15.116:0/719473354 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x557dd5941890 con 0x557dd5cd3ca0 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29d1ffb640 10 client.16107 handle_command_reply: tid=0 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 client.16107 shutdown 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 msgr2=0x557dd5c52d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c52d30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c6d0 tx=0x7f29e40bcf00 comp rx=0 tx=0).stop 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 shutdown_connections 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557dd5c586d0 0x557dd5c52d30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557dd5cd3ca0 0x557dd5cd60c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29d1ffb640 0 client.16107 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557dd5c4faf0 0x557dd5c527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 --2- 172.21.15.116:0/719473354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557dd5c4f1d0 0x557dd5c522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 >> 172.21.15.116:0/719473354 conn(0x557dd5c49900 msgr2=0x557dd5c55f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 shutdown_connections 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 1 -- 172.21.15.116:0/719473354 wait complete. 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.006+0000 7f29f3b70740 20 client.16107 trim_cache size 0 max 16384 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (224 inodes in the stack)", 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: "64ba0957-78f8-41e7-8b29-d085c06b3937": { 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "64ba0957-78f8-41e7-8b29-d085c06b3937", 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:23:18.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:23:18.013 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:23:18.013 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:23:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29d80a2350 msgr2=0x7f29d80a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29d80a2350 0x7f29d80a4810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f29ec105cd0 tx=0x7f29d4011920 comp rx=0 tx=0).stop 2024-03-24T09:23:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 msgr2=0x7f29ec1968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 0x7f29ec1968e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f29e4004af0 tx=0x7f29e4032f30 comp rx=0 tx=0).stop 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 shutdown_connections 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29ec106660 0x7f29ec1968e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29d80a2350 0x7f29d80a4810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec100f80 0x7f29ec19cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 --2- 172.21.15.116:0/3983069538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec1005c0 0x7f29ec19c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 >> 172.21.15.116:0/3983069538 conn(0x7f29ec076990 msgr2=0x7f29ec0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 shutdown_connections 2024-03-24T09:23:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:18.007+0000 7f29f0f97640 1 -- 172.21.15.116:0/3983069538 wait complete. 2024-03-24T09:23:18.018 INFO:tasks.cephfs.filesystem:scrub status for tag:64ba0957-78f8-41e7-8b29-d085c06b3937 - {'path': '/', 'tag': '64ba0957-78f8-41e7-8b29-d085c06b3937', 'options': 'recursive,force'} 2024-03-24T09:23:18.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: pgmap v39: 129 pgs: 129 active+clean; 28 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.3 MiB/s wr, 246 op/s 2024-03-24T09:23:18.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:23:18.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:18.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:18.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:18.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: pgmap v39: 129 pgs: 129 active+clean; 28 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.3 MiB/s wr, 246 op/s 2024-03-24T09:23:18.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:23:18.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: pgmap v39: 129 pgs: 129 active+clean; 28 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 1.3 MiB/s wr, 246 op/s 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:23:18.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:19.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:19 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:19.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:19 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:19.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:19 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:20.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: pgmap v40: 129 pgs: 129 active+clean; 28 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 1.0 MiB/s wr, 213 op/s 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.785 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:20.786 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.787 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:20 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: pgmap v40: 129 pgs: 129 active+clean; 28 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 1.0 MiB/s wr, 213 op/s 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.818 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:20 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: pgmap v40: 129 pgs: 129 active+clean; 28 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 1.0 MiB/s wr, 213 op/s 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:20.820 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:20 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:22.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:22 smithi143 ceph-mon[35965]: pgmap v41: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.6 MiB/s wr, 303 op/s 2024-03-24T09:23:22.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:22 smithi117 ceph-mon[35788]: pgmap v41: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.6 MiB/s wr, 303 op/s 2024-03-24T09:23:22.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:22 smithi116 ceph-mon[32207]: pgmap v41: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.6 MiB/s wr, 303 op/s 2024-03-24T09:23:24.575 INFO:tasks.workunit.client.0.smithi116.stderr:Updating files: 7% (321/4277) Updating files: 8% (343/4277) Updating files: 9% (385/4277) Updating files: 10% (428/4277) Updating files: 11% (471/4277) Updating files: 11% (489/4277) Updating files: 12% (514/4277) Updating files: 13% (557/4277) Updating files: 14% (599/4277) Updating files: 15% (642/4277) Updating files: 16% (685/4277) Updating files: 16% (715/4277) Updating files: 17% (728/4277) Updating files: 18% (770/4277) Updating files: 19% (813/4277) Updating files: 20% (856/4277) Updating files: 21% (899/4277) Updating files: 22% (941/4277) Updating files: 23% (984/4277) Updating files: 24% (1027/4277) Updating files: 24% (1046/4277) Updating files: 25% (1070/4277) Updating files: 26% (1113/4277) Updating files: 27% (1155/4277) Updating files: 28% (1198/4277) Updating files: 29% (1241/4277) Updating files: 29% (1283/4277) Updating files: 30% (1284/4277) Updating files: 31% (1326/4277) Updating files: 32% (1369/4277) Updating files: 33% (1412/4277) Updating files: 34% (1455/4277) Updating files: 35% (1497/4277) Updating files: 36% (1540/4277) Updating files: 36% (1572/4277) Updating files: 37% (1583/4277) Updating files: 38% (1626/4277) Updating files: 39% (1669/4277) Updating files: 40% (1711/4277) Updating files: 41% (1754/4277) Updating files: 42% (1797/4277) Updating files: 43% (1840/4277) Updating files: 44% (1882/4277) Updating files: 44% (1921/4277) Updating files: 45% (1925/4277) Updating files: 46% (1968/4277) Updating files: 47% (2011/4277) Updating files: 48% (2053/4277) Updating files: 49% (2096/4277) Updating files: 49% (2137/4277) Updating files: 50% (2139/4277) Updating files: 51% (2182/4277) Updating files: 52% (2225/4277) Updating files: 53% (2267/4277) Updating files: 54% (2310/4277) Updating files: 54% (2331/4277) Updating files: 55% (2353/4277) Updating files: 56% (2396/4277) Updating files: 57% (2438/4277) Updating files: 58% (2481/4277) Updating files: 59% (2524/4277) Updating files: 59% (2531/4277) Updating files: 60% (2567/4277) Updating files: 61% (2609/4277) Updating files: 62% (2652/4277) Updating files: 63% (2695/4277) Updating files: 64% (2738/4277) Updating files: 64% (2773/4277) Updating files: 65% (2781/4277) Updating files: 66% (2823/4277) Updating files: 67% (2866/4277) Updating files: 67% (2902/4277) Updating files: 68% (2909/4277) Updating files: 69% (2952/4277) Updating files: 70% (2994/4277) Updating files: 71% (3037/4277) Updating files: 72% (3080/4277) Updating files: 73% (3123/4277) Updating files: 73% (3142/4277) Updating files: 74% (3165/4277) Updating files: 75% (3208/4277) Updating files: 76% (3251/4277) Updating files: 77% (3294/4277) Updating files: 78% (3337/4277) Updating files: 79% (3379/4277) Updating files: 80% (3422/4277) Updating files: 80% (3460/4277) Updating files: 81% (3465/4277) Updating files: 82% (3508/4277) Updating files: 83% (3550/4277) Updating files: 84% (3593/4277) Updating files: 85% (3636/4277) Updating files: 86% (3679/4277) Updating files: 87% (3721/4277) Updating files: 87% (3736/4277) Updating files: 88% (3764/4277) Updating files: 89% (3807/4277) Updating files: 90% (3850/4277) Updating files: 91% (3893/4277) Updating files: 92% (3935/4277) Updating files: 93% (3978/4277) Updating files: 93% (3983/4277) Updating files: 94% (4021/4277) Updating files: 95% (4064/4277) Updating files: 96% (4106/4277) Updating files: 96% (4138/4277) Updating files: 97% (4149/4277) Updating files: 98% (4192/4277) Updating files: 99% (4235/4277) Updating files: 100% (4277/4277) Updating files: 100% (4277/4277), done. 2024-03-24T09:23:24.712 INFO:tasks.workunit.client.0.smithi116.stderr:+ cd xfstests-dev 2024-03-24T09:23:24.713 INFO:tasks.workunit.client.0.smithi116.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-24T09:23:24.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:24 smithi143 ceph-mon[35965]: pgmap v42: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.2 MiB/s wr, 230 op/s 2024-03-24T09:23:24.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:24 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:24.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:24 smithi117 ceph-mon[35788]: pgmap v42: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.2 MiB/s wr, 230 op/s 2024-03-24T09:23:24.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:24 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:24.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:24 smithi116 ceph-mon[32207]: pgmap v42: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.2 MiB/s wr, 230 op/s 2024-03-24T09:23:24.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:24 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:23:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:26 smithi143 ceph-mon[35965]: pgmap v43: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.2 MiB/s wr, 217 op/s 2024-03-24T09:23:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:26 smithi117 ceph-mon[35788]: pgmap v43: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.2 MiB/s wr, 217 op/s 2024-03-24T09:23:26.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:26 smithi116 ceph-mon[32207]: pgmap v43: 129 pgs: 129 active+clean; 34 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 1.2 MiB/s wr, 217 op/s 2024-03-24T09:23:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:28 smithi143 ceph-mon[35965]: pgmap v44: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 304 op/s 2024-03-24T09:23:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:28 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:28.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:28 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:28 smithi117 ceph-mon[35788]: pgmap v44: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 304 op/s 2024-03-24T09:23:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:28 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:28.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:28 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:28.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:28 smithi116 ceph-mon[32207]: pgmap v44: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 304 op/s 2024-03-24T09:23:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:28 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:28 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:29.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:29 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:29.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:29 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:29.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:29 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:30.718 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:30 smithi116 ceph-mon[32207]: pgmap v45: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 1.2 MiB/s wr, 177 op/s 2024-03-24T09:23:30.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:30 smithi143 ceph-mon[35965]: pgmap v45: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 1.2 MiB/s wr, 177 op/s 2024-03-24T09:23:30.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:30 smithi117 ceph-mon[35788]: pgmap v45: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 1.2 MiB/s wr, 177 op/s 2024-03-24T09:23:32.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:32 smithi143 ceph-mon[35965]: pgmap v46: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 244 op/s 2024-03-24T09:23:32.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:32 smithi117 ceph-mon[35788]: pgmap v46: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 244 op/s 2024-03-24T09:23:32.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:32 smithi116 ceph-mon[32207]: pgmap v46: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 244 op/s 2024-03-24T09:23:34.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:34 smithi143 ceph-mon[35965]: pgmap v47: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.0 MiB/s wr, 154 op/s 2024-03-24T09:23:34.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:34 smithi117 ceph-mon[35788]: pgmap v47: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.0 MiB/s wr, 154 op/s 2024-03-24T09:23:34.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:34 smithi116 ceph-mon[32207]: pgmap v47: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.0 MiB/s wr, 154 op/s 2024-03-24T09:23:36.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:36 smithi143 ceph-mon[35965]: pgmap v48: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.0 MiB/s wr, 154 op/s 2024-03-24T09:23:36.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:36 smithi117 ceph-mon[35788]: pgmap v48: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.0 MiB/s wr, 154 op/s 2024-03-24T09:23:36.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:36 smithi116 ceph-mon[32207]: pgmap v48: 129 pgs: 129 active+clean; 44 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 1.0 MiB/s wr, 154 op/s 2024-03-24T09:23:37.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:37 smithi143 ceph-mon[35965]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-24T09:23:37.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:37 smithi117 ceph-mon[35788]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-24T09:23:37.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:37 smithi116 ceph-mon[32207]: mgrmap e26: x(active, since 91s), standbys: z, y 2024-03-24T09:23:38.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:38 smithi143 ceph-mon[35965]: pgmap v49: 129 pgs: 129 active+clean; 50 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 249 op/s 2024-03-24T09:23:38.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:38 smithi117 ceph-mon[35788]: pgmap v49: 129 pgs: 129 active+clean; 50 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 249 op/s 2024-03-24T09:23:38.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:38 smithi116 ceph-mon[32207]: pgmap v49: 129 pgs: 129 active+clean; 50 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 249 op/s 2024-03-24T09:23:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:40 smithi143 ceph-mon[35965]: pgmap v50: 129 pgs: 129 active+clean; 50 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1019 KiB/s wr, 161 op/s 2024-03-24T09:23:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:40 smithi117 ceph-mon[35788]: pgmap v50: 129 pgs: 129 active+clean; 50 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1019 KiB/s wr, 161 op/s 2024-03-24T09:23:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:40 smithi116 ceph-mon[32207]: pgmap v50: 129 pgs: 129 active+clean; 50 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 1019 KiB/s wr, 161 op/s 2024-03-24T09:23:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:42 smithi143 ceph-mon[35965]: pgmap v51: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 299 op/s 2024-03-24T09:23:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:42 smithi117 ceph-mon[35788]: pgmap v51: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 299 op/s 2024-03-24T09:23:43.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:42 smithi116 ceph-mon[32207]: pgmap v51: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 299 op/s 2024-03-24T09:23:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:44 smithi143 ceph-mon[35965]: pgmap v52: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 232 op/s 2024-03-24T09:23:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:44 smithi117 ceph-mon[35788]: pgmap v52: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 232 op/s 2024-03-24T09:23:45.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:44 smithi116 ceph-mon[32207]: pgmap v52: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 232 op/s 2024-03-24T09:23:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:46 smithi143 ceph-mon[35965]: pgmap v53: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 232 op/s 2024-03-24T09:23:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:46 smithi117 ceph-mon[35788]: pgmap v53: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 232 op/s 2024-03-24T09:23:47.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:46 smithi116 ceph-mon[32207]: pgmap v53: 129 pgs: 129 active+clean; 56 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 232 op/s 2024-03-24T09:23:48.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:23:48.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:23:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 -- 172.21.15.116:0/3807089453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 msgr2=0x7f1a40106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 --2- 172.21.15.116:0/3807089453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a40106a40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009890 tx=0x7f1a3002bb70 comp rx=0 tx=0).stop 2024-03-24T09:23:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 -- 172.21.15.116:0/3807089453 shutdown_connections 2024-03-24T09:23:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 --2- 172.21.15.116:0/3807089453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a4010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 --2- 172.21.15.116:0/3807089453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a401005a0 0x7f1a40100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 --2- 172.21.15.116:0/3807089453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a40106a40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 -- 172.21.15.116:0/3807089453 >> 172.21.15.116:0/3807089453 conn(0x7f1a40077710 msgr2=0x7f1a40077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 -- 172.21.15.116:0/3807089453 shutdown_connections 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.154+0000 7f1a468f4640 1 -- 172.21.15.116:0/3807089453 wait complete. 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 Processor -- start 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 -- start start 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a401005a0 0x7f1a40198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a40198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a401a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40199850 con 0x7f1a40100f60 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1a401a4050 con 0x7f1a401005a0 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a468f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1a401a4190 con 0x7f1a40106660 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.155+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a40198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a401a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a401005a0 0x7f1a40198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a40198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49196/0 (socket says 172.21.15.116:49196) 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/1009619150 learned_addr learned my addr 172.21.15.116:0/1009619150 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 -- 172.21.15.116:0/1009619150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a401005a0 msgr2=0x7f1a40198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a401005a0 0x7f1a40198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 -- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 msgr2=0x7f1a40198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a40198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 -- 172.21.15.116:0/1009619150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a30009490 con 0x7f1a40106660 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a40198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a44e91640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a401a3b10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a3800ef00 tx=0x7f1a3800c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3d7fa640 1 -- 172.21.15.116:0/1009619150 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38004b20 con 0x7f1a40106660 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3d7fa640 1 -- 172.21.15.116:0/1009619150 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a38003e20 con 0x7f1a40106660 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a401a4390 con 0x7f1a40106660 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a3d7fa640 1 -- 172.21.15.116:0/1009619150 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a380047a0 con 0x7f1a40106660 2024-03-24T09:23:48.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.156+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a401a4840 con 0x7f1a40106660 2024-03-24T09:23:48.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3d7fa640 1 -- 172.21.15.116:0/1009619150 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f1a3800a040 con 0x7f1a40106660 2024-03-24T09:23:48.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 Processor -- start 2024-03-24T09:23:48.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 -- start start 2024-03-24T09:23:48.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6101d0 0x561cff6105b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff619190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff61bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cff534f30 con 0x561cff610af0 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561cff57e810 con 0x561cff6196d0 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a495b1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561cff57bd40 con 0x561cff6101d0 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff619190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff619190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49202/0 (socket says 172.21.15.116:49202) 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 -- 172.21.15.116:0/567027899 learned_addr learned my addr 172.21.15.116:0/567027899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff61bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3d7fa640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1a280a22b0 0x7f1a280a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6101d0 0x561cff6105b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 -- 172.21.15.116:0/567027899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6101d0 msgr2=0x561cff6105b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6101d0 0x561cff6105b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3d7fa640 1 -- 172.21.15.116:0/1009619150 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1a380ce200 con 0x7f1a40106660 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a44e91640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1a280a22b0 0x7f1a280a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 -- 172.21.15.116:0/567027899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 msgr2=0x561cff61bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff61bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3ffff640 1 -- 172.21.15.116:0/567027899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cff538ea0 con 0x561cff610af0 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.159+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff61bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff619190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009890 tx=0x7f1a30005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3494a35b428fc42f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a267fc640 1 -- 172.21.15.116:0/567027899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30034cb0 con 0x561cff610af0 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a44e91640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1a280a22b0 0x7f1a280a4770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1a3800cc40 tx=0x7f1a38085040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a267fc640 1 -- 172.21.15.116:0/567027899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a3002ce90 con 0x561cff610af0 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a267fc640 1 -- 172.21.15.116:0/567027899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3002c770 con 0x561cff610af0 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 -- 172.21.15.116:0/567027899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 msgr2=0x561cff619190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff619190 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009890 tx=0x7f1a30005b20 comp rx=0 tx=0).stop 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 -- 172.21.15.116:0/567027899 shutdown_connections 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff61bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff619190 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 --2- 172.21.15.116:0/567027899 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6101d0 0x561cff6105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.160+0000 7f1a495b1740 1 -- 172.21.15.116:0/567027899 >> 172.21.15.116:0/567027899 conn(0x561cff60a900 msgr2=0x561cff60ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 -- 172.21.15.116:0/567027899 shutdown_connections 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 -- 172.21.15.116:0/567027899 wait complete. 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 -- start start 2024-03-24T09:23:48.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 Processor -- start 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6101d0 0x561cff616780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff616cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff617200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cff577520 con 0x561cff610af0 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561cff24a430 con 0x561cff6101d0 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561cff206e00 con 0x561cff6196d0 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a495b1740 10 client.0 _ll_register_callbacks cb 0x561cff515d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff616cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff616cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49206/0 (socket says 172.21.15.116:49206) 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6101d0 0x561cff616780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.161+0000 7f1a3ffff640 1 -- 172.21.15.116:0/2256257550 learned_addr learned my addr 172.21.15.116:0/2256257550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff617200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3ffff640 1 -- 172.21.15.116:0/2256257550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 msgr2=0x561cff617200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff617200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3ffff640 1 -- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6101d0 msgr2=0x561cff616780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6101d0 0x561cff616780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3ffff640 1 -- 172.21.15.116:0/2256257550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4010dd20 con 0x561cff610af0 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6101d0 0x561cff616780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff616cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1a30000c00 tx=0x7f1a30034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4eab9d048f2e312 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/2256257550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a30004520 con 0x561cff610af0 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/2256257550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1a3002c710 con 0x561cff610af0 2024-03-24T09:23:48.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561cff2767e0 con 0x561cff610af0 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/2256257550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a300041e0 con 0x561cff610af0 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a495b1740 10 client.16119 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561cff575ba0 con 0x561cff610af0 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.162+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/2256257550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561cff2767e0 con 0x561cff610af0 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/2256257550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f1a300035f0 con 0x561cff610af0 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a495b1740 10 client.16119 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a495b1740 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a495b1740 20 client.16119 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a495b1740 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff629a50 0x561cff62be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a495b1740 4 client.16119 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cff2b3720 con 0x561cff629a50 2024-03-24T09:23:48.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff629a50 0x561cff62be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff629a50 0x561cff62be70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1de9a8d5372dfa3 server_cookie=4107336a7b6563f0 in_seq=0 out_seq=0 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.163+0000 7f1a1f7fe640 10 client.16119 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.165+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/2256257550 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cff2b3720 con 0x561cff629a50 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.165+0000 7f1a1f7fe640 10 client.16119 handle_command_reply: tid=0 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.165+0000 7f1a495b1740 1 client.16119 shutdown 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 msgr2=0x561cff616cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff616cc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1a30000c00 tx=0x7f1a30034670 comp rx=0 tx=0).stop 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 shutdown_connections 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff617200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff610af0 0x561cff616cc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff629a50 0x561cff62be70 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 --2- 172.21.15.116:0/2256257550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6101d0 0x561cff616780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 >> 172.21.15.116:0/2256257550 conn(0x561cff60a900 msgr2=0x561cff60c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a1f7fe640 0 client.16119 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 shutdown_connections 2024-03-24T09:23:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 1 -- 172.21.15.116:0/2256257550 wait complete. 2024-03-24T09:23:48.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.166+0000 7f1a495b1740 20 client.16119 trim_cache size 0 max 16384 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 Processor -- start 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 -- start start 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff618eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff610af0 0x561cff60ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff624f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cff4fc430 con 0x561cff6101d0 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561cff5fc500 con 0x561cff6196d0 2024-03-24T09:23:48.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a495b1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561cff60fbd0 con 0x561cff610af0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff618eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff618eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49220/0 (socket says 172.21.15.116:49220) 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/1575155713 learned_addr learned my addr 172.21.15.116:0/1575155713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff610af0 0x561cff60ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/1575155713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff610af0 msgr2=0x561cff60ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff610af0 0x561cff60ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/1575155713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 msgr2=0x561cff624f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff624f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.185+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/1575155713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cff511b20 con 0x561cff6101d0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.186+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff618eb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1a34001eb0 tx=0x7f1a34009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df1967d67d971cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.186+0000 7f1a25ffb640 1 -- 172.21.15.116:0/1575155713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400fec0 con 0x561cff6101d0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.186+0000 7f1a25ffb640 1 -- 172.21.15.116:0/1575155713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a340054d0 con 0x561cff6101d0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.186+0000 7f1a25ffb640 1 -- 172.21.15.116:0/1575155713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34025340 con 0x561cff6101d0 2024-03-24T09:23:48.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.186+0000 7f1a495b1740 1 -- 172.21.15.116:0/1575155713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 msgr2=0x561cff618eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.186+0000 7f1a495b1740 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff618eb0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1a34001eb0 tx=0x7f1a34009f90 comp rx=0 tx=0).stop 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- 172.21.15.116:0/1575155713 shutdown_connections 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff6196d0 0x561cff624f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff610af0 0x561cff60ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 --2- 172.21.15.116:0/1575155713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff618eb0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- 172.21.15.116:0/1575155713 >> 172.21.15.116:0/1575155713 conn(0x561cff60a900 msgr2=0x561cff615050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- 172.21.15.116:0/1575155713 shutdown_connections 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- 172.21.15.116:0/1575155713 wait complete. 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- start start 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 Processor -- start 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff6217c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff610af0 0x561cff612ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff613530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cff613c00 con 0x561cff6101d0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561cff613d70 con 0x561cff610af0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561cff68ed30 con 0x561cff6196d0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a495b1740 10 client.0 _ll_register_callbacks cb 0x7f1a34009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.187+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff6217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff6217c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49228/0 (socket says 172.21.15.116:49228) 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/3783643549 learned_addr learned my addr 172.21.15.116:0/3783643549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff610af0 0x561cff612ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff613530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/3783643549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 msgr2=0x561cff613530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff613530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff610af0 msgr2=0x561cff612ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff610af0 0x561cff612ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 -- 172.21.15.116:0/3783643549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a34005c50 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a3f7fe640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff6217c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1a34004c90 tx=0x7f1a34012cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df95026fe16e9bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/3783643549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a3400fec0 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/3783643549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1a34000e40 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a3002ce90 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/3783643549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a34004730 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a495b1740 10 client.16125 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561cff537700 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.188+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/3783643549 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1a3002ce90 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/3783643549 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f1a34025490 con 0x561cff6101d0 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 10 client.16125 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 20 client.16125 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff690390 0x561cff6927b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 4 client.16125 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561cff302890 con 0x561cff690390 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff690390 0x561cff6927b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a1f7fe640 10 client.16125 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff690390 0x561cff6927b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b166a9e8d0bb3d1 server_cookie=a8fddd33c19605eb in_seq=0 out_seq=0 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a1f7fe640 1 -- 172.21.15.116:0/3783643549 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x561cff302890 con 0x561cff690390 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a1f7fe640 10 client.16125 handle_command_reply: tid=0 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.189+0000 7f1a495b1740 1 client.16125 shutdown 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 msgr2=0x561cff6217c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff6217c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1a34004c90 tx=0x7f1a34012cf0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 shutdown_connections 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561cff6196d0 0x561cff613530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561cff690390 0x561cff6927b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561cff610af0 0x561cff612ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a1f7fe640 0 client.16125 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 --2- 172.21.15.116:0/3783643549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff6101d0 0x561cff6217c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 >> 172.21.15.116:0/3783643549 conn(0x561cff60a900 msgr2=0x561cff6166a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 shutdown_connections 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 1 -- 172.21.15.116:0/3783643549 wait complete. 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a495b1740 20 client.16125 trim_cache size 0 max 16384 2024-03-24T09:23:48.197 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:23:48.198 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:23:48.198 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:23:48.198 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1a280a22b0 msgr2=0x7f1a280a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a468f4640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1a280a22b0 0x7f1a280a4770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1a3800cc40 tx=0x7f1a38085040 comp rx=0 tx=0).stop 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 msgr2=0x7f1a401a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a468f4640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a401a3b10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1a3800ef00 tx=0x7f1a3800c740 comp rx=0 tx=0).stop 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.190+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 shutdown_connections 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1a280a22b0 0x7f1a280a4770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1a40106660 0x7f1a401a3b10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a40100f60 0x7f1a40198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 --2- 172.21.15.116:0/1009619150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a401005a0 0x7f1a40198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 >> 172.21.15.116:0/1009619150 conn(0x7f1a40077710 msgr2=0x7f1a400ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 shutdown_connections 2024-03-24T09:23:48.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.191+0000 7f1a468f4640 1 -- 172.21.15.116:0/1009619150 wait complete. 2024-03-24T09:23:48.202 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:23:48.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:23:48.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 -- 172.21.15.116:0/1176682930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 msgr2=0x7f7680107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 --2- 172.21.15.116:0/1176682930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680107ab0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7670009930 tx=0x7f767002bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:48.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 -- 172.21.15.116:0/1176682930 shutdown_connections 2024-03-24T09:23:48.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 --2- 172.21.15.116:0/1176682930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76801005b0 0x7f768010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 --2- 172.21.15.116:0/1176682930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f7680100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 --2- 172.21.15.116:0/1176682930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680107ab0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 -- 172.21.15.116:0/1176682930 >> 172.21.15.116:0/1176682930 conn(0x7f7680076bd0 msgr2=0x7f76800fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 -- 172.21.15.116:0/1176682930 shutdown_connections 2024-03-24T09:23:48.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 -- 172.21.15.116:0/1176682930 wait complete. 2024-03-24T09:23:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.308+0000 7f7685d3e640 1 Processor -- start 2024-03-24T09:23:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 -- start start 2024-03-24T09:23:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f768019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76801005b0 0x7f768019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7680198110 con 0x7f76801076d0 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7680198280 con 0x7f76801005b0 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f7685d3e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7680198440 con 0x7f76800ffc10 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767f7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f768019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49246/0 (socket says 172.21.15.116:49246) 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767f7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f768019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:36026/0 (socket says 172.21.15.116:36026) 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 -- 172.21.15.116:0/844797864 learned_addr learned my addr 172.21.15.116:0/844797864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767effd640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76801005b0 0x7f768019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 -- 172.21.15.116:0/844797864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 msgr2=0x7f768019d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f768019d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 -- 172.21.15.116:0/844797864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76801005b0 msgr2=0x7f768019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76801005b0 0x7f768019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 -- 172.21.15.116:0/844797864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7670009560 con 0x7f76801076d0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767f7fe640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f768019d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.309+0000 7f767ffff640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680197ac0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f767800eb10 tx=0x7f767800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.310+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678004b20 con 0x7f76801076d0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.310+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7678004c80 con 0x7f76801076d0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.310+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7680198720 con 0x7f76801076d0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.310+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678010690 con 0x7f76801076d0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.310+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76801a49b0 con 0x7f76801076d0 2024-03-24T09:23:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.311+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7680068950 con 0x7f76801076d0 2024-03-24T09:23:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.312+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7678004df0 con 0x7f76801076d0 2024-03-24T09:23:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.312+0000 7f767cff9640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76680a2230 0x7f76680a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.313+0000 7f767f7fe640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76680a2230 0x7f76680a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.313+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7678014070 con 0x7f76801076d0 2024-03-24T09:23:48.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.313+0000 7f767f7fe640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76680a2230 0x7f76680a46f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7670005ab0 tx=0x7f767002c6d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.318+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7678097260 con 0x7f76801076d0 2024-03-24T09:23:48.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.551+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7680109cc0 con 0x7f76801076d0 2024-03-24T09:23:48.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.551+0000 7f767cff9640 1 -- 172.21.15.116:0/844797864 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7f767809a920 con 0x7f76801076d0 2024-03-24T09:23:48.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:23:48.552 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:23:48.552 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:23:48.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76680a2230 msgr2=0x7f76680a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76680a2230 0x7f76680a46f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7670005ab0 tx=0x7f767002c6d0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 msgr2=0x7f7680197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680197ac0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f767800eb10 tx=0x7f767800c2d0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 shutdown_connections 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76801076d0 0x7f7680197ac0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76801005b0 0x7f768019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76680a2230 0x7f76680a46f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 --2- 172.21.15.116:0/844797864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76800ffc10 0x7f768019d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.553+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 >> 172.21.15.116:0/844797864 conn(0x7f7680076bd0 msgr2=0x7f76800fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.554+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 shutdown_connections 2024-03-24T09:23:48.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.554+0000 7f7685d3e640 1 -- 172.21.15.116:0/844797864 wait complete. 2024-03-24T09:23:48.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:23:48.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 -- 172.21.15.116:0/1156906081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 msgr2=0x7f92fc100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 --2- 172.21.15.116:0/1156906081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc100a20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009900 tx=0x7f92f002bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 -- 172.21.15.116:0/1156906081 shutdown_connections 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 --2- 172.21.15.116:0/1156906081 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92fc100f60 0x7f92fc10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 --2- 172.21.15.116:0/1156906081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc100a20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 --2- 172.21.15.116:0/1156906081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92fc106660 0x7f92fc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 -- 172.21.15.116:0/1156906081 >> 172.21.15.116:0/1156906081 conn(0x7f92fc077710 msgr2=0x7f92fc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 -- 172.21.15.116:0/1156906081 shutdown_connections 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 -- 172.21.15.116:0/1156906081 wait complete. 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 Processor -- start 2024-03-24T09:23:48.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.670+0000 7f9301977640 1 -- start start 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92fc100f60 0x7f92fc19dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92fc106660 0x7f92fc197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc198090 con 0x7f92fc1005a0 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc198200 con 0x7f92fc100f60 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f92fc1983c0 con 0x7f92fc106660 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc19d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49268/0 (socket says 172.21.15.116:49268) 2024-03-24T09:23:48.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 -- 172.21.15.116:0/3901080669 learned_addr learned my addr 172.21.15.116:0/3901080669 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92fa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92fc100f60 0x7f92fc19dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92fc106660 0x7f92fc197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 -- 172.21.15.116:0/3901080669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92fc106660 msgr2=0x7f92fc197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92fc106660 0x7f92fc197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 -- 172.21.15.116:0/3901080669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92fc100f60 msgr2=0x7f92fc19dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92fc100f60 0x7f92fc19dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 -- 172.21.15.116:0/3901080669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f0009560 con 0x7f92fc1005a0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f92faffd640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc19d840 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00e9f0 tx=0x7f92ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9300975640 1 -- 172.21.15.116:0/3901080669 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec009700 con 0x7f92fc1005a0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92fc1986a0 con 0x7f92fc1005a0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9300975640 1 -- 172.21.15.116:0/3901080669 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92ec00cd80 con 0x7f92fc1005a0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9300975640 1 -- 172.21.15.116:0/3901080669 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec018840 con 0x7f92fc1005a0 2024-03-24T09:23:48.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.671+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc1a48e0 con 0x7f92fc1005a0 2024-03-24T09:23:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.673+0000 7f9300975640 1 -- 172.21.15.116:0/3901080669 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f92ec0189a0 con 0x7f92fc1005a0 2024-03-24T09:23:48.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 Processor -- start 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 -- start start 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f9300975640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92e40a2230 0x7f92e40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f9300975640 1 -- 172.21.15.116:0/3901080669 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f92ec014070 con 0x7f92fc1005a0 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92e40a2230 0x7f92e40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c3f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c4ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e21bad840 con 0x563e21c488f0 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563e21baad70 con 0x563e21c3fd10 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f930462d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e21ba6550 con 0x563e21c3f3f0 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f92faffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c3f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f92fa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f92faffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c3f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:36034/0 (socket says 172.21.15.116:36034) 2024-03-24T09:23:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f92faffd640 1 -- 172.21.15.116:0/2989535676 learned_addr learned my addr 172.21.15.116:0/2989535676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.674+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c4ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92faffd640 1 -- 172.21.15.116:0/2989535676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 msgr2=0x563e21c483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92faffd640 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92faffd640 1 -- 172.21.15.116:0/2989535676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 msgr2=0x563e21c4ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92faffd640 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c4ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92faffd640 1 -- 172.21.15.116:0/2989535676 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e21b3bde0 con 0x563e21c3f3f0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92e40a2230 0x7f92e40a46f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f92f4005fd0 tx=0x7f92f4011810 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92faffd640 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c3f7d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f92ec097850 tx=0x7f92ec09a8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc629aed09cb71a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92e1ffb640 1 -- 172.21.15.116:0/2989535676 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec010b50 con 0x563e21c3f3f0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92e1ffb640 1 -- 172.21.15.116:0/2989535676 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92ec0cbdb0 con 0x563e21c3f3f0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.675+0000 7f92e1ffb640 1 -- 172.21.15.116:0/2989535676 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec096bd0 con 0x563e21c3f3f0 2024-03-24T09:23:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- 172.21.15.116:0/2989535676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 msgr2=0x563e21c3f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c3f7d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f92ec097850 tx=0x7f92ec09a8e0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- 172.21.15.116:0/2989535676 shutdown_connections 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c4ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- 172.21.15.116:0/2989535676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c3f7d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- 172.21.15.116:0/2989535676 >> 172.21.15.116:0/2989535676 conn(0x563e21c39b20 msgr2=0x563e21c39f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- 172.21.15.116:0/2989535676 shutdown_connections 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- 172.21.15.116:0/2989535676 wait complete. 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- start start 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 Processor -- start 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c459f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 0x563e21c45f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c488f0 0x563e21c46470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e21879430 con 0x563e21c3fd10 2024-03-24T09:23:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.676+0000 7f930462d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563e21835e00 con 0x563e21c488f0 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f930462d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e21b2b590 con 0x563e21c3f3f0 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f930462d740 10 client.0 _ll_register_callbacks cb 0x563e21b9afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 0x563e21c45f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92faffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 0x563e21c45f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49284/0 (socket says 172.21.15.116:49284) 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 -- 172.21.15.116:0/2456494799 learned_addr learned my addr 172.21.15.116:0/2456494799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c488f0 0x563e21c46470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 -- 172.21.15.116:0/2456494799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 msgr2=0x563e21c459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 -- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c488f0 msgr2=0x563e21c46470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c488f0 0x563e21c46470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 -- 172.21.15.116:0/2456494799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92fc1a48e0 con 0x563e21c3fd10 2024-03-24T09:23:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c488f0 0x563e21c46470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92faffd640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 0x563e21c45f30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009900 tx=0x7f92f002c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d356eea09340852a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92daffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92daffd640 1 -- 172.21.15.116:0/2456494799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f002cb40 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92daffd640 1 -- 172.21.15.116:0/2456494799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f92f0004250 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563e218a57e0 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f92daffd640 1 -- 172.21.15.116:0/2456494799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f92f0004410 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f930462d740 10 client.16143 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.677+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e21ba4bd0 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f92daffd640 1 -- 172.21.15.116:0/2456494799 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563e218a57e0 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f92daffd640 1 -- 172.21.15.116:0/2456494799 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f92f003c040 con 0x563e21c3fd10 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f930462d740 10 client.16143 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f930462d740 10 client.16143 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f930462d740 20 client.16143 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f930462d740 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21c5ce60 0x563e21c5f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f930462d740 4 client.16143 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e218e2720 con 0x563e21c5ce60 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.678+0000 7f92faffd640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21c5ce60 0x563e21c5f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.679+0000 7f92faffd640 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21c5ce60 0x563e21c5f280 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab67a4e6955a5b23 server_cookie=bf9496c16a1b6a23 in_seq=0 out_seq=0 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.679+0000 7f92daffd640 10 client.16143 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.679+0000 7f92daffd640 1 -- 172.21.15.116:0/2456494799 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e218e2720 con 0x563e21c5ce60 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.679+0000 7f92daffd640 10 client.16143 handle_command_reply: tid=0 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.679+0000 7f930462d740 1 client.16143 shutdown 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 msgr2=0x563e21c45f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 0x563e21c45f30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f92f0009900 tx=0x7f92f002c9d0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 shutdown_connections 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c488f0 0x563e21c46470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21c5ce60 0x563e21c5f280 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3fd10 0x563e21c45f30 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f92daffd640 0 client.16143 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 --2- 172.21.15.116:0/2456494799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3f3f0 0x563e21c459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 >> 172.21.15.116:0/2456494799 conn(0x563e21c39b20 msgr2=0x563e21c3b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 shutdown_connections 2024-03-24T09:23:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 1 -- 172.21.15.116:0/2456494799 wait complete. 2024-03-24T09:23:48.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.680+0000 7f930462d740 20 client.16143 trim_cache size 0 max 16384 2024-03-24T09:23:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 Processor -- start 2024-03-24T09:23:48.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 -- start start 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3f3f0 0x563e21c584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3fd10 0x563e21c46290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c38b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e21c598d0 con 0x563e21c488f0 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563e21c3ab50 con 0x563e21c3f3f0 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f930462d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e21c58a10 con 0x563e21c3fd10 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f92fb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c38b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f92fb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c38b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49298/0 (socket says 172.21.15.116:49298) 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f92fb7fe640 1 -- 172.21.15.116:0/3869770730 learned_addr learned my addr 172.21.15.116:0/3869770730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.693+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3fd10 0x563e21c46290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92faffd640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3f3f0 0x563e21c584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fb7fe640 1 -- 172.21.15.116:0/3869770730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3fd10 msgr2=0x563e21c46290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3fd10 0x563e21c46290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fb7fe640 1 -- 172.21.15.116:0/3869770730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3f3f0 msgr2=0x563e21c584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3f3f0 0x563e21c584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fb7fe640 1 -- 172.21.15.116:0/3869770730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e21b40c80 con 0x563e21c488f0 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3fd10 0x563e21c46290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92faffd640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3f3f0 0x563e21c584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c38b40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f92f400fcc0 tx=0x7f92f401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ab507bb9b5c235 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92e17fa640 1 -- 172.21.15.116:0/3869770730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4023070 con 0x563e21c488f0 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92e17fa640 1 -- 172.21.15.116:0/3869770730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92f401ee00 con 0x563e21c488f0 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.694+0000 7f92e17fa640 1 -- 172.21.15.116:0/3869770730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92f4027bf0 con 0x563e21c488f0 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 -- 172.21.15.116:0/3869770730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 msgr2=0x563e21c38b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c38b40 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f92f400fcc0 tx=0x7f92f401c000 comp rx=0 tx=0).stop 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 -- 172.21.15.116:0/3869770730 shutdown_connections 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c488f0 0x563e21c38b40 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c3fd10 0x563e21c46290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 --2- 172.21.15.116:0/3869770730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3f3f0 0x563e21c584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 -- 172.21.15.116:0/3869770730 >> 172.21.15.116:0/3869770730 conn(0x563e21c39b20 msgr2=0x563e21c38430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 -- 172.21.15.116:0/3869770730 shutdown_connections 2024-03-24T09:23:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 -- 172.21.15.116:0/3869770730 wait complete. 2024-03-24T09:23:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 -- start start 2024-03-24T09:23:48.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.695+0000 7f930462d740 1 Processor -- start 2024-03-24T09:23:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 0x563e21c42620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c42b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c488f0 0x563e21c430a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563e21cc21a0 con 0x563e21c3f3f0 2024-03-24T09:23:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563e21cc22e0 con 0x563e21c3fd10 2024-03-24T09:23:48.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563e21cc2450 con 0x563e21c488f0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 10 client.0 _ll_register_callbacks cb 0x7f92f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 0x563e21c42620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 0x563e21c42620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49300/0 (socket says 172.21.15.116:49300) 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 -- 172.21.15.116:0/1613508167 learned_addr learned my addr 172.21.15.116:0/1613508167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c42b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92fb7fe640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c488f0 0x563e21c430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 -- 172.21.15.116:0/1613508167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c488f0 msgr2=0x563e21c430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c488f0 0x563e21c430a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 -- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 msgr2=0x563e21c42b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c42b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 -- 172.21.15.116:0/1613508167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e21b3bde0 con 0x563e21c3f3f0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92faffd640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 0x563e21c42620 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f92ec09ae10 tx=0x7f92ec09f0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96e5a3f0a147db58 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92daffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92daffd640 1 -- 172.21.15.116:0/1613508167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec0b91b0 con 0x563e21c3f3f0 2024-03-24T09:23:48.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92daffd640 1 -- 172.21.15.116:0/1613508167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f92ec09a2d0 con 0x563e21c3f3f0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92ec0cbdb0 con 0x563e21c3f3f0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f92daffd640 1 -- 172.21.15.116:0/1613508167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f92ec0b9310 con 0x563e21c3f3f0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 10 client.16149 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.696+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563e21ba4bd0 con 0x563e21c3f3f0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92daffd640 1 -- 172.21.15.116:0/1613508167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92ec0cbdb0 con 0x563e21c3f3f0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92daffd640 1 -- 172.21.15.116:0/1613508167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f92ec096c60 con 0x563e21c3f3f0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 10 client.16149 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 10 client.16149 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 20 client.16149 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21cc39a0 0x563e21cc5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 4 client.16149 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e21931890 con 0x563e21cc39a0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21cc39a0 0x563e21cc5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92fa7fc640 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21cc39a0 0x563e21cc5dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b5394534b0fae767 server_cookie=791a986b9db117b0 in_seq=0 out_seq=0 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92daffd640 10 client.16149 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92daffd640 1 -- 172.21.15.116:0/1613508167 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e21931890 con 0x563e21cc39a0 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f92daffd640 10 client.16149 handle_command_reply: tid=0 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.697+0000 7f930462d740 1 client.16149 shutdown 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 msgr2=0x563e21c42620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 0x563e21c42620 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f92ec09ae10 tx=0x7f92ec09f0a0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 shutdown_connections 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563e21c488f0 0x563e21c430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21c3fd10 0x563e21c42b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563e21cc39a0 0x563e21cc5dc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 --2- 172.21.15.116:0/1613508167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563e21c3f3f0 0x563e21c42620 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 >> 172.21.15.116:0/1613508167 conn(0x563e21c39b20 msgr2=0x563e21c3b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f92daffd640 0 client.16149 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 shutdown_connections 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 1 -- 172.21.15.116:0/1613508167 wait complete. 2024-03-24T09:23:48.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f930462d740 20 client.16149 trim_cache size 0 max 16384 2024-03-24T09:23:48.705 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:48.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92e40a2230 msgr2=0x7f92e40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92e40a2230 0x7f92e40a46f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f92f4005fd0 tx=0x7f92f4011810 comp rx=0 tx=0).stop 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 msgr2=0x7f92fc19d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc19d840 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00e9f0 tx=0x7f92ec00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 shutdown_connections 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92e40a2230 0x7f92e40a46f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92fc106660 0x7f92fc197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92fc100f60 0x7f92fc19dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 --2- 172.21.15.116:0/3901080669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92fc1005a0 0x7f92fc19d840 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.698+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 >> 172.21.15.116:0/3901080669 conn(0x7f92fc077710 msgr2=0x7f92fc0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.699+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 shutdown_connections 2024-03-24T09:23:48.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.699+0000 7f9301977640 1 -- 172.21.15.116:0/3901080669 wait complete. 2024-03-24T09:23:48.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 -- 172.21.15.116:0/3058453689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c100f60 msgr2=0x7f764c10d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 --2- 172.21.15.116:0/3058453689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c100f60 0x7f764c10d420 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f7648009900 tx=0x7f764802bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 -- 172.21.15.116:0/3058453689 shutdown_connections 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 --2- 172.21.15.116:0/3058453689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c100f60 0x7f764c10d420 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 --2- 172.21.15.116:0/3058453689 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f764c1005a0 0x7f764c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 --2- 172.21.15.116:0/3058453689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f764c106660 0x7f764c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 -- 172.21.15.116:0/3058453689 >> 172.21.15.116:0/3058453689 conn(0x7f764c077710 msgr2=0x7f764c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 -- 172.21.15.116:0/3058453689 shutdown_connections 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 -- 172.21.15.116:0/3058453689 wait complete. 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 Processor -- start 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 -- start start 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.817+0000 7f765455e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 0x7f764c19fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f764c100f60 0x7f764c19ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f764c106660 0x7f764c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f764c19a310 con 0x7f764c1005a0 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f764c19a480 con 0x7f764c106660 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f764c19a640 con 0x7f764c100f60 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 0x7f764c19fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 0x7f764c19fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49316/0 (socket says 172.21.15.116:49316) 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 -- 172.21.15.116:0/2207976310 learned_addr learned my addr 172.21.15.116:0/2207976310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f7652afb640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f764c106660 0x7f764c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f7651af9640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f764c100f60 0x7f764c19ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 -- 172.21.15.116:0/2207976310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f764c100f60 msgr2=0x7f764c19ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f764c100f60 0x7f764c19ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 -- 172.21.15.116:0/2207976310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f764c106660 msgr2=0x7f764c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f764c106660 0x7f764c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 -- 172.21.15.116:0/2207976310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7648009560 con 0x7f764c1005a0 2024-03-24T09:23:48.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76522fa640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 0x7f764c19fa70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f764400ebc0 tx=0x7f764400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76437fe640 1 -- 172.21.15.116:0/2207976310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440099b0 con 0x7f764c1005a0 2024-03-24T09:23:48.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f764c19a920 con 0x7f764c1005a0 2024-03-24T09:23:48.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76437fe640 1 -- 172.21.15.116:0/2207976310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f764400eec0 con 0x7f764c1005a0 2024-03-24T09:23:48.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f76437fe640 1 -- 172.21.15.116:0/2207976310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644018a80 con 0x7f764c1005a0 2024-03-24T09:23:48.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.818+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f764c1a69f0 con 0x7f764c1005a0 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.820+0000 7f76437fe640 1 -- 172.21.15.116:0/2207976310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7644018be0 con 0x7f764c1005a0 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.821+0000 7f7657218740 1 Processor -- start 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.821+0000 7f7657218740 1 -- start start 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.821+0000 7f76437fe640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76380a2230 0x7f76380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.821+0000 7f7652afb640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76380a2230 0x7f76380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7657218740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7657218740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7657218740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76437fe640 1 -- 172.21.15.116:0/2207976310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7644014070 con 0x7f764c1005a0 2024-03-24T09:23:48.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7657218740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561df590f840 con 0x561df59aa8f0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7657218740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561df590cd70 con 0x561df59a1d10 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7657218740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561df5908550 con 0x561df59a13f0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7652afb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:36112/0 (socket says 172.21.15.116:36112) 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 -- 172.21.15.116:0/3187571562 learned_addr learned my addr 172.21.15.116:0/3187571562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7651af9640 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 -- 172.21.15.116:0/3187571562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 msgr2=0x561df59aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59aa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 -- 172.21.15.116:0/3187571562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 msgr2=0x561df59acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 -- 172.21.15.116:0/3187571562 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561df589dde0 con 0x561df59a13f0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7652afb640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76380a2230 0x7f76380a46f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7648005ab0 tx=0x7f7648004af0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f76522fa640 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a17d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f76440a32b0 tx=0x7f7644009200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ab0cd4cb02b22e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7640bfe640 1 -- 172.21.15.116:0/3187571562 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440a37e0 con 0x561df59a13f0 2024-03-24T09:23:48.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7640bfe640 1 -- 172.21.15.116:0/3187571562 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76440b2070 con 0x561df59a13f0 2024-03-24T09:23:48.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.822+0000 7f7640bfe640 1 -- 172.21.15.116:0/3187571562 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7644097080 con 0x561df59a13f0 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 -- 172.21.15.116:0/3187571562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 msgr2=0x561df59a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a17d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f76440a32b0 tx=0x7f7644009200 comp rx=0 tx=0).stop 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 -- 172.21.15.116:0/3187571562 shutdown_connections 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 --2- 172.21.15.116:0/3187571562 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a17d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 -- 172.21.15.116:0/3187571562 >> 172.21.15.116:0/3187571562 conn(0x561df599bb20 msgr2=0x561df599bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 -- 172.21.15.116:0/3187571562 shutdown_connections 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.823+0000 7f7657218740 1 -- 172.21.15.116:0/3187571562 wait complete. 2024-03-24T09:23:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 -- start start 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 Processor -- start 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59a7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59a8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561df55db430 con 0x561df59aa8f0 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561df5597e00 con 0x561df59a1d10 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561df588d590 con 0x561df59a13f0 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7657218740 10 client.0 _ll_register_callbacks cb 0x561df58fcfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f76522fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59a8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49338/0 (socket says 172.21.15.116:49338) 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f76522fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:36126/0 (socket says 172.21.15.116:36126) 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 -- 172.21.15.116:0/1697919759 learned_addr learned my addr 172.21.15.116:0/1697919759 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7651af9640 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59a7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 -- 172.21.15.116:0/1697919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 msgr2=0x561df59a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 -- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 msgr2=0x561df59a7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59a7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.824+0000 7f7652afb640 1 -- 172.21.15.116:0/1697919759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f764c1a69f0 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7652afb640 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59a8400 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7648035000 tx=0x7f764802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91eeef96f3ade7e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7635ffb640 1 -- 172.21.15.116:0/1697919759 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764802cb90 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561df56077e0 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7635ffb640 1 -- 172.21.15.116:0/1697919759 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7648003800 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7635ffb640 1 -- 172.21.15.116:0/1697919759 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f76480039c0 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7657218740 10 client.16161 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561df5906bd0 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.825+0000 7f7635ffb640 1 -- 172.21.15.116:0/1697919759 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561df56077e0 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7635ffb640 1 -- 172.21.15.116:0/1697919759 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f764803c040 con 0x561df59aa8f0 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7657218740 10 client.16161 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7657218740 10 client.16161 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:23:48.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7657218740 20 client.16161 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7657218740 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df59baa90 0x561df59bceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7657218740 4 client.16161 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561df5644720 con 0x561df59baa90 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f76522fa640 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df59baa90 0x561df59bceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f7635ffb640 10 client.16161 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.826+0000 7f76522fa640 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df59baa90 0x561df59bceb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f0ab33e6ad0fbd21 server_cookie=94e1cd8615735efe in_seq=0 out_seq=0 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7635ffb640 1 -- 172.21.15.116:0/1697919759 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561df5644720 con 0x561df59baa90 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7635ffb640 10 client.16161 handle_command_reply: tid=0 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 client.16161 shutdown 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 msgr2=0x561df59a8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59a8400 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7648035000 tx=0x7f764802bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 shutdown_connections 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df59baa90 0x561df59bceb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59aa8f0 0x561df59a8400 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a1d10 0x561df59a7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7635ffb640 0 client.16161 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:48.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 --2- 172.21.15.116:0/1697919759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df59a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 >> 172.21.15.116:0/1697919759 conn(0x561df599bb20 msgr2=0x561df599d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 shutdown_connections 2024-03-24T09:23:48.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 1 -- 172.21.15.116:0/1697919759 wait complete. 2024-03-24T09:23:48.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.828+0000 7f7657218740 20 client.16161 trim_cache size 0 max 16384 2024-03-24T09:23:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 Processor -- start 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 -- start start 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a13f0 0x561df59b6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df59c2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59aa8f0 0x561df59a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561df599b3d0 con 0x561df59a1d10 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561df599cb20 con 0x561df59a13f0 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7657218740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561df599de30 con 0x561df59aa8f0 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7651af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df59c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7651af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df59c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49346/0 (socket says 172.21.15.116:49346) 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7651af9640 1 -- 172.21.15.116:0/630699201 learned_addr learned my addr 172.21.15.116:0/630699201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f76522fa640 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a13f0 0x561df59b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.842+0000 7f7652afb640 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59aa8f0 0x561df59a7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7651af9640 1 -- 172.21.15.116:0/630699201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59aa8f0 msgr2=0x561df59a7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7651af9640 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59aa8f0 0x561df59a7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7651af9640 1 -- 172.21.15.116:0/630699201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a13f0 msgr2=0x561df59b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7651af9640 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a13f0 0x561df59b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7651af9640 1 -- 172.21.15.116:0/630699201 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561df58a2c80 con 0x561df59a1d10 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7651af9640 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df59c2740 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f763c009ea0 tx=0x7f763c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e8bdfa0c7e0654d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7637fff640 1 -- 172.21.15.116:0/630699201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763c017070 con 0x561df59a1d10 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7637fff640 1 -- 172.21.15.116:0/630699201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f763c018040 con 0x561df59a1d10 2024-03-24T09:23:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.843+0000 7f7637fff640 1 -- 172.21.15.116:0/630699201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763c003dd0 con 0x561df59a1d10 2024-03-24T09:23:48.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- 172.21.15.116:0/630699201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 msgr2=0x561df59c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df59c2740 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f763c009ea0 tx=0x7f763c005f90 comp rx=0 tx=0).stop 2024-03-24T09:23:48.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- 172.21.15.116:0/630699201 shutdown_connections 2024-03-24T09:23:48.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59aa8f0 0x561df59a7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df59c2740 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- 172.21.15.116:0/630699201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59a13f0 0x561df59b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- 172.21.15.116:0/630699201 >> 172.21.15.116:0/630699201 conn(0x561df599bb20 msgr2=0x561df599acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- 172.21.15.116:0/630699201 shutdown_connections 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- 172.21.15.116:0/630699201 wait complete. 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- start start 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 Processor -- start 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df5a1a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df5a1a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59aa8f0 0x561df59ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561df59aed40 con 0x561df59a1d10 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561df59aeeb0 con 0x561df59aa8f0 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561df59af070 con 0x561df59a13f0 2024-03-24T09:23:48.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7657218740 10 client.0 _ll_register_callbacks cb 0x7f7648009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.844+0000 7f7651af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df5a1a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df5a1a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7651af9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df5a1a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49354/0 (socket says 172.21.15.116:49354) 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7652afb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59aa8f0 0x561df59ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7651af9640 1 -- 172.21.15.116:0/481357829 learned_addr learned my addr 172.21.15.116:0/481357829 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 -- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59aa8f0 msgr2=0x561df59ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59aa8f0 0x561df59ae7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 -- 172.21.15.116:0/481357829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 msgr2=0x561df5a1a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df5a1a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 -- 172.21.15.116:0/481357829 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f763c009b50 con 0x561df59a13f0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7651af9640 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df5a1a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f76522fa640 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df5a1a080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f764409b080 tx=0x7f7644018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb0efa505c6eed8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7635ffb640 1 -- 172.21.15.116:0/481357829 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76440a3860 con 0x561df59a13f0 2024-03-24T09:23:48.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7635ffb640 1 -- 172.21.15.116:0/481357829 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f76440993f0 con 0x561df59a13f0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76440b2070 con 0x561df59a13f0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7635ffb640 1 -- 172.21.15.116:0/481357829 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7644097760 con 0x561df59a13f0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7657218740 10 client.25885 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.845+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561df58c8920 con 0x561df59a13f0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7635ffb640 1 -- 172.21.15.116:0/481357829 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76440b2070 con 0x561df59a13f0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7635ffb640 1 -- 172.21.15.116:0/481357829 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f7644096e40 con 0x561df59a13f0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 10 client.25885 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 10 client.25885 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 20 client.25885 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df5a21d80 0x561df5a241a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 4 client.25885 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561df5693890 con 0x561df5a21d80 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7651af9640 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df5a21d80 0x561df5a241a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7635ffb640 10 client.25885 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7651af9640 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df5a21d80 0x561df5a241a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d4ca089ce4d23f24 server_cookie=574919922338bbfb in_seq=0 out_seq=0 2024-03-24T09:23:48.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7635ffb640 1 -- 172.21.15.116:0/481357829 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561df5693890 con 0x561df5a21d80 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7635ffb640 10 client.25885 handle_command_reply: tid=0 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 1 client.25885 shutdown 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.846+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 msgr2=0x561df5a1a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df5a1a080 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f764409b080 tx=0x7f7644018420 comp rx=0 tx=0).stop 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 shutdown_connections 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561df5a21d80 0x561df5a241a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561df59aa8f0 0x561df59ae7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561df59a1d10 0x561df5a1a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 --2- 172.21.15.116:0/481357829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561df59a13f0 0x561df5a1a080 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7635ffb640 0 client.25885 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 >> 172.21.15.116:0/481357829 conn(0x561df599bb20 msgr2=0x561df599ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 shutdown_connections 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 1 -- 172.21.15.116:0/481357829 wait complete. 2024-03-24T09:23:48.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f7657218740 20 client.25885 trim_cache size 0 max 16384 2024-03-24T09:23:48.854 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:48.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76380a2230 msgr2=0x7f76380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76380a2230 0x7f76380a46f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7648005ab0 tx=0x7f7648004af0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 msgr2=0x7f764c19fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 0x7f764c19fa70 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f764400ebc0 tx=0x7f764400c6a0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 shutdown_connections 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f764c106660 0x7f764c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76380a2230 0x7f76380a46f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f764c100f60 0x7f764c19ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 --2- 172.21.15.116:0/2207976310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f764c1005a0 0x7f764c19fa70 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 >> 172.21.15.116:0/2207976310 conn(0x7f764c077710 msgr2=0x7f764c068870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 shutdown_connections 2024-03-24T09:23:48.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.847+0000 7f765455e640 1 -- 172.21.15.116:0/2207976310 wait complete. 2024-03-24T09:23:48.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/1241909419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 msgr2=0x7fbb3410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/1241909419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb3410d460 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c009ca0 tx=0x7fbb2c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/1241909419 shutdown_connections 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/1241909419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb3410d460 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/1241909419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb34105190 0x7fbb34105610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/1241909419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34100bc0 0x7fbb34100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/1241909419 >> 172.21.15.116:0/1241909419 conn(0x7fbb340773d0 msgr2=0x7fbb340777e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/1241909419 shutdown_connections 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.966+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/1241909419 wait complete. 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 Processor -- start 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 -- start start 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb34100bc0 0x7fbb3419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34105190 0x7fbb3419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb34197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb34197fe0 con 0x7fbb34105b50 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbb34198150 con 0x7fbb34100bc0 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb3a7c7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbb34198310 con 0x7fbb34105190 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb38d64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb34197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb33fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb34100bc0 0x7fbb3419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34105190 0x7fbb3419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb38d64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb34197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49376/0 (socket says 172.21.15.116:49376) 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb38d64640 1 -- 172.21.15.116:0/267336597 learned_addr learned my addr 172.21.15.116:0/267336597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.967+0000 7fbb38d64640 1 -- 172.21.15.116:0/267336597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34105190 msgr2=0x7fbb3419dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb38d64640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34105190 0x7fbb3419dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb38d64640 1 -- 172.21.15.116:0/267336597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb34100bc0 msgr2=0x7fbb3419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb38d64640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb34100bc0 0x7fbb3419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb38d64640 1 -- 172.21.15.116:0/267336597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb2c009900 con 0x7fbb34105b50 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb337fe640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34105190 0x7fbb3419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb38d64640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb34197990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c005e30 tx=0x7fbb2c034a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb317fa640 1 -- 172.21.15.116:0/267336597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c032820 con 0x7fbb34105b50 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb317fa640 1 -- 172.21.15.116:0/267336597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb2c003e20 con 0x7fbb34105b50 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb34198590 con 0x7fbb34105b50 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb317fa640 1 -- 172.21.15.116:0/267336597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c004790 con 0x7fbb34105b50 2024-03-24T09:23:48.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.968+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb341a4880 con 0x7fbb34105b50 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.970+0000 7fbb317fa640 1 -- 172.21.15.116:0/267336597 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fbb2c00a040 con 0x7fbb34105b50 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 Processor -- start 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 -- start start 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fc1a840 con 0x55ab8fcac3f0 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fc17d70 con 0x55ab8fcb58f0 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fc13550 con 0x55ab8fcacd10 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb317fa640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb1c0a2230 0x7fbb1c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb38d64640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb1c0a2230 0x7fbb1c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcac7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49382/0 (socket says 172.21.15.116:49382) 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 -- 172.21.15.116:0/772549403 learned_addr learned my addr 172.21.15.116:0/772549403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb33fff640 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb317fa640 1 -- 172.21.15.116:0/267336597 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb2c0f0f00 con 0x7fbb34105b50 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 -- 172.21.15.116:0/772549403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 msgr2=0x55ab8fcb53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 -- 172.21.15.116:0/772549403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 msgr2=0x55ab8fcb7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 -- 172.21.15.116:0/772549403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab8fba8de0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb337fe640 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcac7d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbb28013be0 tx=0x7fbb28009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2509e74ac80ca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.971+0000 7fbb38d64640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb1c0a2230 0x7fbb1c0a46f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0ee460 tx=0x7fbb2c0a8180 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb1a7fc640 1 -- 172.21.15.116:0/772549403 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28013cd0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb1a7fc640 1 -- 172.21.15.116:0/772549403 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb280044e0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb1a7fc640 1 -- 172.21.15.116:0/772549403 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28003ac0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/772549403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 msgr2=0x55ab8fcac7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcac7d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbb28013be0 tx=0x7fbb28009b50 comp rx=0 tx=0).stop 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/772549403 shutdown_connections 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/772549403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcac7d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/772549403 >> 172.21.15.116:0/772549403 conn(0x55ab8fca6b20 msgr2=0x55ab8fca6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/772549403 shutdown_connections 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.972+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/772549403 wait complete. 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 -- start start 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 Processor -- start 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcb2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ab8f8e6430 con 0x55ab8fcac3f0 2024-03-24T09:23:48.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ab8f8a2e00 con 0x55ab8fcb58f0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fb98590 con 0x55ab8fcacd10 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb3d3a2740 10 client.0 _ll_register_callbacks cb 0x55ab8fc07fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcb2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcb2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49392/0 (socket says 172.21.15.116:49392) 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 -- 172.21.15.116:0/2596784550 learned_addr learned my addr 172.21.15.116:0/2596784550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb33fff640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 -- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 msgr2=0x55ab8fcb2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 -- 172.21.15.116:0/2596784550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 msgr2=0x55ab8fcb3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.973+0000 7fbb337fe640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb337fe640 1 -- 172.21.15.116:0/2596784550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb2801b040 con 0x55ab8fcac3f0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb337fe640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcb2990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbb28012010 tx=0x7fbb28003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fb02f4cf1550aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb137fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb137fe640 1 -- 172.21.15.116:0/2596784550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280055c0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb137fe640 1 -- 172.21.15.116:0/2596784550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb280046c0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ab8f9127e0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb137fe640 1 -- 172.21.15.116:0/2596784550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbb28004880 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 10 client.16179 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ab8fc11bd0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb137fe640 1 -- 172.21.15.116:0/2596784550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ab8f9127e0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb137fe640 1 -- 172.21.15.116:0/2596784550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fbb280208e0 con 0x55ab8fcac3f0 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 10 client.16179 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 10 client.16179 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 20 client.16179 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fcc5900 0x55ab8fcc7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 4 client.16179 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.974+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab8f94f720 con 0x55ab8fcc5900 2024-03-24T09:23:48.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.975+0000 7fbb33fff640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fcc5900 0x55ab8fcc7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.975+0000 7fbb137fe640 10 client.16179 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.975+0000 7fbb33fff640 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fcc5900 0x55ab8fcc7d20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=12dc09c60c9b7a3b server_cookie=4d2d750612073b7f in_seq=0 out_seq=0 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb137fe640 1 -- 172.21.15.116:0/2596784550 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab8f94f720 con 0x55ab8fcc5900 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb137fe640 10 client.16179 handle_command_reply: tid=0 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 client.16179 shutdown 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 msgr2=0x55ab8fcb2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcb2990 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbb28012010 tx=0x7fbb28003690 comp rx=0 tx=0).stop 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 shutdown_connections 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fcc5900 0x55ab8fcc7d20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/2596784550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcb2990 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 >> 172.21.15.116:0/2596784550 conn(0x55ab8fca6b20 msgr2=0x55ab8fca8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb137fe640 0 client.16179 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:48.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 shutdown_connections 2024-03-24T09:23:48.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/2596784550 wait complete. 2024-03-24T09:23:48.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.976+0000 7fbb3d3a2740 20 client.16179 trim_cache size 0 max 16384 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 Processor -- start 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 -- start start 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcc1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fccd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fcac050 con 0x55ab8fcac3f0 2024-03-24T09:23:48.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fc97e20 con 0x55ab8fcb58f0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fcb2770 con 0x55ab8fcacd10 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcc1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcc1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49400/0 (socket says 172.21.15.116:49400) 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 -- 172.21.15.116:0/3331634240 learned_addr learned my addr 172.21.15.116:0/3331634240 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb33fff640 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fccd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 -- 172.21.15.116:0/3331634240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 msgr2=0x55ab8fccd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fccd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 -- 172.21.15.116:0/3331634240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 msgr2=0x55ab8fcb2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 -- 172.21.15.116:0/3331634240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab8fbadc80 con 0x55ab8fcac3f0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb33fff640 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fccd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.991+0000 7fbb337fe640 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcc1120 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbb28011c40 tx=0x7fbb28005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2d23947444b7bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb19ffb640 1 -- 172.21.15.116:0/3331634240 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28009e60 con 0x55ab8fcac3f0 2024-03-24T09:23:48.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb19ffb640 1 -- 172.21.15.116:0/3331634240 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb28013e00 con 0x55ab8fcac3f0 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb19ffb640 1 -- 172.21.15.116:0/3331634240 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28020810 con 0x55ab8fcac3f0 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/3331634240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 msgr2=0x55ab8fcc1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcc1120 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbb28011c40 tx=0x7fbb28005970 comp rx=0 tx=0).stop 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/3331634240 shutdown_connections 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcacd10 0x55ab8fccd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/3331634240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcc1120 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/3331634240 >> 172.21.15.116:0/3331634240 conn(0x55ab8fca6b20 msgr2=0x55ab8fcadc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/3331634240 shutdown_connections 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.992+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/3331634240 wait complete. 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 -- start start 2024-03-24T09:23:48.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 Processor -- start 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcbda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fcb99d0 con 0x55ab8fcacd10 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fcb9b40 con 0x55ab8fcb58f0 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ab8fd2ac00 con 0x55ab8fcac3f0 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb3d3a2740 10 client.0 _ll_register_callbacks cb 0x7fbb280136b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb33fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb33fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49414/0 (socket says 172.21.15.116:49414) 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb33fff640 1 -- 172.21.15.116:0/561608856 learned_addr learned my addr 172.21.15.116:0/561608856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:48.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb337fe640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcbda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.993+0000 7fbb337fe640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb33fff640 1 -- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcac3f0 msgr2=0x55ab8fcbda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb33fff640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcbda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb33fff640 1 -- 172.21.15.116:0/561608856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 msgr2=0x55ab8fcb92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb33fff640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb33fff640 1 -- 172.21.15.116:0/561608856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb24009c60 con 0x55ab8fcacd10 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb33fff640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb8d90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x55ab8fcabec0 tx=0x7fbb24020710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89d2baf8accdde9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb137fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb137fe640 1 -- 172.21.15.116:0/561608856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb24017030 con 0x55ab8fcacd10 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb137fe640 1 -- 172.21.15.116:0/561608856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb24003a00 con 0x55ab8fcacd10 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ab8f99e890 con 0x55ab8fcacd10 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb137fe640 1 -- 172.21.15.116:0/561608856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fbb240044d0 con 0x55ab8fcacd10 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 10 client.16185 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:49.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ab8fc11bd0 con 0x55ab8fcacd10 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb137fe640 1 -- 172.21.15.116:0/561608856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ab8f99e890 con 0x55ab8fcacd10 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb137fe640 1 -- 172.21.15.116:0/561608856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fbb24029420 con 0x55ab8fcacd10 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 10 client.16185 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 10 client.16185 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 20 client.16185 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fd2c4e0 0x55ab8fd2e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 4 client.16185 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.994+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab8f8be700 con 0x55ab8fd2c4e0 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb337fe640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fd2c4e0 0x55ab8fd2e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb337fe640 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fd2c4e0 0x55ab8fd2e900 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16f0bc36d1e6e615 server_cookie=59326ac1153500e1 in_seq=0 out_seq=0 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb137fe640 10 client.16185 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb137fe640 1 -- 172.21.15.116:0/561608856 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab8f8be700 con 0x55ab8fd2c4e0 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb137fe640 10 client.16185 handle_command_reply: tid=0 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 client.16185 shutdown 2024-03-24T09:23:49.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 msgr2=0x55ab8fcb8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb8d90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x55ab8fcabec0 tx=0x7fbb24020710 comp rx=0 tx=0).stop 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 shutdown_connections 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ab8fd2c4e0 0x55ab8fd2e900 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ab8fcb58f0 0x55ab8fcb92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ab8fcacd10 0x55ab8fcb8d90 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb137fe640 0 client.16185 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 --2- 172.21.15.116:0/561608856 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ab8fcac3f0 0x55ab8fcbda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 >> 172.21.15.116:0/561608856 conn(0x55ab8fca6b20 msgr2=0x55ab8fcb2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 shutdown_connections 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.995+0000 7fbb3d3a2740 1 -- 172.21.15.116:0/561608856 wait complete. 2024-03-24T09:23:49.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3d3a2740 20 client.16185 trim_cache size 0 max 16384 2024-03-24T09:23:49.003 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb1c0a2230 msgr2=0x7fbb1c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb1c0a2230 0x7fbb1c0a46f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0ee460 tx=0x7fbb2c0a8180 comp rx=0 tx=0).stop 2024-03-24T09:23:49.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 msgr2=0x7fbb34197990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb34197990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c005e30 tx=0x7fbb2c034a60 comp rx=0 tx=0).stop 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 shutdown_connections 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34105b50 0x7fbb34197990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb1c0a2230 0x7fbb1c0a46f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb34105190 0x7fbb3419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 --2- 172.21.15.116:0/267336597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb34100bc0 0x7fbb3419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 >> 172.21.15.116:0/267336597 conn(0x7fbb340773d0 msgr2=0x7fbb3406a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 shutdown_connections 2024-03-24T09:23:49.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:48.996+0000 7fbb3a7c7640 1 -- 172.21.15.116:0/267336597 wait complete. 2024-03-24T09:23:49.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 -- 172.21.15.116:0/533323459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100f60 msgr2=0x7fec8010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 --2- 172.21.15.116:0/533323459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100f60 0x7fec8010d420 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fec78009ca0 tx=0x7fec7802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 -- 172.21.15.116:0/533323459 shutdown_connections 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 --2- 172.21.15.116:0/533323459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100f60 0x7fec8010d420 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 --2- 172.21.15.116:0/533323459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec801005a0 0x7fec80100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 --2- 172.21.15.116:0/533323459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80106660 0x7fec80106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 -- 172.21.15.116:0/533323459 >> 172.21.15.116:0/533323459 conn(0x7fec80077710 msgr2=0x7fec80077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 -- 172.21.15.116:0/533323459 shutdown_connections 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.119+0000 7fec8508d640 1 -- 172.21.15.116:0/533323459 wait complete. 2024-03-24T09:23:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 Processor -- start 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 -- start start 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec801005a0 0x7fec8019c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80100f60 0x7fec8019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 0x7fec80196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec80196ec0 con 0x7fec80106660 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec80197030 con 0x7fec80100f60 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec8508d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fec801971f0 con 0x7fec801005a0 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec7f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 0x7fec80196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec7f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 0x7fec80196870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49436/0 (socket says 172.21.15.116:49436) 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec7f59e640 1 -- 172.21.15.116:0/2096783859 learned_addr learned my addr 172.21.15.116:0/2096783859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec801005a0 0x7fec8019c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.120+0000 7fec77fff640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80100f60 0x7fec8019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7f59e640 1 -- 172.21.15.116:0/2096783859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec801005a0 msgr2=0x7fec8019c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7f59e640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec801005a0 0x7fec8019c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7f59e640 1 -- 172.21.15.116:0/2096783859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80100f60 msgr2=0x7fec8019cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7f59e640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80100f60 0x7fec8019cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7f59e640 1 -- 172.21.15.116:0/2096783859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec78009900 con 0x7fec80106660 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7f59e640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 0x7fec80196870 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fec7802c700 tx=0x7fec78034ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7cd99640 1 -- 172.21.15.116:0/2096783859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78032820 con 0x7fec80106660 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7cd99640 1 -- 172.21.15.116:0/2096783859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec78003e20 con 0x7fec80106660 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec80197470 con 0x7fec80106660 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec7cd99640 1 -- 172.21.15.116:0/2096783859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec780047d0 con 0x7fec80106660 2024-03-24T09:23:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.121+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec801a3760 con 0x7fec80106660 2024-03-24T09:23:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.123+0000 7fec7cd99640 1 -- 172.21.15.116:0/2096783859 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fec7800a040 con 0x7fec80106660 2024-03-24T09:23:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 Processor -- start 2024-03-24T09:23:49.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 -- start start 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e194f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e19585c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e195aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8e18bb840 con 0x55e8e194f600 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e8e18b8d70 con 0x55e8e194ff20 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec87c62740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8e18b4550 con 0x55e8e1958b00 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec7cd99640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec6c0a2230 0x7fec6c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e194f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec7ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e19585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e194f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49442/0 (socket says 172.21.15.116:49442) 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 -- 172.21.15.116:0/2376210047 learned_addr learned my addr 172.21.15.116:0/2376210047 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec7f59e640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec6c0a2230 0x7fec6c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e195aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec7cd99640 1 -- 172.21.15.116:0/2096783859 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fec780f0fa0 con 0x7fec80106660 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 -- 172.21.15.116:0/2376210047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 msgr2=0x55e8e195aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e195aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 -- 172.21.15.116:0/2376210047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 msgr2=0x55e8e19585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e19585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec77fff640 1 -- 172.21.15.116:0/2376210047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8e1849de0 con 0x55e8e194f600 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.124+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e19585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec7f59e640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec6c0a2230 0x7fec6c0a46f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec780ee500 tx=0x7fec78090760 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec77fff640 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e194f9e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fec680137c0 tx=0x7fec68009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f92e9aece0d8238 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec757fa640 1 -- 172.21.15.116:0/2376210047 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68013d40 con 0x55e8e194f600 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec757fa640 1 -- 172.21.15.116:0/2376210047 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec680044e0 con 0x55e8e194f600 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec757fa640 1 -- 172.21.15.116:0/2376210047 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec680035f0 con 0x55e8e194f600 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec87c62740 1 -- 172.21.15.116:0/2376210047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 msgr2=0x55e8e194f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.125+0000 7fec87c62740 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e194f9e0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fec680137c0 tx=0x7fec68009780 comp rx=0 tx=0).stop 2024-03-24T09:23:49.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- 172.21.15.116:0/2376210047 shutdown_connections 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e195aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e19585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 --2- 172.21.15.116:0/2376210047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e194f9e0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- 172.21.15.116:0/2376210047 >> 172.21.15.116:0/2376210047 conn(0x55e8e1949d30 msgr2=0x55e8e194a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- 172.21.15.116:0/2376210047 shutdown_connections 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- 172.21.15.116:0/2376210047 wait complete. 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- start start 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 Processor -- start 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1955be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e1956120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e1956660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1587430 con 0x55e8e194f600 2024-03-24T09:23:49.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1543e00 con 0x55e8e194ff20 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1839590 con 0x55e8e1958b00 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec87c62740 10 client.0 _ll_register_callbacks cb 0x55e8e18a8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1955be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1955be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49456/0 (socket says 172.21.15.116:49456) 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec77fff640 1 -- 172.21.15.116:0/1701858166 learned_addr learned my addr 172.21.15.116:0/1701858166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.126+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e1956120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e1956660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 -- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 msgr2=0x55e8e1956660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e1956660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 -- 172.21.15.116:0/1701858166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 msgr2=0x55e8e1956120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e1956120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 -- 172.21.15.116:0/1701858166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec6801b040 con 0x55e8e194f600 2024-03-24T09:23:49.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec77fff640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1955be0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x55e8e194efb0 tx=0x7fec68011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8031cfe90bcca364 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec667fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec667fc640 1 -- 172.21.15.116:0/1701858166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec680052d0 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec667fc640 1 -- 172.21.15.116:0/1701858166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fec68005430 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e8e15b37e0 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec667fc640 1 -- 172.21.15.116:0/1701858166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fec680055f0 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec87c62740 10 client.16197 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.127+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e8e18b2bd0 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec667fc640 1 -- 172.21.15.116:0/1701858166 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e8e15b37e0 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec667fc640 1 -- 172.21.15.116:0/1701858166 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fec68020540 con 0x55e8e194f600 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec87c62740 10 client.16197 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec87c62740 10 client.16197 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec87c62740 20 client.16197 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:49.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec87c62740 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e1968c60 0x55e8e196b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec87c62740 4 client.16197 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8e15f0720 con 0x55e8e1968c60 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.128+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e1968c60 0x55e8e196b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.129+0000 7fec667fc640 10 client.16197 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.129+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e1968c60 0x55e8e196b080 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c42844ddcee49a39 server_cookie=a2645f9e4df4316f in_seq=0 out_seq=0 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.129+0000 7fec667fc640 1 -- 172.21.15.116:0/1701858166 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e8e15f0720 con 0x55e8e1968c60 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.129+0000 7fec667fc640 10 client.16197 handle_command_reply: tid=0 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.129+0000 7fec87c62740 1 client.16197 shutdown 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 msgr2=0x55e8e1955be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1955be0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x55e8e194efb0 tx=0x7fec68011d30 comp rx=0 tx=0).stop 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 shutdown_connections 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e1968c60 0x55e8e196b080 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e1956660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194ff20 0x55e8e1956120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 --2- 172.21.15.116:0/1701858166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1955be0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 >> 172.21.15.116:0/1701858166 conn(0x55e8e1949d30 msgr2=0x55e8e1946cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec667fc640 0 client.16197 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 shutdown_connections 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 1 -- 172.21.15.116:0/1701858166 wait complete. 2024-03-24T09:23:49.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.130+0000 7fec87c62740 20 client.16197 trim_cache size 0 max 16384 2024-03-24T09:23:49.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.143+0000 7fec87c62740 1 Processor -- start 2024-03-24T09:23:49.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 -- start start 2024-03-24T09:23:49.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1958310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e194ff20 0x55e8e19544b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e1958b00 0x55e8e1964390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1956bb0 con 0x55e8e194f600 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e8e194e6f0 con 0x55e8e1958b00 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec87c62740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1943210 con 0x55e8e194ff20 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1958310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1958310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49470/0 (socket says 172.21.15.116:49470) 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 -- 172.21.15.116:0/2732036931 learned_addr learned my addr 172.21.15.116:0/2732036931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e194ff20 0x55e8e19544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e1958b00 0x55e8e1964390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 -- 172.21.15.116:0/2732036931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e194ff20 msgr2=0x55e8e19544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e194ff20 0x55e8e19544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 -- 172.21.15.116:0/2732036931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e1958b00 msgr2=0x55e8e1964390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e1958b00 0x55e8e1964390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 -- 172.21.15.116:0/2732036931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8e184ec80 con 0x55e8e194f600 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.144+0000 7fec77fff640 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1958310 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fec680059f0 tx=0x7fec68013590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9698afa9aba7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec74ff9640 1 -- 172.21.15.116:0/2732036931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68009d30 con 0x55e8e194f600 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec74ff9640 1 -- 172.21.15.116:0/2732036931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec68011da0 con 0x55e8e194f600 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec74ff9640 1 -- 172.21.15.116:0/2732036931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68004650 con 0x55e8e194f600 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 -- 172.21.15.116:0/2732036931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 msgr2=0x55e8e1958310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1958310 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fec680059f0 tx=0x7fec68013590 comp rx=0 tx=0).stop 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 -- 172.21.15.116:0/2732036931 shutdown_connections 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e1958b00 0x55e8e1964390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e194ff20 0x55e8e19544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 --2- 172.21.15.116:0/2732036931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194f600 0x55e8e1958310 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 -- 172.21.15.116:0/2732036931 >> 172.21.15.116:0/2732036931 conn(0x55e8e1949d30 msgr2=0x55e8e1946cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 -- 172.21.15.116:0/2732036931 shutdown_connections 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.145+0000 7fec87c62740 1 -- 172.21.15.116:0/2732036931 wait complete. 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 -- start start 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 Processor -- start 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194f600 0x55e8e19488e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 0x55e8e1948e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e19473d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1947990 con 0x55e8e194ff20 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1947b00 con 0x55e8e194f600 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8e1947cc0 con 0x55e8e1958b00 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec87c62740 10 client.0 _ll_register_callbacks cb 0x7fec7000b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 0x55e8e1948e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194f600 0x55e8e19488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 0x55e8e1948e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49474/0 (socket says 172.21.15.116:49474) 2024-03-24T09:23:49.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 -- 172.21.15.116:0/446134385 learned_addr learned my addr 172.21.15.116:0/446134385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec77fff640 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e19473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 -- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 msgr2=0x55e8e19473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e19473d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 -- 172.21.15.116:0/446134385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194f600 msgr2=0x55e8e19488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194f600 0x55e8e19488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.146+0000 7fec7ed9d640 1 -- 172.21.15.116:0/446134385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8e18b2bd0 con 0x55e8e194ff20 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec7ed9d640 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 0x55e8e1948e20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b7e0 tx=0x7fec70012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6ed7ef1c8ab8f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec667fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec667fc640 1 -- 172.21.15.116:0/446134385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70016070 con 0x55e8e194ff20 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec667fc640 1 -- 172.21.15.116:0/446134385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fec70004220 con 0x55e8e194ff20 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec680044e0 con 0x55e8e194ff20 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec667fc640 1 -- 172.21.15.116:0/446134385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fec70004400 con 0x55e8e194ff20 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 10 client.16203 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e8e1874920 con 0x55e8e194ff20 2024-03-24T09:23:49.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec667fc640 1 -- 172.21.15.116:0/446134385 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec680044e0 con 0x55e8e194ff20 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec667fc640 1 -- 172.21.15.116:0/446134385 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fec700047a0 con 0x55e8e194ff20 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 10 client.16203 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 10 client.16203 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 20 client.16203 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e19d2360 0x55e8e19d4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 4 client.16203 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.147+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8e163f890 con 0x55e8e19d2360 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec77fff640 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e19d2360 0x55e8e19d4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec667fc640 10 client.16203 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec77fff640 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e19d2360 0x55e8e19d4780 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7328f6ca4022b468 server_cookie=2ed836d62e900f35 in_seq=0 out_seq=0 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec667fc640 1 -- 172.21.15.116:0/446134385 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8e163f890 con 0x55e8e19d2360 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec667fc640 10 client.16203 handle_command_reply: tid=0 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 client.16203 shutdown 2024-03-24T09:23:49.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 msgr2=0x55e8e1948e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 0x55e8e1948e20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fec7000b7e0 tx=0x7fec70012820 comp rx=0 tx=0).stop 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 shutdown_connections 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e8e19d2360 0x55e8e19d4780 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8e1958b00 0x55e8e19473d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec667fc640 0 client.16203 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8e194ff20 0x55e8e1948e20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 --2- 172.21.15.116:0/446134385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8e194f600 0x55e8e19488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 >> 172.21.15.116:0/446134385 conn(0x55e8e1949d30 msgr2=0x55e8e1955ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 shutdown_connections 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 1 -- 172.21.15.116:0/446134385 wait complete. 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.148+0000 7fec87c62740 20 client.16203 trim_cache size 0 max 16384 2024-03-24T09:23:49.155 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec6c0a2230 msgr2=0x7fec6c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec6c0a2230 0x7fec6c0a46f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec780ee500 tx=0x7fec78090760 comp rx=0 tx=0).stop 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 msgr2=0x7fec80196870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 0x7fec80196870 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fec7802c700 tx=0x7fec78034ae0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 shutdown_connections 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106660 0x7fec80196870 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80100f60 0x7fec8019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec6c0a2230 0x7fec6c0a46f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 --2- 172.21.15.116:0/2096783859 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec801005a0 0x7fec8019c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 >> 172.21.15.116:0/2096783859 conn(0x7fec80077710 msgr2=0x7fec800ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 shutdown_connections 2024-03-24T09:23:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.149+0000 7fec8508d640 1 -- 172.21.15.116:0/2096783859 wait complete. 2024-03-24T09:23:49.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:48 smithi116 ceph-mon[32207]: pgmap v54: 129 pgs: 129 active+clean; 60 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 354 op/s 2024-03-24T09:23:49.159 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:48 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/844797864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:23:49.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:23:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 -- 172.21.15.116:0/155443316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81016a0 msgr2=0x7f76d8101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/155443316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81016a0 0x7f76d8101b20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009cd0 tx=0x7f76cc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 -- 172.21.15.116:0/155443316 shutdown_connections 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/155443316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8102060 0x7f76d810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/155443316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d81016a0 0x7f76d8101b20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/155443316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d81076b0 0x7f76d8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 -- 172.21.15.116:0/155443316 >> 172.21.15.116:0/155443316 conn(0x7f76d8078b10 msgr2=0x7f76d8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 -- 172.21.15.116:0/155443316 shutdown_connections 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 -- 172.21.15.116:0/155443316 wait complete. 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 Processor -- start 2024-03-24T09:23:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.268+0000 7f76dd7d0640 1 -- start start 2024-03-24T09:23:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d81016a0 0x7f76d8073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 0x7f76d8073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d81076b0 0x7f76d8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81a6020 con 0x7f76d8102060 2024-03-24T09:23:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81a6190 con 0x7f76d81016a0 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81a6350 con 0x7f76d81076b0 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 0x7f76d8073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 0x7f76d8073f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49486/0 (socket says 172.21.15.116:49486) 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 -- 172.21.15.116:0/2295222931 learned_addr learned my addr 172.21.15.116:0/2295222931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d81016a0 0x7f76d8073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d77fe640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d81076b0 0x7f76d8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 -- 172.21.15.116:0/2295222931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d81076b0 msgr2=0x7f76d8074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d81076b0 0x7f76d8074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 -- 172.21.15.116:0/2295222931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d81016a0 msgr2=0x7f76d8073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d81016a0 0x7f76d8073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 -- 172.21.15.116:0/2295222931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76cc009900 con 0x7f76d8102060 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76d67fc640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 0x7f76d8073f50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f76cc02c700 tx=0x7f76cc035ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76bffff640 1 -- 172.21.15.116:0/2295222931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0329e0 con 0x7f76d8102060 2024-03-24T09:23:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d81a65d0 con 0x7f76d8102060 2024-03-24T09:23:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76bffff640 1 -- 172.21.15.116:0/2295222931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76cc035e90 con 0x7f76d8102060 2024-03-24T09:23:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76bffff640 1 -- 172.21.15.116:0/2295222931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0046f0 con 0x7f76d8102060 2024-03-24T09:23:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.269+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d81a6a80 con 0x7f76d8102060 2024-03-24T09:23:49.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76bffff640 1 -- 172.21.15.116:0/2295222931 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f76cc03b040 con 0x7f76d8102060 2024-03-24T09:23:49.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 Processor -- start 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 -- start start 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31ac3f0 0x55f1b31ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1b311a840 con 0x55f1b31acd10 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f1b3117d70 con 0x55f1b31ac3f0 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76e03b3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f1b3113550 con 0x55f1b31b58f0 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.272+0000 7f76bffff640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76c00a2230 0x7f76c00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31ac3f0 0x55f1b31ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31ac3f0 0x55f1b31ac7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58162/0 (socket says 172.21.15.116:58162) 2024-03-24T09:23:49.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d77fe640 1 -- 172.21.15.116:0/2310031294 learned_addr learned my addr 172.21.15.116:0/2310031294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d77fe640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76c00a2230 0x7f76c00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 -- 172.21.15.116:0/2310031294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 msgr2=0x55f1b31b7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76bffff640 1 -- 172.21.15.116:0/2295222931 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f76cc0f0c90 con 0x7f76d8102060 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 -- 172.21.15.116:0/2310031294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31ac3f0 msgr2=0x55f1b31ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31ac3f0 0x55f1b31ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 -- 172.21.15.116:0/2310031294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1b30a8de0 con 0x55f1b31acd10 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b53b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009c20 tx=0x7f76d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd7625543959205 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76d77fe640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76c00a2230 0x7f76c00a46f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009fd0 tx=0x7f76c8004730 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76bd7fa640 1 -- 172.21.15.116:0/2310031294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0004050 con 0x55f1b31acd10 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76bd7fa640 1 -- 172.21.15.116:0/2310031294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76d0005020 con 0x55f1b31acd10 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.273+0000 7f76bd7fa640 1 -- 172.21.15.116:0/2310031294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0004bc0 con 0x55f1b31acd10 2024-03-24T09:23:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 -- 172.21.15.116:0/2310031294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 msgr2=0x55f1b31b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b53b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009c20 tx=0x7f76d0011810 comp rx=0 tx=0).stop 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 -- 172.21.15.116:0/2310031294 shutdown_connections 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b53b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 --2- 172.21.15.116:0/2310031294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31ac3f0 0x55f1b31ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 -- 172.21.15.116:0/2310031294 >> 172.21.15.116:0/2310031294 conn(0x55f1b31a6b20 msgr2=0x55f1b31a6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 -- 172.21.15.116:0/2310031294 shutdown_connections 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 -- 172.21.15.116:0/2310031294 wait complete. 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 -- start start 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.274+0000 7f76e03b3740 1 Processor -- start 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2de6430 con 0x55f1b31acd10 2024-03-24T09:23:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2da2e00 con 0x55f1b31b58f0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f1b3098590 con 0x55f1b31ac3f0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76e03b3740 10 client.0 _ll_register_callbacks cb 0x55f1b3107fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49504/0 (socket says 172.21.15.116:49504) 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 -- 172.21.15.116:0/3446722027 learned_addr learned my addr 172.21.15.116:0/3446722027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d77fe640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 -- 172.21.15.116:0/3446722027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 msgr2=0x55f1b31b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 -- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 msgr2=0x55f1b31b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 -- 172.21.15.116:0/3446722027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d0009850 con 0x55f1b31acd10 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76d67fc640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b2e90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0b9d50 tx=0x7f76cc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18bee9c658d081ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.275+0000 7f76baff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76baff5640 1 -- 172.21.15.116:0/3446722027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0bd380 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76baff5640 1 -- 172.21.15.116:0/3446722027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f76cc0c14d0 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f1b2e127e0 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76baff5640 1 -- 172.21.15.116:0/3446722027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f76cc0c5b20 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76e03b3740 10 client.16215 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f1b3111bd0 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.276+0000 7f76baff5640 1 -- 172.21.15.116:0/3446722027 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f1b2e127e0 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76baff5640 1 -- 172.21.15.116:0/3446722027 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f76cc0c5690 con 0x55f1b31acd10 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76e03b3740 10 client.16215 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76e03b3740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76e03b3740 20 client.16215 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b31c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76e03b3740 4 client.16215 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1b2e4f720 con 0x55f1b31c5800 2024-03-24T09:23:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.277+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b31c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:48 smithi143 ceph-mon[35965]: pgmap v54: 129 pgs: 129 active+clean; 60 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 354 op/s 2024-03-24T09:23:49.283 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:48 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/844797864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.278+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b31c7c60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fb7dec782288911 server_cookie=422f3319c8cc1c5b in_seq=0 out_seq=0 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.278+0000 7f76baff5640 10 client.16215 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76baff5640 1 -- 172.21.15.116:0/3446722027 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1b2e4f720 con 0x55f1b31c5800 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76baff5640 10 client.16215 handle_command_reply: tid=0 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 client.16215 shutdown 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 msgr2=0x55f1b31b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b2e90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0b9d50 tx=0x7f76cc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 shutdown_connections 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b31c7c60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31b2e90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76baff5640 0 client.16215 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3446722027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 >> 172.21.15.116:0/3446722027 conn(0x55f1b31a6b20 msgr2=0x55f1b31a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 shutdown_connections 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 1 -- 172.21.15.116:0/3446722027 wait complete. 2024-03-24T09:23:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.279+0000 7f76e03b3740 20 client.16215 trim_cache size 0 max 16384 2024-03-24T09:23:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.292+0000 7f76e03b3740 1 Processor -- start 2024-03-24T09:23:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 -- start start 2024-03-24T09:23:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31acd10 0x55f1b31a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1b31c24a0 con 0x55f1b31ac3f0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f1b31ac050 con 0x55f1b31acd10 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76e03b3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f1b31a7b50 con 0x55f1b31b58f0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49516/0 (socket says 172.21.15.116:49516) 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 -- 172.21.15.116:0/3137982667 learned_addr learned my addr 172.21.15.116:0/3137982667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d67fc640 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31acd10 0x55f1b31a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d77fe640 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 -- 172.21.15.116:0/3137982667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 msgr2=0x55f1b31b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 -- 172.21.15.116:0/3137982667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31acd10 msgr2=0x55f1b31a5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31acd10 0x55f1b31a5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 -- 172.21.15.116:0/3137982667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1b31a63c0 con 0x55f1b31ac3f0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b5080 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76d0014040 tx=0x7f76d00097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981cbe03ad8d60bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76bcff9640 1 -- 172.21.15.116:0/3137982667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0009d10 con 0x55f1b31ac3f0 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76bcff9640 1 -- 172.21.15.116:0/3137982667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76d0009e70 con 0x55f1b31ac3f0 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.293+0000 7f76bcff9640 1 -- 172.21.15.116:0/3137982667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0003d10 con 0x55f1b31ac3f0 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 -- 172.21.15.116:0/3137982667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 msgr2=0x55f1b31b5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b5080 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f76d0014040 tx=0x7f76d00097a0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 -- 172.21.15.116:0/3137982667 shutdown_connections 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31b58f0 0x55f1b31b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31acd10 0x55f1b31a5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 --2- 172.21.15.116:0/3137982667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31ac3f0 0x55f1b31b5080 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 -- 172.21.15.116:0/3137982667 >> 172.21.15.116:0/3137982667 conn(0x55f1b31a6b20 msgr2=0x55f1b31a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 -- 172.21.15.116:0/3137982667 shutdown_connections 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.294+0000 7f76e03b3740 1 -- 172.21.15.116:0/3137982667 wait complete. 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 -- start start 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 Processor -- start 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31bdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f1b31ba340 con 0x55f1b31acd10 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f1b31ba4b0 con 0x55f1b31b58f0 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f1b31ba670 con 0x55f1b31ac3f0 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76e03b3740 10 client.0 _ll_register_callbacks cb 0x7f76cc009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31bdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31bdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49528/0 (socket says 172.21.15.116:49528) 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 -- 172.21.15.116:0/1877220036 learned_addr learned my addr 172.21.15.116:0/1877220036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d77fe640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 -- 172.21.15.116:0/1877220036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 msgr2=0x55f1b31bd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31bd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 -- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 msgr2=0x55f1b31b9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 -- 172.21.15.116:0/1877220036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1b30a8de0 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d77fe640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76d67fc640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31bdc00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009df0 tx=0x7f76cc0c13a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599700b6ab053c65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.295+0000 7f76baff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76baff5640 1 -- 172.21.15.116:0/1877220036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0d8030 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76baff5640 1 -- 172.21.15.116:0/1877220036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f76cc0d9070 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76d0005020 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76baff5640 1 -- 172.21.15.116:0/1877220036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f76cc0bcce0 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 10 client.16221 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f1b30adc80 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76baff5640 1 -- 172.21.15.116:0/1877220036 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f76d0005020 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76baff5640 1 -- 172.21.15.116:0/1877220036 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f76cc0bd670 con 0x55f1b31acd10 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 10 client.16221 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 20 client.16221 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b322ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 4 client.16221 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1b2e9e890 con 0x55f1b31c5800 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.296+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b322ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76baff5640 10 client.16221 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76d6ffd640 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b322ea80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb18e8d7d45ca38b server_cookie=5306e48f86f4dbbe in_seq=0 out_seq=0 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76baff5640 1 -- 172.21.15.116:0/1877220036 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1b2e9e890 con 0x55f1b31c5800 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76baff5640 10 client.16221 handle_command_reply: tid=0 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 client.16221 shutdown 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 msgr2=0x55f1b31bdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31bdc00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009df0 tx=0x7f76cc0c13a0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 shutdown_connections 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f1b31c5800 0x55f1b322ea80 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f1b31b58f0 0x55f1b31b9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f1b31acd10 0x55f1b31bdc00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76baff5640 0 client.16221 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 --2- 172.21.15.116:0/1877220036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f1b31ac3f0 0x55f1b31bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 >> 172.21.15.116:0/1877220036 conn(0x55f1b31a6b20 msgr2=0x55f1b31a8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 shutdown_connections 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 1 -- 172.21.15.116:0/1877220036 wait complete. 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.297+0000 7f76e03b3740 20 client.16221 trim_cache size 0 max 16384 2024-03-24T09:23:49.304 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76c00a2230 msgr2=0x7f76c00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76c00a2230 0x7f76c00a46f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009fd0 tx=0x7f76c8004730 comp rx=0 tx=0).stop 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 msgr2=0x7f76d8073f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 0x7f76d8073f50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f76cc02c700 tx=0x7f76cc035ca0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 shutdown_connections 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f76c00a2230 0x7f76c00a46f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76d81076b0 0x7f76d8074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76d8102060 0x7f76d8073f50 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 --2- 172.21.15.116:0/2295222931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d81016a0 0x7f76d8073a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 >> 172.21.15.116:0/2295222931 conn(0x7f76d8078b10 msgr2=0x7f76d8104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 shutdown_connections 2024-03-24T09:23:49.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:49.298+0000 7f76dd7d0640 1 -- 172.21.15.116:0/2295222931 wait complete. 2024-03-24T09:23:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:48 smithi117 ceph-mon[35788]: pgmap v54: 129 pgs: 129 active+clean; 60 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 354 op/s 2024-03-24T09:23:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:48 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/844797864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:23:50.310 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:23:50.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:23:50.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:23:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/343171496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a78106660 msgr2=0x7f0a78106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/343171496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a78106660 0x7f0a78106a40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009900 tx=0x7f0a6802bc30 comp rx=0 tx=0).stop 2024-03-24T09:23:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/343171496 shutdown_connections 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/343171496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a78100f60 0x7f0a7810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/343171496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a781005a0 0x7f0a78100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/343171496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a78106660 0x7f0a78106a40 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/343171496 >> 172.21.15.116:0/343171496 conn(0x7f0a78077710 msgr2=0x7f0a78077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/343171496 shutdown_connections 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/343171496 wait complete. 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 Processor -- start 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.421+0000 7f0a7e8cd640 1 -- start start 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 0x7f0a78102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78100f60 0x7f0a78102750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a78106660 0x7f0a78102c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a6000 con 0x7f0a781005a0 2024-03-24T09:23:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a6170 con 0x7f0a78100f60 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a6330 con 0x7f0a78106660 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 0x7f0a78102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78100f60 0x7f0a78102750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 0x7f0a78102210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49544/0 (socket says 172.21.15.116:49544) 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 -- 172.21.15.116:0/482662086 learned_addr learned my addr 172.21.15.116:0/482662086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a78106660 0x7f0a78102c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 -- 172.21.15.116:0/482662086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a78106660 msgr2=0x7f0a78102c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a78106660 0x7f0a78102c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 -- 172.21.15.116:0/482662086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78100f60 msgr2=0x7f0a78102750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78100f60 0x7f0a78102750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 -- 172.21.15.116:0/482662086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a68009560 con 0x7f0a781005a0 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a77fff640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 0x7f0a78102210 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0a68000c00 tx=0x7f0a68005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a757fa640 1 -- 172.21.15.116:0/482662086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68004490 con 0x7f0a781005a0 2024-03-24T09:23:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a757fa640 1 -- 172.21.15.116:0/482662086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a68003aa0 con 0x7f0a781005a0 2024-03-24T09:23:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781a65b0 con 0x7f0a781005a0 2024-03-24T09:23:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.422+0000 7f0a757fa640 1 -- 172.21.15.116:0/482662086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68005450 con 0x7f0a781005a0 2024-03-24T09:23:50.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.423+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a781a6b10 con 0x7f0a781005a0 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a757fa640 1 -- 172.21.15.116:0/482662086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f0a68003c10 con 0x7f0a781005a0 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 Processor -- start 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 -- start start 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a757fa640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0a600a22b0 0x7f0a600a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f8ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f93720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5f96050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598a5ef8530 con 0x5598a5f8b080 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5598a5ef5a60 con 0x5598a5f93c60 2024-03-24T09:23:50.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a81588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5598a5ef1240 con 0x5598a5f8a760 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0a600a22b0 0x7f0a600a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f93720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f8ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f93720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49552/0 (socket says 172.21.15.116:49552) 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.425+0000 7f0a777fe640 1 -- 172.21.15.116:0/3113640981 learned_addr learned my addr 172.21.15.116:0/3113640981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5f96050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a757fa640 1 -- 172.21.15.116:0/482662086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0a680f0a10 con 0x7f0a781005a0 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a777fe640 1 -- 172.21.15.116:0/3113640981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 msgr2=0x5598a5f8ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f8ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a777fe640 1 -- 172.21.15.116:0/3113640981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 msgr2=0x5598a5f96050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5f96050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a777fe640 1 -- 172.21.15.116:0/3113640981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598a5eef8c0 con 0x5598a5f8b080 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0a600a22b0 0x7f0a600a4770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5598a5f86020 tx=0x7f0a70009b70 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f93720 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009c90 tx=0x7f0a6c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02575af760f430b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a5e7fc640 1 -- 172.21.15.116:0/3113640981 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c019630 con 0x5598a5f8b080 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a5e7fc640 1 -- 172.21.15.116:0/3113640981 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a6c009e40 con 0x5598a5f8b080 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.426+0000 7f0a5e7fc640 1 -- 172.21.15.116:0/3113640981 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00e9f0 con 0x5598a5f8b080 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 -- 172.21.15.116:0/3113640981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 msgr2=0x5598a5f93720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f93720 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009c90 tx=0x7f0a6c011810 comp rx=0 tx=0).stop 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 -- 172.21.15.116:0/3113640981 shutdown_connections 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5f96050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f93720 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 --2- 172.21.15.116:0/3113640981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f8ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 -- 172.21.15.116:0/3113640981 >> 172.21.15.116:0/3113640981 conn(0x5598a5f84e90 msgr2=0x5598a5f85270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 -- 172.21.15.116:0/3113640981 shutdown_connections 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 -- 172.21.15.116:0/3113640981 wait complete. 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 -- start start 2024-03-24T09:23:50.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.427+0000 7f0a81588740 1 Processor -- start 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 0x5598a5f90cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f93c60 0x5598a5f91740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598a5e95f80 con 0x5598a5f8b080 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5598a5bc44b0 con 0x5598a5f8a760 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5598a5e76ad0 con 0x5598a5f93c60 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 10 client.0 _ll_register_callbacks cb 0x5598a5ea9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 0x5598a5f90cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 0x5598a5f90cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58226/0 (socket says 172.21.15.116:58226) 2024-03-24T09:23:50.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 -- 172.21.15.116:0/4037799403 learned_addr learned my addr 172.21.15.116:0/4037799403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49564/0 (socket says 172.21.15.116:49564) 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f93c60 0x5598a5f91740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 -- 172.21.15.116:0/4037799403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f93c60 msgr2=0x5598a5f91740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f93c60 0x5598a5f91740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 -- 172.21.15.116:0/4037799403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 msgr2=0x5598a5f91200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 -- 172.21.15.116:0/4037799403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6c0098f0 con 0x5598a5f8a760 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a777fe640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a77fff640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 0x5598a5f90cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006010 tx=0x7f0a680045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=224d30f86a5fb62b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a577fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a577fe640 1 -- 172.21.15.116:0/4037799403 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a680bd160 con 0x5598a5f8a760 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5598a5f157a0 con 0x5598a5f8a760 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.428+0000 7f0a577fe640 1 -- 172.21.15.116:0/4037799403 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0a680be490 con 0x5598a5f8a760 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a577fe640 1 -- 172.21.15.116:0/4037799403 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0a680be6c0 con 0x5598a5f8a760 2024-03-24T09:23:50.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 10 client.26063 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5598a5ea76d0 con 0x5598a5f8a760 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a577fe640 1 -- 172.21.15.116:0/4037799403 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a680d5e90 con 0x5598a5f8a760 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a577fe640 1 -- 172.21.15.116:0/4037799403 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f0a680d4020 con 0x5598a5f8a760 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 10 client.26063 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 20 client.26063 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a5fa3a90 0x5598a5fa5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 4 client.26063 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598a5ee71f0 con 0x5598a5fa3a90 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a777fe640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a5fa3a90 0x5598a5fa5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a577fe640 10 client.26063 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.429+0000 7f0a777fe640 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a5fa3a90 0x5598a5fa5ef0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cc1271166d4012d server_cookie=f52850b47ef77f48 in_seq=0 out_seq=0 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a577fe640 1 -- 172.21.15.116:0/4037799403 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5598a5ee71f0 con 0x5598a5fa3a90 2024-03-24T09:23:50.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a577fe640 10 client.26063 handle_command_reply: tid=0 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 client.26063 shutdown 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 msgr2=0x5598a5f90cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 0x5598a5f90cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006010 tx=0x7f0a680045f0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 shutdown_connections 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f93c60 0x5598a5f91740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a5fa3a90 0x5598a5fa5ef0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 --2- 172.21.15.116:0/4037799403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8a760 0x5598a5f90cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a577fe640 0 client.26063 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 >> 172.21.15.116:0/4037799403 conn(0x5598a5f84e90 msgr2=0x5598a5f86be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 shutdown_connections 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 1 -- 172.21.15.116:0/4037799403 wait complete. 2024-03-24T09:23:50.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.432+0000 7f0a81588740 20 client.26063 trim_cache size 0 max 16384 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 Processor -- start 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 -- start start 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f933f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5fab800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598a5f80620 con 0x5598a5f8b080 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5598a5f8f230 con 0x5598a5f93c60 2024-03-24T09:23:50.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a81588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5598a5f96380 con 0x5598a5f8a760 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49574/0 (socket says 172.21.15.116:49574) 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 -- 172.21.15.116:0/3282746553 learned_addr learned my addr 172.21.15.116:0/3282746553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a77fff640 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f933f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5fab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 -- 172.21.15.116:0/3282746553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 msgr2=0x5598a5f933f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f933f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 -- 172.21.15.116:0/3282746553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 msgr2=0x5598a5fab800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.451+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5fab800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a777fe640 1 -- 172.21.15.116:0/3282746553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598a5e8bc60 con 0x5598a5f8b080 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a77fff640 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f933f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91560 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e5c0 tx=0x7f0a6c00e550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe8c8de7817b40a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a5dffb640 1 -- 172.21.15.116:0/3282746553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0037f0 con 0x5598a5f8b080 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a5dffb640 1 -- 172.21.15.116:0/3282746553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0a6c005470 con 0x5598a5f8b080 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a5dffb640 1 -- 172.21.15.116:0/3282746553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00ec30 con 0x5598a5f8b080 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a81588740 1 -- 172.21.15.116:0/3282746553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 msgr2=0x5598a5f91560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.452+0000 7f0a81588740 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91560 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c00e5c0 tx=0x7f0a6c00e550 comp rx=0 tx=0).stop 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- 172.21.15.116:0/3282746553 shutdown_connections 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f93c60 0x5598a5fab800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f8b080 0x5598a5f91560 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 --2- 172.21.15.116:0/3282746553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f933f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- 172.21.15.116:0/3282746553 >> 172.21.15.116:0/3282746553 conn(0x5598a5f84e90 msgr2=0x5598a5f83660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- 172.21.15.116:0/3282746553 shutdown_connections 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- 172.21.15.116:0/3282746553 wait complete. 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- start start 2024-03-24T09:23:50.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 Processor -- start 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f98ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8b080 0x5598a5f99220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 0x5598a6009e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5598a5f998f0 con 0x5598a5f93c60 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5598a600a380 con 0x5598a5f8b080 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5598a600a4c0 con 0x5598a5f8a760 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a81588740 10 client.0 _ll_register_callbacks cb 0x7f0a680bcd70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a7ce6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 0x5598a6009e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a77fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f98ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a7ce6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 0x5598a6009e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49578/0 (socket says 172.21.15.116:49578) 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a7ce6a640 1 -- 172.21.15.116:0/3334056790 learned_addr learned my addr 172.21.15.116:0/3334056790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.453+0000 7f0a777fe640 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8b080 0x5598a5f99220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a7ce6a640 1 -- 172.21.15.116:0/3334056790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 msgr2=0x5598a5f98ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f98ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a7ce6a640 1 -- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8b080 msgr2=0x5598a5f99220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8b080 0x5598a5f99220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a7ce6a640 1 -- 172.21.15.116:0/3334056790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598a5ea76d0 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a7ce6a640 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 0x5598a6009e40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0a700132b0 tx=0x7f0a70009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74396ef6fc6437dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a577fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a577fe640 1 -- 172.21.15.116:0/3334056790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70027070 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a577fe640 1 -- 172.21.15.116:0/3334056790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0a70003e00 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5598a5bf0860 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a577fe640 1 -- 172.21.15.116:0/3334056790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0a700132e0 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a81588740 10 client.16239 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5598a5eef8c0 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.454+0000 7f0a577fe640 1 -- 172.21.15.116:0/3334056790 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5598a5bf0860 con 0x5598a5f93c60 2024-03-24T09:23:50.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a577fe640 1 -- 172.21.15.116:0/3334056790 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f0a70012450 con 0x5598a5f93c60 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a81588740 10 client.16239 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a81588740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a81588740 20 client.16239 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a81588740 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a600be70 0x5598a600e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a81588740 4 client.16239 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5598a5c41410 con 0x5598a600be70 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a77fff640 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a600be70 0x5598a600e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a577fe640 10 client.16239 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.455+0000 7f0a77fff640 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a600be70 0x5598a600e290 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fd9904689b86bd9 server_cookie=d39d6ce13a3c473d in_seq=0 out_seq=0 2024-03-24T09:23:50.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a577fe640 1 -- 172.21.15.116:0/3334056790 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5598a5c41410 con 0x5598a600be70 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a577fe640 10 client.16239 handle_command_reply: tid=0 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a81588740 1 client.16239 shutdown 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 msgr2=0x5598a6009e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a81588740 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 0x5598a6009e40 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f0a700132b0 tx=0x7f0a70009500 comp rx=0 tx=0).stop 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 shutdown_connections 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.456+0000 7f0a81588740 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5598a5f93c60 0x5598a6009e40 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5598a600be70 0x5598a600e290 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5598a5f8b080 0x5598a5f99220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 1 --2- 172.21.15.116:0/3334056790 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5598a5f8a760 0x5598a5f98ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a577fe640 0 client.16239 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 >> 172.21.15.116:0/3334056790 conn(0x5598a5f84e90 msgr2=0x5598a5f83ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 shutdown_connections 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 1 -- 172.21.15.116:0/3334056790 wait complete. 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a81588740 20 client.16239 trim_cache size 0 max 16384 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "b74a7c56-4216-49d5-a804-bab5472bad85", 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:23:50.464 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:23:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0a600a22b0 msgr2=0x7f0a600a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0a600a22b0 0x7f0a600a4770 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5598a5f86020 tx=0x7f0a70009b70 comp rx=0 tx=0).stop 2024-03-24T09:23:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 msgr2=0x7f0a78102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 0x7f0a78102210 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f0a68000c00 tx=0x7f0a68005e30 comp rx=0 tx=0).stop 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 shutdown_connections 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0a600a22b0 0x7f0a600a4770 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0a78106660 0x7f0a78102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a78100f60 0x7f0a78102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 --2- 172.21.15.116:0/482662086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a781005a0 0x7f0a78102210 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 >> 172.21.15.116:0/482662086 conn(0x7f0a78077710 msgr2=0x7f0a780ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 shutdown_connections 2024-03-24T09:23:50.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.457+0000 7f0a7e8cd640 1 -- 172.21.15.116:0/482662086 wait complete. 2024-03-24T09:23:50.469 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:23:50.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 -- 172.21.15.116:0/3307721934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec100f70 msgr2=0x7f67ec10d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 --2- 172.21.15.116:0/3307721934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec100f70 0x7f67ec10d480 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f67e4009cd0 tx=0x7f67e402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 -- 172.21.15.116:0/3307721934 shutdown_connections 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 --2- 172.21.15.116:0/3307721934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec100f70 0x7f67ec10d480 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 --2- 172.21.15.116:0/3307721934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec1005b0 0x7f67ec100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 --2- 172.21.15.116:0/3307721934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67ec1055d0 0x7f67ec1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 -- 172.21.15.116:0/3307721934 >> 172.21.15.116:0/3307721934 conn(0x7f67ec0776a0 msgr2=0x7f67ec077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 -- 172.21.15.116:0/3307721934 shutdown_connections 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 -- 172.21.15.116:0/3307721934 wait complete. 2024-03-24T09:23:50.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.601+0000 7f67f142e640 1 Processor -- start 2024-03-24T09:23:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 -- start start 2024-03-24T09:23:50.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 0x7f67ec1967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec100f70 0x7f67ec196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67ec1055d0 0x7f67ec19b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec197560 con 0x7f67ec1005b0 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec19bd50 con 0x7f67ec1055d0 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67f142e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec19bf10 con 0x7f67ec100f70 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 0x7f67ec1967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 0x7f67ec1967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49600/0 (socket says 172.21.15.116:49600) 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67eaffd640 1 -- 172.21.15.116:0/658237714 learned_addr learned my addr 172.21.15.116:0/658237714 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67ec1055d0 0x7f67ec19b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.602+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec100f70 0x7f67ec196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67eaffd640 1 -- 172.21.15.116:0/658237714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec100f70 msgr2=0x7f67ec196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67eaffd640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec100f70 0x7f67ec196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67eaffd640 1 -- 172.21.15.116:0/658237714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67ec1055d0 msgr2=0x7f67ec19b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67eaffd640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67ec1055d0 0x7f67ec19b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67eaffd640 1 -- 172.21.15.116:0/658237714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e4009900 con 0x7f67ec1005b0 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec100f70 0x7f67ec196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67eaffd640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 0x7f67ec1967c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00e9c0 tx=0x7f67dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67d3fff640 1 -- 172.21.15.116:0/658237714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc009700 con 0x7f67ec1005b0 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67d3fff640 1 -- 172.21.15.116:0/658237714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67dc0044d0 con 0x7f67ec1005b0 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67ec19c1f0 con 0x7f67ec1005b0 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67d3fff640 1 -- 172.21.15.116:0/658237714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc010430 con 0x7f67ec1005b0 2024-03-24T09:23:50.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.603+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ec19c700 con 0x7f67ec1005b0 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.605+0000 7f67d3fff640 1 -- 172.21.15.116:0/658237714 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f67dc003f30 con 0x7f67ec1005b0 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 Processor -- start 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 -- start start 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67d3fff640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67d40a2230 0x7f67d40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcf55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67d3fff640 1 -- 172.21.15.116:0/658237714 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67dc014070 con 0x7f67ec1005b0 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fd00ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67d40a2230 0x7f67d40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a76fc19f30 con 0x55a76fcf5af0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a76fc63810 con 0x55a76fcfe6d0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67f4012740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a76fc60d40 con 0x55a76fcf51d0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfe190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49614/0 (socket says 172.21.15.116:49614) 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.606+0000 7f67ea7fc640 1 -- 172.21.15.116:0/1468508962 learned_addr learned my addr 172.21.15.116:0/1468508962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fd00ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67ea7fc640 1 -- 172.21.15.116:0/1468508962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 msgr2=0x55a76fcf55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcf55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67ea7fc640 1 -- 172.21.15.116:0/1468508962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 msgr2=0x55a76fd00ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fd00ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67ea7fc640 1 -- 172.21.15.116:0/1468508962 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a76fc1dea0 con 0x55a76fcf5af0 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fd00ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:23:50.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67d40a2230 0x7f67d40a46f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f67e402c630 tx=0x7f67e4032f10 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfe190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f67e000bd30 tx=0x7f67e000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a156ff00a7a5aa54 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67d17fa640 1 -- 172.21.15.116:0/1468508962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e001a660 con 0x55a76fcf5af0 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67d17fa640 1 -- 172.21.15.116:0/1468508962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67e000be20 con 0x55a76fcf5af0 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.607+0000 7f67d17fa640 1 -- 172.21.15.116:0/1468508962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0011a40 con 0x55a76fcf5af0 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 -- 172.21.15.116:0/1468508962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 msgr2=0x55a76fcfe190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfe190 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f67e000bd30 tx=0x7f67e000f730 comp rx=0 tx=0).stop 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 -- 172.21.15.116:0/1468508962 shutdown_connections 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fd00ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfe190 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 --2- 172.21.15.116:0/1468508962 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcf55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 -- 172.21.15.116:0/1468508962 >> 172.21.15.116:0/1468508962 conn(0x55a76fcef900 msgr2=0x55a76fcefce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 -- 172.21.15.116:0/1468508962 shutdown_connections 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 -- 172.21.15.116:0/1468508962 wait complete. 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.608+0000 7f67f4012740 1 -- start start 2024-03-24T09:23:50.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 Processor -- start 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcfb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fcfc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a76fc5c520 con 0x55a76fcf5af0 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a76f92f430 con 0x55a76fcfe6d0 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a76f8ebe00 con 0x55a76fcf51d0 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67f4012740 10 client.0 _ll_register_callbacks cb 0x55a76fbfad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67eaffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcfb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49626/0 (socket says 172.21.15.116:49626) 2024-03-24T09:23:50.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67ea7fc640 1 -- 172.21.15.116:0/3484718474 learned_addr learned my addr 172.21.15.116:0/3484718474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.609+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fcfc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ea7fc640 1 -- 172.21.15.116:0/3484718474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 msgr2=0x55a76fcfb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcfb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ea7fc640 1 -- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 msgr2=0x55a76fcfc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fcfc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ea7fc640 1 -- 172.21.15.116:0/3484718474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e000b960 con 0x55a76fcf5af0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfbce0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f67e0005880 tx=0x7f67e0005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d5d48a1d92c555 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ceff5640 1 -- 172.21.15.116:0/3484718474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0020070 con 0x55a76fcf5af0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ceff5640 1 -- 172.21.15.116:0/3484718474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67e0011820 con 0x55a76fcf5af0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a76f95b7e0 con 0x55a76fcf5af0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67ceff5640 1 -- 172.21.15.116:0/3484718474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f67e0005670 con 0x55a76fcf5af0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67f4012740 10 client.16251 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.610+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a76fc5aba0 con 0x55a76fcf5af0 2024-03-24T09:23:50.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67ceff5640 1 -- 172.21.15.116:0/3484718474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a76f95b7e0 con 0x55a76fcf5af0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67ceff5640 1 -- 172.21.15.116:0/3484718474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f67e001a460 con 0x55a76fcf5af0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67f4012740 10 client.16251 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67f4012740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67f4012740 20 client.16251 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67f4012740 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd11000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67f4012740 4 client.16251 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a76f998720 con 0x55a76fd0ebe0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.611+0000 7f67eaffd640 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd11000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.612+0000 7f67ceff5640 10 client.16251 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.612+0000 7f67eaffd640 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd11000 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1079d71f1c85044 server_cookie=ce4767de2ace0440 in_seq=0 out_seq=0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.613+0000 7f67ceff5640 1 -- 172.21.15.116:0/3484718474 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a76f998720 con 0x55a76fd0ebe0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.613+0000 7f67ceff5640 10 client.16251 handle_command_reply: tid=0 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.613+0000 7f67f4012740 1 client.16251 shutdown 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.613+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 msgr2=0x55a76fcfbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.613+0000 7f67f4012740 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfbce0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f67e0005880 tx=0x7f67e0005270 comp rx=0 tx=0).stop 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 shutdown_connections 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd11000 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67ceff5640 0 client.16251 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcfe6d0 0x55a76fcfc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fcfbce0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 --2- 172.21.15.116:0/3484718474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcf51d0 0x55a76fcfb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 >> 172.21.15.116:0/3484718474 conn(0x55a76fcef900 msgr2=0x55a76fcf1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 shutdown_connections 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 1 -- 172.21.15.116:0/3484718474 wait complete. 2024-03-24T09:23:50.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.614+0000 7f67f4012740 20 client.16251 trim_cache size 0 max 16384 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 Processor -- start 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 -- start start 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 0x55a76fcfded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf5af0 0x55a76fd09f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fcee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a76fbe1430 con 0x55a76fcf51d0 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a76fcf0900 con 0x55a76fcf5af0 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67f4012740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a76fcf4c10 con 0x55a76fcfe6d0 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 0x55a76fcfded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 0x55a76fcfded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49636/0 (socket says 172.21.15.116:49636) 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67eaffd640 1 -- 172.21.15.116:0/2611489085 learned_addr learned my addr 172.21.15.116:0/2611489085 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf5af0 0x55a76fd09f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.637+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fcee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67eaffd640 1 -- 172.21.15.116:0/2611489085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 msgr2=0x55a76fcee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67eaffd640 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fcee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67eaffd640 1 -- 172.21.15.116:0/2611489085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf5af0 msgr2=0x55a76fd09f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67eaffd640 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf5af0 0x55a76fd09f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67eaffd640 1 -- 172.21.15.116:0/2611489085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a76fbf6b20 con 0x55a76fcf51d0 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67eaffd640 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 0x55a76fcfded0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f67dc09abe0 tx=0x7f67dc0036b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01a125f3ef3b4e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67d0ff9640 1 -- 172.21.15.116:0/2611489085 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc09f030 con 0x55a76fcf51d0 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67d0ff9640 1 -- 172.21.15.116:0/2611489085 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67dc0a3680 con 0x55a76fcf51d0 2024-03-24T09:23:50.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.638+0000 7f67d0ff9640 1 -- 172.21.15.116:0/2611489085 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc0cbec0 con 0x55a76fcf51d0 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 -- 172.21.15.116:0/2611489085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 msgr2=0x55a76fcfded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 0x55a76fcfded0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f67dc09abe0 tx=0x7f67dc0036b0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 -- 172.21.15.116:0/2611489085 shutdown_connections 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fcee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf5af0 0x55a76fd09f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 --2- 172.21.15.116:0/2611489085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf51d0 0x55a76fcfded0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 -- 172.21.15.116:0/2611489085 >> 172.21.15.116:0/2611489085 conn(0x55a76fcef900 msgr2=0x55a76fcee2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 -- 172.21.15.116:0/2611489085 shutdown_connections 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.639+0000 7f67f4012740 1 -- 172.21.15.116:0/2611489085 wait complete. 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 -- start start 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 Processor -- start 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf51d0 0x55a76fd02680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fd02bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fd73ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a76fd74400 con 0x55a76fcf5af0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fd02bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a76fd74540 con 0x55a76fcf51d0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a76fd74680 con 0x55a76fcfe6d0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67ea7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fd02bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49642/0 (socket says 172.21.15.116:49642) 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67ea7fc640 1 -- 172.21.15.116:0/2461354879 learned_addr learned my addr 172.21.15.116:0/2461354879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67eaffd640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf51d0 0x55a76fd02680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67f4012740 10 client.0 _ll_register_callbacks cb 0x7f67e000bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.640+0000 7f67eb7fe640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fd73ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ea7fc640 1 -- 172.21.15.116:0/2461354879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 msgr2=0x55a76fd73ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fd73ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ea7fc640 1 -- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf51d0 msgr2=0x55a76fd02680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf51d0 0x55a76fd02680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ea7fc640 1 -- 172.21.15.116:0/2461354879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ec19c700 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ea7fc640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fd02bc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x55a76fcfb000 tx=0x7f67e001a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886fe9fbb517acb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ceff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ceff5640 1 -- 172.21.15.116:0/2461354879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0020040 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ceff5640 1 -- 172.21.15.116:0/2461354879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67e000b3a0 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67e000be20 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67ceff5640 1 -- 172.21.15.116:0/2461354879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f67e0005670 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67f4012740 10 client.16257 fetch_fsmap learned FSMap version 32 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.641+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a76fc1c700 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67ceff5640 1 -- 172.21.15.116:0/2461354879 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67e000be20 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67ceff5640 1 -- 172.21.15.116:0/2461354879 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f67e003b020 con 0x55a76fcf5af0 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67f4012740 10 client.16257 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67f4012740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67f4012740 20 client.16257 populate_metadata read hostname 'smithi116' 2024-03-24T09:23:50.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67f4012740 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd77dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:23:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67f4012740 4 client.16257 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:23:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a76f9e7890 con 0x55a76fd0ebe0 2024-03-24T09:23:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.642+0000 7f67eaffd640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd77dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:23:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.643+0000 7f67eaffd640 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd77dc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e696df37dbf2ccc server_cookie=137621fa138915e1 in_seq=0 out_seq=0 2024-03-24T09:23:50.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.643+0000 7f67ceff5640 10 client.16257 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.645+0000 7f67ceff5640 1 -- 172.21.15.116:0/2461354879 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55a76f9e7890 con 0x55a76fd0ebe0 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.645+0000 7f67ceff5640 10 client.16257 handle_command_reply: tid=0 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.645+0000 7f67f4012740 1 client.16257 shutdown 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 msgr2=0x55a76fd02bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fd02bc0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x55a76fcfb000 tx=0x7f67e001a710 comp rx=0 tx=0).stop 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 shutdown_connections 2024-03-24T09:23:50.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a76fcfe6d0 0x55a76fd73ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a76fcf5af0 0x55a76fd02bc0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a76fd0ebe0 0x55a76fd77dc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 --2- 172.21.15.116:0/2461354879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a76fcf51d0 0x55a76fd02680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67ceff5640 0 client.16257 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 >> 172.21.15.116:0/2461354879 conn(0x55a76fcef900 msgr2=0x55a76fceeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 shutdown_connections 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 1 -- 172.21.15.116:0/2461354879 wait complete. 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.646+0000 7f67f4012740 20 client.16257 trim_cache size 0 max 16384 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:23:50.649 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "scrub active (444 inodes in the stack)", 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: "b74a7c56-4216-49d5-a804-bab5472bad85": { 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "b74a7c56-4216-49d5-a804-bab5472bad85", 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:23:50.650 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67d40a2230 msgr2=0x7f67d40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67d40a2230 0x7f67d40a46f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f67e402c630 tx=0x7f67e4032f10 comp rx=0 tx=0).stop 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 msgr2=0x7f67ec1967c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 0x7f67ec1967c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00e9c0 tx=0x7f67dc00ee80 comp rx=0 tx=0).stop 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 shutdown_connections 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67ec1055d0 0x7f67ec19b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67d40a2230 0x7f67d40a46f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67ec100f70 0x7f67ec196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 --2- 172.21.15.116:0/658237714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67ec1005b0 0x7f67ec1967c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:23:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 >> 172.21.15.116:0/658237714 conn(0x7f67ec0776a0 msgr2=0x7f67ec0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:23:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 shutdown_connections 2024-03-24T09:23:50.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:23:50.647+0000 7f67f142e640 1 -- 172.21.15.116:0/658237714 wait complete. 2024-03-24T09:23:50.659 INFO:tasks.cephfs.filesystem:scrub status for tag:b74a7c56-4216-49d5-a804-bab5472bad85 - {'path': '/', 'tag': 'b74a7c56-4216-49d5-a804-bab5472bad85', 'options': 'recursive,force'} 2024-03-24T09:23:51.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: pgmap v55: 129 pgs: 129 active+clean; 60 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.1 MiB/s wr, 260 op/s 2024-03-24T09:23:51.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:51.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:51.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:51.279 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: pgmap v55: 129 pgs: 129 active+clean; 60 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.1 MiB/s wr, 260 op/s 2024-03-24T09:23:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: pgmap v55: 129 pgs: 129 active+clean; 60 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 1.1 MiB/s wr, 260 op/s 2024-03-24T09:23:51.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:23:52.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:23:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:52.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:23:52.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:52.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:23:53.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:52 smithi143 ceph-mon[35965]: pgmap v56: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 353 op/s 2024-03-24T09:23:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:52 smithi117 ceph-mon[35788]: pgmap v56: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 353 op/s 2024-03-24T09:23:53.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:52 smithi116 ceph-mon[32207]: pgmap v56: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 353 op/s 2024-03-24T09:23:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:54 smithi143 ceph-mon[35965]: pgmap v57: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-03-24T09:23:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:54 smithi117 ceph-mon[35788]: pgmap v57: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-03-24T09:23:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:54 smithi116 ceph-mon[32207]: pgmap v57: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-03-24T09:23:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:56 smithi143 ceph-mon[35965]: pgmap v58: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-03-24T09:23:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:56 smithi117 ceph-mon[35788]: pgmap v58: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-03-24T09:23:57.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:56 smithi116 ceph-mon[32207]: pgmap v58: 129 pgs: 129 active+clean; 67 MiB data, 591 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-03-24T09:23:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:58 smithi143 ceph-mon[35965]: pgmap v59: 129 pgs: 129 active+clean; 74 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 302 op/s 2024-03-24T09:23:59.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:59.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:59.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:59.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:59.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:23:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:23:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:58 smithi117 ceph-mon[35788]: pgmap v59: 129 pgs: 129 active+clean; 74 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 302 op/s 2024-03-24T09:23:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:59.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:59.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:59.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:59.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:23:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:23:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:58 smithi116 ceph-mon[32207]: pgmap v59: 129 pgs: 129 active+clean; 74 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 302 op/s 2024-03-24T09:23:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:23:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:23:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:24:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:00 smithi143 ceph-mon[35965]: pgmap v60: 129 pgs: 129 active+clean; 74 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 180 op/s 2024-03-24T09:24:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:00 smithi117 ceph-mon[35788]: pgmap v60: 129 pgs: 129 active+clean; 74 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 180 op/s 2024-03-24T09:24:01.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:00 smithi116 ceph-mon[32207]: pgmap v60: 129 pgs: 129 active+clean; 74 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 180 op/s 2024-03-24T09:24:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:02 smithi143 ceph-mon[35965]: pgmap v61: 129 pgs: 129 active+clean; 81 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 274 op/s 2024-03-24T09:24:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:02 smithi117 ceph-mon[35788]: pgmap v61: 129 pgs: 129 active+clean; 81 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 274 op/s 2024-03-24T09:24:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:02 smithi116 ceph-mon[32207]: pgmap v61: 129 pgs: 129 active+clean; 81 MiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 274 op/s 2024-03-24T09:24:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:04 smithi143 ceph-mon[35965]: pgmap v62: 129 pgs: 129 active+clean; 81 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T09:24:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:04 smithi117 ceph-mon[35788]: pgmap v62: 129 pgs: 129 active+clean; 81 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T09:24:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:04 smithi116 ceph-mon[32207]: pgmap v62: 129 pgs: 129 active+clean; 81 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T09:24:07.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:06 smithi143 ceph-mon[35965]: pgmap v63: 129 pgs: 129 active+clean; 81 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T09:24:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:06 smithi117 ceph-mon[35788]: pgmap v63: 129 pgs: 129 active+clean; 81 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T09:24:07.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:06 smithi116 ceph-mon[32207]: pgmap v63: 129 pgs: 129 active+clean; 81 MiB data, 656 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-03-24T09:24:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:08 smithi143 ceph-mon[35965]: pgmap v64: 129 pgs: 129 active+clean; 90 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 274 op/s 2024-03-24T09:24:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:08 smithi117 ceph-mon[35788]: pgmap v64: 129 pgs: 129 active+clean; 90 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 274 op/s 2024-03-24T09:24:09.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:08 smithi116 ceph-mon[32207]: pgmap v64: 129 pgs: 129 active+clean; 90 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 274 op/s 2024-03-24T09:24:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:10 smithi143 ceph-mon[35965]: pgmap v65: 129 pgs: 129 active+clean; 90 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-03-24T09:24:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:10 smithi117 ceph-mon[35788]: pgmap v65: 129 pgs: 129 active+clean; 90 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-03-24T09:24:11.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:10 smithi116 ceph-mon[32207]: pgmap v65: 129 pgs: 129 active+clean; 90 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 187 op/s 2024-03-24T09:24:13.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:12 smithi143 ceph-mon[35965]: pgmap v66: 129 pgs: 129 active+clean; 97 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 277 op/s 2024-03-24T09:24:13.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:12 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:24:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:12 smithi117 ceph-mon[35788]: pgmap v66: 129 pgs: 129 active+clean; 97 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 277 op/s 2024-03-24T09:24:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:12 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:24:13.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:12 smithi116 ceph-mon[32207]: pgmap v66: 129 pgs: 129 active+clean; 97 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.1 MiB/s wr, 277 op/s 2024-03-24T09:24:13.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:12 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:24:14.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:13 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:24:14.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:13 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:24:14.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:13 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:24:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:13 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:24:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:13 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:24:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:13 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:24:14.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:13 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:24:14.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:13 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:24:14.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:13 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:24:15.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:15 smithi143 ceph-mon[35965]: pgmap v67: 129 pgs: 129 active+clean; 99 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-03-24T09:24:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:15 smithi117 ceph-mon[35788]: pgmap v67: 129 pgs: 129 active+clean; 99 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-03-24T09:24:15.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:15 smithi116 ceph-mon[32207]: pgmap v67: 129 pgs: 129 active+clean; 99 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 198 op/s 2024-03-24T09:24:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:17 smithi117 ceph-mon[35788]: pgmap v68: 129 pgs: 129 active+clean; 99 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-03-24T09:24:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:17 smithi116 ceph-mon[32207]: pgmap v68: 129 pgs: 129 active+clean; 99 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-03-24T09:24:17.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:17 smithi143 ceph-mon[35965]: pgmap v68: 129 pgs: 129 active+clean; 99 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 193 op/s 2024-03-24T09:24:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:19 smithi117 ceph-mon[35788]: pgmap v69: 129 pgs: 129 active+clean; 111 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-24T09:24:19.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:19 smithi116 ceph-mon[32207]: pgmap v69: 129 pgs: 129 active+clean; 111 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-24T09:24:19.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:19 smithi143 ceph-mon[35965]: pgmap v69: 129 pgs: 129 active+clean; 111 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-24T09:24:20.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:24:20.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.767+0000 7fa02bfc7640 1 -- 172.21.15.116:0/755453817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0241016b0 msgr2=0x7fa024101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.767+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/755453817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0241016b0 0x7fa024101b30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa014009930 tx=0x7fa01402bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 -- 172.21.15.116:0/755453817 shutdown_connections 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/755453817 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa024102070 0x7fa02410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/755453817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0241016b0 0x7fa024101b30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/755453817 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0241076c0 0x7fa024107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 -- 172.21.15.116:0/755453817 >> 172.21.15.116:0/755453817 conn(0x7fa0240fd140 msgr2=0x7fa0240ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 -- 172.21.15.116:0/755453817 shutdown_connections 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 -- 172.21.15.116:0/755453817 wait complete. 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 Processor -- start 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.768+0000 7fa02bfc7640 1 -- start start 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 0x7fa024073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0241076c0 0x7fa024073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa024074490 0x7fa0241a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241a6630 con 0x7fa024102070 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241a67a0 con 0x7fa024074490 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa0241a6960 con 0x7fa0241076c0 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 0x7fa024073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 0x7fa024073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44012/0 (socket says 172.21.15.116:44012) 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 -- 172.21.15.116:0/2141695692 learned_addr learned my addr 172.21.15.116:0/2141695692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:20.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02a564640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa024074490 0x7fa0241a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029562640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0241076c0 0x7fa024073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 -- 172.21.15.116:0/2141695692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0241076c0 msgr2=0x7fa024073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0241076c0 0x7fa024073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 -- 172.21.15.116:0/2141695692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa024074490 msgr2=0x7fa0241a6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa024074490 0x7fa0241a6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 -- 172.21.15.116:0/2141695692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c009840 con 0x7fa024102070 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa029d63640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 0x7fa024073a10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa02000e9f0 tx=0x7fa02000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa01affd640 1 -- 172.21.15.116:0/2141695692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020009720 con 0x7fa024102070 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa014009560 con 0x7fa024102070 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa01affd640 1 -- 172.21.15.116:0/2141695692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa0200054d0 con 0x7fa024102070 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa01affd640 1 -- 172.21.15.116:0/2141695692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa020010430 con 0x7fa024102070 2024-03-24T09:24:20.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.769+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a6f40 con 0x7fa024102070 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.771+0000 7fa01affd640 1 -- 172.21.15.116:0/2141695692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fa020004f90 con 0x7fa024102070 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 Processor -- start 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 -- start start 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa01affd640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0100a2260 0x7fa0100a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 0x55681c86a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c873190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c875ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55681c78ef30 con 0x55681c8736d0 2024-03-24T09:24:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55681c7d8810 con 0x55681c86a1d0 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02eba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55681c7d5d40 con 0x55681c86aaf0 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02a564640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0100a2260 0x7fa0100a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa029d63640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 0x55681c86a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa029562640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c873190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa029d63640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 0x55681c86a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34558/0 (socket says 172.21.15.116:34558) 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa029d63640 1 -- 172.21.15.116:0/3828198466 learned_addr learned my addr 172.21.15.116:0/3828198466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa02a564640 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c875ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.772+0000 7fa01affd640 1 -- 172.21.15.116:0/2141695692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa020014070 con 0x7fa024102070 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa029d63640 1 -- 172.21.15.116:0/3828198466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 msgr2=0x55681c873190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa029d63640 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c873190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa029d63640 1 -- 172.21.15.116:0/3828198466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 msgr2=0x55681c875ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa029d63640 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c875ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa029d63640 1 -- 172.21.15.116:0/3828198466 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55681c792ea0 con 0x55681c86a1d0 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa029d63640 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 0x55681c86a5b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa020009380 tx=0x7fa020097150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb4d7ff8cd0999d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa02a564640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0100a2260 0x7fa0100a4720 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa01c005fd0 tx=0x7fa01c0114f0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa00ffff640 1 -- 172.21.15.116:0/3828198466 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02009b4f0 con 0x55681c86a1d0 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa00ffff640 1 -- 172.21.15.116:0/3828198466 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa02006f2e0 con 0x55681c86a1d0 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.773+0000 7fa00ffff640 1 -- 172.21.15.116:0/3828198466 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02009ef40 con 0x55681c86a1d0 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 -- 172.21.15.116:0/3828198466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 msgr2=0x55681c86a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 0x55681c86a5b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa020009380 tx=0x7fa020097150 comp rx=0 tx=0).stop 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 -- 172.21.15.116:0/3828198466 shutdown_connections 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c875ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c873190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 --2- 172.21.15.116:0/3828198466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86a1d0 0x55681c86a5b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 -- 172.21.15.116:0/3828198466 >> 172.21.15.116:0/3828198466 conn(0x55681c864900 msgr2=0x55681c864ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 -- 172.21.15.116:0/3828198466 shutdown_connections 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 -- 172.21.15.116:0/3828198466 wait complete. 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 -- start start 2024-03-24T09:24:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.774+0000 7fa02eba7740 1 Processor -- start 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86a1d0 0x55681c870790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86aaf0 0x55681c870cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c871210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55681c7d1520 con 0x55681c8736d0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55681c4a4430 con 0x55681c86aaf0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55681c460e00 con 0x55681c86a1d0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02eba7740 10 client.0 _ll_register_callbacks cb 0x55681c76fd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c871210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa029562640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86aaf0 0x55681c870cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa029d63640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86a1d0 0x55681c870790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c871210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44030/0 (socket says 172.21.15.116:44030) 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa029d63640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86a1d0 0x55681c870790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56334/0 (socket says 172.21.15.116:56334) 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa029562640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86aaf0 0x55681c870cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34566/0 (socket says 172.21.15.116:34566) 2024-03-24T09:24:20.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 -- 172.21.15.116:0/2183568708 learned_addr learned my addr 172.21.15.116:0/2183568708 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 -- 172.21.15.116:0/2183568708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86a1d0 msgr2=0x55681c870790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86a1d0 0x55681c870790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 -- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86aaf0 msgr2=0x55681c870cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86aaf0 0x55681c870cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 -- 172.21.15.116:0/2183568708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0241a6f40 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa02a564640 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c871210 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00fe60 tx=0x7fa01c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4825a12e1fe64 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.775+0000 7fa00d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2183568708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c003e00 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55681c4d07e0 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2183568708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa01c004540 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2183568708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa01c004700 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 10 client.16269 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55681c7cfba0 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2183568708 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55681c4d07e0 con 0x55681c8736d0 2024-03-24T09:24:20.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2183568708 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa01c01d420 con 0x55681c8736d0 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 10 client.16269 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 10 client.16269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 20 client.16269 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c883c70 0x55681c886090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 4 client.16269 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55681c50d720 con 0x55681c883c70 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.776+0000 7fa029d63640 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c883c70 0x55681c886090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.777+0000 7fa00d7fa640 10 client.16269 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.777+0000 7fa029d63640 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c883c70 0x55681c886090 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b653632d5ff7a26 server_cookie=e6bd21f77b1136dc in_seq=0 out_seq=0 2024-03-24T09:24:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2183568708 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55681c50d720 con 0x55681c883c70 2024-03-24T09:24:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa00d7fa640 10 client.16269 handle_command_reply: tid=0 2024-03-24T09:24:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 client.16269 shutdown 2024-03-24T09:24:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 msgr2=0x55681c871210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c871210 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00fe60 tx=0x7fa01c009500 comp rx=0 tx=0).stop 2024-03-24T09:24:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 shutdown_connections 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c8736d0 0x55681c871210 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c883c70 0x55681c886090 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c86aaf0 0x55681c870cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa00d7fa640 0 client.16269 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2183568708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86a1d0 0x55681c870790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 >> 172.21.15.116:0/2183568708 conn(0x55681c864900 msgr2=0x55681c863b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 shutdown_connections 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 1 -- 172.21.15.116:0/2183568708 wait complete. 2024-03-24T09:24:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.778+0000 7fa02eba7740 20 client.16269 trim_cache size 0 max 16384 2024-03-24T09:24:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 Processor -- start 2024-03-24T09:24:20.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 -- start start 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c87ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c871030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c88b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55681c756430 con 0x55681c86a1d0 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55681c869ba0 con 0x55681c8736d0 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02eba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55681c8866f0 con 0x55681c86aaf0 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa029d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c87ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa029d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c87ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44038/0 (socket says 172.21.15.116:44038) 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa029562640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c871030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa029d63640 1 -- 172.21.15.116:0/1737856721 learned_addr learned my addr 172.21.15.116:0/1737856721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.796+0000 7fa02a564640 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c88b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa029d63640 1 -- 172.21.15.116:0/1737856721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 msgr2=0x55681c871030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa029d63640 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c871030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa029d63640 1 -- 172.21.15.116:0/1737856721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 msgr2=0x55681c88b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa029d63640 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c88b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa029d63640 1 -- 172.21.15.116:0/1737856721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55681c76bb20 con 0x55681c86a1d0 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa029d63640 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c87ef00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa02000c6a0 tx=0x7fa0200b0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d175673d9fdbf9e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa00f7fe640 1 -- 172.21.15.116:0/1737856721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02009b4f0 con 0x55681c86a1d0 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa00f7fe640 1 -- 172.21.15.116:0/1737856721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa02009adf0 con 0x55681c86a1d0 2024-03-24T09:24:20.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.797+0000 7fa00f7fe640 1 -- 172.21.15.116:0/1737856721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200990f0 con 0x55681c86a1d0 2024-03-24T09:24:20.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- 172.21.15.116:0/1737856721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 msgr2=0x55681c87ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c87ef00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa02000c6a0 tx=0x7fa0200b0040 comp rx=0 tx=0).stop 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- 172.21.15.116:0/1737856721 shutdown_connections 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c88b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c871030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- 172.21.15.116:0/1737856721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c87ef00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- 172.21.15.116:0/1737856721 >> 172.21.15.116:0/1737856721 conn(0x55681c864900 msgr2=0x55681c863b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- 172.21.15.116:0/1737856721 shutdown_connections 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- 172.21.15.116:0/1737856721 wait complete. 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- start start 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 Processor -- start 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c86d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c86d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c86de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55681c8e8f90 con 0x55681c86a1d0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55681c8e90d0 con 0x55681c8736d0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55681c8e9210 con 0x55681c86aaf0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.798+0000 7fa02eba7740 10 client.0 _ll_register_callbacks cb 0x7fa01c009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c86d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c86d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44040/0 (socket says 172.21.15.116:44040) 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 -- 172.21.15.116:0/2814860823 learned_addr learned my addr 172.21.15.116:0/2814860823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029562640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c86d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa02a564640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c86de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 -- 172.21.15.116:0/2814860823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 msgr2=0x55681c86d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c86d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 -- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 msgr2=0x55681c86de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c86de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 -- 172.21.15.116:0/2814860823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55681c792ea0 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa029d63640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c86d3a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa0200990a0 tx=0x7fa02000cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d021f4e78a29799 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa00d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2814860823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0200a3590 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2814860823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa0200b3040 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa02006f2e0 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2814860823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa02009b4f0 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa02eba7740 10 client.16275 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55681c791700 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.799+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2814860823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa02006f2e0 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2814860823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa02001f060 con 0x55681c86a1d0 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa02eba7740 10 client.16275 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa02eba7740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa02eba7740 20 client.16275 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:20.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c8ea660 0x55681c8eca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa02eba7740 4 client.16275 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55681c55c890 con 0x55681c8ea660 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa029562640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c8ea660 0x55681c8eca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa00d7fa640 10 client.16275 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.800+0000 7fa029562640 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c8ea660 0x55681c8eca80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10255b7c489d79b4 server_cookie=c35bbee9eb3a04df in_seq=0 out_seq=0 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa00d7fa640 1 -- 172.21.15.116:0/2814860823 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55681c55c890 con 0x55681c8ea660 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa00d7fa640 10 client.16275 handle_command_reply: tid=0 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 client.16275 shutdown 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 msgr2=0x55681c86d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c86d3a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa0200990a0 tx=0x7fa02000cee0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 shutdown_connections 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55681c8ea660 0x55681c8eca80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55681c8736d0 0x55681c86de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa00d7fa640 0 client.16275 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:20.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55681c86aaf0 0x55681c86d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 --2- 172.21.15.116:0/2814860823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55681c86a1d0 0x55681c86d3a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 >> 172.21.15.116:0/2814860823 conn(0x55681c864900 msgr2=0x55681c865ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 shutdown_connections 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 1 -- 172.21.15.116:0/2814860823 wait complete. 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02eba7740 20 client.16275 trim_cache size 0 max 16384 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:24:20.808 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:24:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0100a2260 msgr2=0x7fa0100a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0100a2260 0x7fa0100a4720 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa01c005fd0 tx=0x7fa01c0114f0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 msgr2=0x7fa024073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 0x7fa024073a10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa02000e9f0 tx=0x7fa02000edf0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 shutdown_connections 2024-03-24T09:24:20.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa024074490 0x7fa0241a6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0100a2260 0x7fa0100a4720 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0241076c0 0x7fa024073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 --2- 172.21.15.116:0/2141695692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa024102070 0x7fa024073a10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.801+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 >> 172.21.15.116:0/2141695692 conn(0x7fa0240fd140 msgr2=0x7fa0240fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.802+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 shutdown_connections 2024-03-24T09:24:20.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.802+0000 7fa02bfc7640 1 -- 172.21.15.116:0/2141695692 wait complete. 2024-03-24T09:24:20.813 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:24:20.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- 172.21.15.116:0/3470700422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68181076c0 msgr2=0x7f6818107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- 172.21.15.116:0/3470700422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68181076c0 0x7f6818107aa0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6808009930 tx=0x7f680802bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- 172.21.15.116:0/3470700422 shutdown_connections 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- 172.21.15.116:0/3470700422 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6818102070 0x7f681810e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- 172.21.15.116:0/3470700422 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68181016b0 0x7f6818101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- 172.21.15.116:0/3470700422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68181076c0 0x7f6818107aa0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- 172.21.15.116:0/3470700422 >> 172.21.15.116:0/3470700422 conn(0x7f68180fd1a0 msgr2=0x7f68180ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- 172.21.15.116:0/3470700422 shutdown_connections 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- 172.21.15.116:0/3470700422 wait complete. 2024-03-24T09:24:20.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 Processor -- start 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- start start 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68181016b0 0x7f681819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 0x7f681819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68181076c0 0x7f6818197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68181980f0 con 0x7f6818102070 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6818198260 con 0x7f68181076c0 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.919+0000 7f681f902640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6818198420 con 0x7f68181016b0 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 0x7f681819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681d69e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68181016b0 0x7f681819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 0x7f681819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44070/0 (socket says 172.21.15.116:44070) 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 -- 172.21.15.116:0/635846811 learned_addr learned my addr 172.21.15.116:0/635846811 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681de9f640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68181076c0 0x7f6818197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 -- 172.21.15.116:0/635846811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68181016b0 msgr2=0x7f681819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68181016b0 0x7f681819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 -- 172.21.15.116:0/635846811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68181076c0 msgr2=0x7f6818197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68181076c0 0x7f6818197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 -- 172.21.15.116:0/635846811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6808009560 con 0x7f6818102070 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681ce9d640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 0x7f681819de30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f681400d950 tx=0x7f681400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6814009a30 con 0x7f6818102070 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6818198700 con 0x7f6818102070 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68140044d0 con 0x7f6818102070 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68140047a0 con 0x7f6818102070 2024-03-24T09:24:20.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.920+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68181a4990 con 0x7f6818102070 2024-03-24T09:24:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.921+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6818068500 con 0x7f6818102070 2024-03-24T09:24:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.922+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6814009b90 con 0x7f6818102070 2024-03-24T09:24:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.923+0000 7f680e7fc640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68040a2230 0x7f68040a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.923+0000 7f681d69e640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68040a2230 0x7f68040a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.923+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f68140cdda0 con 0x7f6818102070 2024-03-24T09:24:20.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.923+0000 7f681d69e640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68040a2230 0x7f68040a46f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6808005ab0 tx=0x7f6808005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:20.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:20.928+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6814096680 con 0x7f6818102070 2024-03-24T09:24:21.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.180+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6818198fd0 con 0x7f6818102070 2024-03-24T09:24:21.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.181+0000 7f680e7fc640 1 -- 172.21.15.116:0/635846811 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7f6814099d40 con 0x7f6818102070 2024-03-24T09:24:21.182 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:24:21.182 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:24:21.182 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68040a2230 msgr2=0x7f68040a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68040a2230 0x7f68040a46f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6808005ab0 tx=0x7f6808005fd0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 msgr2=0x7f681819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 0x7f681819de30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f681400d950 tx=0x7f681400de10 comp rx=0 tx=0).stop 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 shutdown_connections 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68181076c0 0x7f6818197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6818102070 0x7f681819de30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68040a2230 0x7f68040a46f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 --2- 172.21.15.116:0/635846811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68181016b0 0x7f681819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 >> 172.21.15.116:0/635846811 conn(0x7f68180fd1a0 msgr2=0x7f68180fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 shutdown_connections 2024-03-24T09:24:21.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.183+0000 7f681f902640 1 -- 172.21.15.116:0/635846811 wait complete. 2024-03-24T09:24:21.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 -- 172.21.15.116:0/2267646819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 msgr2=0x7f79b0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 --2- 172.21.15.116:0/2267646819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0100a20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009900 tx=0x7f79a402bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 -- 172.21.15.116:0/2267646819 shutdown_connections 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 --2- 172.21.15.116:0/2267646819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79b0100f60 0x7f79b010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 --2- 172.21.15.116:0/2267646819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0100a20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 --2- 172.21.15.116:0/2267646819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0106660 0x7f79b0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.304+0000 7f79b5055640 1 -- 172.21.15.116:0/2267646819 >> 172.21.15.116:0/2267646819 conn(0x7f79b0077710 msgr2=0x7f79b0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.305+0000 7f79b5055640 1 -- 172.21.15.116:0/2267646819 shutdown_connections 2024-03-24T09:24:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.305+0000 7f79b5055640 1 -- 172.21.15.116:0/2267646819 wait complete. 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.305+0000 7f79b5055640 1 Processor -- start 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 -- start start 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0100f60 0x7f79b0198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79b0106660 0x7f79b01a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79b01997c0 con 0x7f79b01005a0 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f79b01a3fe0 con 0x7f79b0100f60 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79b5055640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f79b01a4120 con 0x7f79b0106660 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44092/0 (socket says 172.21.15.116:44092) 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 -- 172.21.15.116:0/2684772508 learned_addr learned my addr 172.21.15.116:0/2684772508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79af59e640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79b0106660 0x7f79b01a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79ae59c640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0100f60 0x7f79b0198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 -- 172.21.15.116:0/2684772508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79b0106660 msgr2=0x7f79b01a3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79b0106660 0x7f79b01a3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 -- 172.21.15.116:0/2684772508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0100f60 msgr2=0x7f79b0198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0100f60 0x7f79b0198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79aed9d640 1 -- 172.21.15.116:0/2684772508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a4009560 con 0x7f79b01005a0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.306+0000 7f79ae59c640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0100f60 0x7f79b0198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.307+0000 7f79aed9d640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0198a20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f79a000e9f0 tx=0x7f79a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.307+0000 7f7997fff640 1 -- 172.21.15.116:0/2684772508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0009700 con 0x7f79b01005a0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.307+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b01a4380 con 0x7f79b01005a0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.307+0000 7f7997fff640 1 -- 172.21.15.116:0/2684772508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a000cd80 con 0x7f79b01005a0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.307+0000 7f7997fff640 1 -- 172.21.15.116:0/2684772508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0018840 con 0x7f79b01005a0 2024-03-24T09:24:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.307+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b01a48b0 con 0x7f79b01005a0 2024-03-24T09:24:21.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 Processor -- start 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f7997fff640 1 -- 172.21.15.116:0/2684772508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f79a00189a0 con 0x7f79b01005a0 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 -- start start 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aabce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557880a0e840 con 0x557880aa98f0 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557880a0bd70 con 0x557880aa0d10 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79b7c39740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557880a07550 con 0x557880aa03f0 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79ae59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aabce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79af59e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79aed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79af59e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56408/0 (socket says 172.21.15.116:56408) 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79ae59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aabce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44106/0 (socket says 172.21.15.116:44106) 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.309+0000 7f79af59e640 1 -- 172.21.15.116:0/553381491 learned_addr learned my addr 172.21.15.116:0/553381491 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 -- 172.21.15.116:0/553381491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 msgr2=0x557880aa93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f7997fff640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79980a2230 0x7f79980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 -- 172.21.15.116:0/553381491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 msgr2=0x557880aabce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aabce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 -- 172.21.15.116:0/553381491 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55788099cde0 con 0x557880aa03f0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79ae59c640 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aabce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79aed9d640 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f7997fff640 1 -- 172.21.15.116:0/2684772508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f79a0014070 con 0x7f79b01005a0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79980a2230 0x7f79980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79af59e640 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa07d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009fd0 tx=0x7f79a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abba7f63131b38f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79957fa640 1 -- 172.21.15.116:0/553381491 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8004230 con 0x557880aa03f0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79957fa640 1 -- 172.21.15.116:0/553381491 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a8003df0 con 0x557880aa03f0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.310+0000 7f79957fa640 1 -- 172.21.15.116:0/553381491 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a801b420 con 0x557880aa03f0 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 -- 172.21.15.116:0/553381491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 msgr2=0x557880aa07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa07d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009fd0 tx=0x7f79a8011810 comp rx=0 tx=0).stop 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79af59e640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79980a2230 0x7f79980a46f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79a8010010 tx=0x7f79a801bb40 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 -- 172.21.15.116:0/553381491 shutdown_connections 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aabce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 --2- 172.21.15.116:0/553381491 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa07d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 -- 172.21.15.116:0/553381491 >> 172.21.15.116:0/553381491 conn(0x557880a9ab20 msgr2=0x557880a9af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 -- 172.21.15.116:0/553381491 shutdown_connections 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 -- 172.21.15.116:0/553381491 wait complete. 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 -- start start 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.311+0000 7f79b7c39740 1 Processor -- start 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aa73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5578806da430 con 0x557880aa98f0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557880696e00 con 0x557880aa0d10 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55788098c590 con 0x557880aa03f0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 10 client.0 _ll_register_callbacks cb 0x5578809fbfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aa73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79aed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aa73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44114/0 (socket says 172.21.15.116:44114) 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 -- 172.21.15.116:0/1056914866 learned_addr learned my addr 172.21.15.116:0/1056914866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79ae59c640 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 -- 172.21.15.116:0/1056914866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 msgr2=0x557880aa6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 -- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 msgr2=0x557880aa6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 -- 172.21.15.116:0/1056914866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a8009c80 con 0x557880aa98f0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79af59e640 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aa73b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f79a8006010 tx=0x7f79a8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0353c958d66b53d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f7992ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f7992ff5640 1 -- 172.21.15.116:0/1056914866 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8003c90 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f7992ff5640 1 -- 172.21.15.116:0/1056914866 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f79a801b240 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5578807067e0 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f7992ff5640 1 -- 172.21.15.116:0/1056914866 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f79a800f990 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 10 client.16293 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.312+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557880a05bd0 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f7992ff5640 1 -- 172.21.15.116:0/1056914866 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5578807067e0 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f7992ff5640 1 -- 172.21.15.116:0/1056914866 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f79a8005280 con 0x557880aa98f0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79b7c39740 10 client.16293 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79b7c39740 10 client.16293 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79b7c39740 20 client.16293 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79b7c39740 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880abbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79b7c39740 4 client.16293 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557880743720 con 0x557880ab99e0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79aed9d640 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880abbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f7992ff5640 10 client.16293 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.313+0000 7f79aed9d640 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880abbe00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c59f277f113ac2e9 server_cookie=c874ad86a6895a7c in_seq=0 out_seq=0 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f7992ff5640 1 -- 172.21.15.116:0/1056914866 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557880743720 con 0x557880ab99e0 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f7992ff5640 10 client.16293 handle_command_reply: tid=0 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f79b7c39740 1 client.16293 shutdown 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 msgr2=0x557880aa73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f79b7c39740 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aa73b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f79a8006010 tx=0x7f79a8016000 comp rx=0 tx=0).stop 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 shutdown_connections 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.314+0000 7f79b7c39740 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880abbe00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880aa73b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 1 --2- 172.21.15.116:0/1056914866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880aa6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f7992ff5640 0 client.16293 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 >> 172.21.15.116:0/1056914866 conn(0x557880a9ab20 msgr2=0x557880a9c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 shutdown_connections 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 1 -- 172.21.15.116:0/1056914866 wait complete. 2024-03-24T09:24:21.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.315+0000 7f79b7c39740 20 client.16293 trim_cache size 0 max 16384 2024-03-24T09:24:21.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:21 smithi117 ceph-mon[35788]: pgmap v70: 129 pgs: 129 active+clean; 111 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-03-24T09:24:21.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:21 smithi116 ceph-mon[32207]: pgmap v70: 129 pgs: 129 active+clean; 111 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-03-24T09:24:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 Processor -- start 2024-03-24T09:24:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 -- start start 2024-03-24T09:24:21.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880ab5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880ac1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557880a9a650 con 0x557880aa98f0 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557880743720 con 0x557880aa0d10 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79b7c39740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5578807067e0 con 0x557880aa03f0 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880ac1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880ac1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44124/0 (socket says 172.21.15.116:44124) 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 -- 172.21.15.116:0/3414143392 learned_addr learned my addr 172.21.15.116:0/3414143392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79aed9d640 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880ab5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79ae59c640 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 -- 172.21.15.116:0/3414143392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 msgr2=0x557880ab5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880ab5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 -- 172.21.15.116:0/3414143392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 msgr2=0x557880aa71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 -- 172.21.15.116:0/3414143392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578809a1c80 con 0x557880aa98f0 2024-03-24T09:24:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.330+0000 7f79af59e640 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880ac1710 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f79a801bf80 tx=0x7f79a8009320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7057644d3198627 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f7994ff9640 1 -- 172.21.15.116:0/3414143392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a8011d10 con 0x557880aa98f0 2024-03-24T09:24:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f7994ff9640 1 -- 172.21.15.116:0/3414143392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a800f690 con 0x557880aa98f0 2024-03-24T09:24:21.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f7994ff9640 1 -- 172.21.15.116:0/3414143392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a801b040 con 0x557880aa98f0 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 -- 172.21.15.116:0/3414143392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 msgr2=0x557880ac1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880ac1710 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f79a801bf80 tx=0x7f79a8009320 comp rx=0 tx=0).stop 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 -- 172.21.15.116:0/3414143392 shutdown_connections 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa98f0 0x557880ac1710 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aa71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3414143392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa03f0 0x557880ab5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.331+0000 7f79b7c39740 1 -- 172.21.15.116:0/3414143392 >> 172.21.15.116:0/3414143392 conn(0x557880a9ab20 msgr2=0x557880a9c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 -- 172.21.15.116:0/3414143392 shutdown_connections 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 -- 172.21.15.116:0/3414143392 wait complete. 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 -- start start 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 Processor -- start 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 0x557880aad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aad7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa98f0 0x557880aadd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557880b1ede0 con 0x557880aa03f0 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557880b1ef50 con 0x557880aa0d10 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557880b1f110 con 0x557880aa98f0 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79b7c39740 10 client.0 _ll_register_callbacks cb 0x7f79a801e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 0x557880aad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 0x557880aad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44128/0 (socket says 172.21.15.116:44128) 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79aed9d640 1 -- 172.21.15.116:0/3050597349 learned_addr learned my addr 172.21.15.116:0/3050597349 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79ae59c640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aad7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.332+0000 7f79af59e640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa98f0 0x557880aadd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79aed9d640 1 -- 172.21.15.116:0/3050597349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa98f0 msgr2=0x557880aadd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79aed9d640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa98f0 0x557880aadd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79aed9d640 1 -- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 msgr2=0x557880aad7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79aed9d640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aad7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79aed9d640 1 -- 172.21.15.116:0/3050597349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557880a05bd0 con 0x557880aa03f0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79aed9d640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 0x557880aad2a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f79a0097270 tx=0x7f79a00971d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b17a6714f034349 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f7992ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f7992ff5640 1 -- 172.21.15.116:0/3050597349 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a009e070 con 0x557880aa03f0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f7992ff5640 1 -- 172.21.15.116:0/3050597349 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f79a009ae10 con 0x557880aa03f0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79a8003df0 con 0x557880aa03f0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f7992ff5640 1 -- 172.21.15.116:0/3050597349 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f79a009eeb0 con 0x557880aa03f0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 10 client.16299 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5578809c7920 con 0x557880aa03f0 2024-03-24T09:24:21.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f7992ff5640 1 -- 172.21.15.116:0/3050597349 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79a00b9020 con 0x557880aa03f0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f7992ff5640 1 -- 172.21.15.116:0/3050597349 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f79a0097480 con 0x557880aa03f0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 10 client.16299 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 10 client.16299 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 20 client.16299 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880b22930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 4 client.16299 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.333+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557880792890 con 0x557880ab99e0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79ae59c640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880b22930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79ae59c640 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880b22930 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6b7375bfa4a2be51 server_cookie=5b07ce974efdf7c in_seq=0 out_seq=0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f7992ff5640 10 client.16299 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f7992ff5640 1 -- 172.21.15.116:0/3050597349 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557880792890 con 0x557880ab99e0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f7992ff5640 10 client.16299 handle_command_reply: tid=0 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 client.16299 shutdown 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 msgr2=0x557880aad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 0x557880aad2a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f79a0097270 tx=0x7f79a00971d0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 shutdown_connections 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557880aa98f0 0x557880aadd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557880aa0d10 0x557880aad7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x557880ab99e0 0x557880b22930 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 --2- 172.21.15.116:0/3050597349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557880aa03f0 0x557880aad2a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f7992ff5640 0 client.16299 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 >> 172.21.15.116:0/3050597349 conn(0x557880a9ab20 msgr2=0x557880a9bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 shutdown_connections 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.334+0000 7f79b7c39740 1 -- 172.21.15.116:0/3050597349 wait complete. 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b7c39740 20 client.16299 trim_cache size 0 max 16384 2024-03-24T09:24:21.341 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79980a2230 msgr2=0x7f79980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79980a2230 0x7f79980a46f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79a8010010 tx=0x7f79a801bb40 comp rx=0 tx=0).stop 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 msgr2=0x7f79b0198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0198a20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f79a000e9f0 tx=0x7f79a000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 shutdown_connections 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79980a2230 0x7f79980a46f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79b0106660 0x7f79b01a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79b0100f60 0x7f79b0198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 --2- 172.21.15.116:0/2684772508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79b01005a0 0x7f79b0198a20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 >> 172.21.15.116:0/2684772508 conn(0x7f79b0077710 msgr2=0x7f79b00ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 shutdown_connections 2024-03-24T09:24:21.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.335+0000 7f79b5055640 1 -- 172.21.15.116:0/2684772508 wait complete. 2024-03-24T09:24:21.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:24:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 -- 172.21.15.116:0/3323227725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470100f60 msgr2=0x7f147010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 --2- 172.21.15.116:0/3323227725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470100f60 0x7f147010d420 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1460009900 tx=0x7f146002bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 -- 172.21.15.116:0/3323227725 shutdown_connections 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 --2- 172.21.15.116:0/3323227725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470100f60 0x7f147010d420 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 --2- 172.21.15.116:0/3323227725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14701005a0 0x7f1470100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 --2- 172.21.15.116:0/3323227725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1470106660 0x7f1470106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 -- 172.21.15.116:0/3323227725 >> 172.21.15.116:0/3323227725 conn(0x7f1470077710 msgr2=0x7f1470077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 -- 172.21.15.116:0/3323227725 shutdown_connections 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.455+0000 7f14771f4640 1 -- 172.21.15.116:0/3323227725 wait complete. 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 Processor -- start 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 -- start start 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14701005a0 0x7f1470198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470100f60 0x7f1470198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 0x7f14701a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1470199730 con 0x7f1470106660 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f14701a3f10 con 0x7f1470100f60 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f14771f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f14701a4050 con 0x7f14701005a0 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 0x7f14701a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f146ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470100f60 0x7f1470198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 0x7f14701a39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44150/0 (socket says 172.21.15.116:44150) 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 -- 172.21.15.116:0/1260261574 learned_addr learned my addr 172.21.15.116:0/1260261574 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1474f90640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14701005a0 0x7f1470198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 -- 172.21.15.116:0/1260261574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14701005a0 msgr2=0x7f1470198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14701005a0 0x7f1470198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 -- 172.21.15.116:0/1260261574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470100f60 msgr2=0x7f1470198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470100f60 0x7f1470198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 -- 172.21.15.116:0/1260261574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1460009560 con 0x7f1470106660 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f1475791640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 0x7f14701a39d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1460009430 tx=0x7f1460004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.456+0000 7f146dffb640 1 -- 172.21.15.116:0/1260261574 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460004440 con 0x7f1470106660 2024-03-24T09:24:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.457+0000 7f146dffb640 1 -- 172.21.15.116:0/1260261574 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1460005270 con 0x7f1470106660 2024-03-24T09:24:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.457+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14701a42a0 con 0x7f1470106660 2024-03-24T09:24:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.457+0000 7f146dffb640 1 -- 172.21.15.116:0/1260261574 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460003bf0 con 0x7f1470106660 2024-03-24T09:24:21.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.457+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14701a4800 con 0x7f1470106660 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f146dffb640 1 -- 172.21.15.116:0/1260261574 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f1460003d50 con 0x7f1470106660 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 Processor -- start 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 -- start start 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f146dffb640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f145c0a22b0 0x7f145c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e794497d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e794523b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e79454ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e793b7840 con 0x564e794528f0 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564e793b4d70 con 0x564e79449d10 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1479dce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564e793b0550 con 0x564e794493f0 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1475791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e79454ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f146ffff640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f145c0a22b0 0x7f145c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1474f90640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e794497d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1475791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e79454ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44154/0 (socket says 172.21.15.116:44154) 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1475791640 1 -- 172.21.15.116:0/1273313987 learned_addr learned my addr 172.21.15.116:0/1273313987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.459+0000 7f1474f90640 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e794497d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56452/0 (socket says 172.21.15.116:56452) 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f146ffff640 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e794523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f146dffb640 1 -- 172.21.15.116:0/1260261574 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f14600f0420 con 0x7f1470106660 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1475791640 1 -- 172.21.15.116:0/1273313987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 msgr2=0x564e794497d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1475791640 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e794497d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1475791640 1 -- 172.21.15.116:0/1273313987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 msgr2=0x564e794523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1475791640 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e794523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1475791640 1 -- 172.21.15.116:0/1273313987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e79345de0 con 0x564e794528f0 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f146ffff640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f145c0a22b0 0x7f145c0a4770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1470069420 tx=0x7f14580098b0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1475791640 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e79454ce0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f14600bc870 tx=0x7f14600057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888817521b480b4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1466ffd640 1 -- 172.21.15.116:0/1273313987 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600bdea0 con 0x564e794528f0 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1466ffd640 1 -- 172.21.15.116:0/1273313987 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14600d8020 con 0x564e794528f0 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.460+0000 7f1466ffd640 1 -- 172.21.15.116:0/1273313987 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600c5310 con 0x564e794528f0 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 -- 172.21.15.116:0/1273313987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 msgr2=0x564e79454ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e79454ce0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f14600bc870 tx=0x7f14600057d0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 -- 172.21.15.116:0/1273313987 shutdown_connections 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e79454ce0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e794523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 --2- 172.21.15.116:0/1273313987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e794497d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 -- 172.21.15.116:0/1273313987 >> 172.21.15.116:0/1273313987 conn(0x564e79443b20 msgr2=0x564e79443f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 -- 172.21.15.116:0/1273313987 shutdown_connections 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 -- 172.21.15.116:0/1273313987 wait complete. 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 -- start start 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.461+0000 7f1479dce740 1 Processor -- start 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7944f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e7944fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e794503d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e79083430 con 0x564e794528f0 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564e7903fe00 con 0x564e79449d10 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564e79335590 con 0x564e794493f0 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 10 client.0 _ll_register_callbacks cb 0x564e793a4fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e794503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e794503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44160/0 (socket says 172.21.15.116:44160) 2024-03-24T09:24:21.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 -- 172.21.15.116:0/1177741379 learned_addr learned my addr 172.21.15.116:0/1177741379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f146ffff640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e7944fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1474f90640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7944f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 -- 172.21.15.116:0/1177741379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 msgr2=0x564e7944f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7944f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 -- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 msgr2=0x564e7944fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e7944fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 -- 172.21.15.116:0/1177741379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14701a4800 con 0x564e794528f0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1475791640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e794503d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f14600b8fd0 tx=0x7f14600bb080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52171ea28f5d5d46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f144ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f144ffff640 1 -- 172.21.15.116:0/1177741379 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14600bdea0 con 0x564e794528f0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f144ffff640 1 -- 172.21.15.116:0/1177741379 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f14600d9070 con 0x564e794528f0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564e790af7e0 con 0x564e794528f0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f144ffff640 1 -- 172.21.15.116:0/1177741379 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f14600bc4d0 con 0x564e794528f0 2024-03-24T09:24:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 10 client.16311 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564e793aebd0 con 0x564e794528f0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.462+0000 7f144ffff640 1 -- 172.21.15.116:0/1177741379 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564e790af7e0 con 0x564e794528f0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f144ffff640 1 -- 172.21.15.116:0/1177741379 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f146003c070 con 0x564e794528f0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1479dce740 10 client.16311 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1479dce740 10 client.16311 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1479dce740 20 client.16311 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1479dce740 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794623b0 0x564e794647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1479dce740 4 client.16311 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e790ec720 con 0x564e794623b0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1474f90640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794623b0 0x564e794647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f144ffff640 10 client.16311 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.463+0000 7f1474f90640 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794623b0 0x564e794647a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f4a3328bba0e0d8b server_cookie=d542086a6f685986 in_seq=0 out_seq=0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f144ffff640 1 -- 172.21.15.116:0/1177741379 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e790ec720 con 0x564e794623b0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f144ffff640 10 client.16311 handle_command_reply: tid=0 2024-03-24T09:24:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 client.16311 shutdown 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 msgr2=0x564e794503d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e794503d0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f14600b8fd0 tx=0x7f14600bb080 comp rx=0 tx=0).stop 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 shutdown_connections 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794623b0 0x564e794647a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794528f0 0x564e794503d0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f144ffff640 0 client.16311 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e79449d10 0x564e7944fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 --2- 172.21.15.116:0/1177741379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7944f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 >> 172.21.15.116:0/1177741379 conn(0x564e79443b20 msgr2=0x564e79445870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 shutdown_connections 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 1 -- 172.21.15.116:0/1177741379 wait complete. 2024-03-24T09:24:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.464+0000 7f1479dce740 20 client.16311 trim_cache size 0 max 16384 2024-03-24T09:24:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 Processor -- start 2024-03-24T09:24:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 -- start start 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7945df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 0x564e79469fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e7944cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e790af7e0 con 0x564e79449d10 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564e7945ebe0 con 0x564e794528f0 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1479dce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564e79450920 con 0x564e794493f0 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 0x564e79469fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 0x564e79469fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44164/0 (socket says 172.21.15.116:44164) 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1474f90640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7945df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f1475791640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e7944cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 -- 172.21.15.116:0/2743468050 learned_addr learned my addr 172.21.15.116:0/2743468050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 -- 172.21.15.116:0/2743468050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 msgr2=0x564e7945df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7945df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 -- 172.21.15.116:0/2743468050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 msgr2=0x564e7944cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e7944cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 -- 172.21.15.116:0/2743468050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e7934ac80 con 0x564e79449d10 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.478+0000 7f146ffff640 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 0x564e79469fd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1458013270 tx=0x7f1458005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2daeb33a9a377091 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f14667fc640 1 -- 172.21.15.116:0/2743468050 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458025070 con 0x564e79449d10 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f14667fc640 1 -- 172.21.15.116:0/2743468050 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14580132a0 con 0x564e79449d10 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f14667fc640 1 -- 172.21.15.116:0/2743468050 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1458010430 con 0x564e79449d10 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 -- 172.21.15.116:0/2743468050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 msgr2=0x564e79469fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 0x564e79469fd0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1458013270 tx=0x7f1458005f90 comp rx=0 tx=0).stop 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 -- 172.21.15.116:0/2743468050 shutdown_connections 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e7944cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e79449d10 0x564e79469fd0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 --2- 172.21.15.116:0/2743468050 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e794493f0 0x564e7945df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 -- 172.21.15.116:0/2743468050 >> 172.21.15.116:0/2743468050 conn(0x564e79443b20 msgr2=0x564e79442cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 -- 172.21.15.116:0/2743468050 shutdown_connections 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.479+0000 7f1479dce740 1 -- 172.21.15.116:0/2743468050 wait complete. 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 -- start start 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 Processor -- start 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 0x564e7944cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e79449d10 0x564e7944d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e794c7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e794c7b40 con 0x564e794493f0 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564e794c7c80 con 0x564e794528f0 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564e794c7e40 con 0x564e79449d10 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1479dce740 10 client.0 _ll_register_callbacks cb 0x7f1468009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 0x564e7944cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 0x564e7944cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44180/0 (socket says 172.21.15.116:44180) 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 -- 172.21.15.116:0/1637122087 learned_addr learned my addr 172.21.15.116:0/1637122087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f146ffff640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e79449d10 0x564e7944d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1475791640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e794c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 -- 172.21.15.116:0/1637122087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e79449d10 msgr2=0x564e7944d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e79449d10 0x564e7944d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 -- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 msgr2=0x564e794c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e794c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.480+0000 7f1474f90640 1 -- 172.21.15.116:0/1637122087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1458018040 con 0x564e794493f0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1474f90640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 0x564e7944cd30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x564e79464f10 tx=0x7f1468012e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76bf136e1a2ea9d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f144ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f144ffff640 1 -- 172.21.15.116:0/1637122087 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468016070 con 0x564e794493f0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f144ffff640 1 -- 172.21.15.116:0/1637122087 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1468004b20 con 0x564e794493f0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14600d8020 con 0x564e794493f0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f144ffff640 1 -- 172.21.15.116:0/1637122087 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1468004ce0 con 0x564e794493f0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 10 client.16317 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564e79370920 con 0x564e794493f0 2024-03-24T09:24:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f144ffff640 1 -- 172.21.15.116:0/1637122087 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f14600d8020 con 0x564e794493f0 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f144ffff640 1 -- 172.21.15.116:0/1637122087 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f146801c040 con 0x564e794493f0 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 10 client.16317 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 10 client.16317 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 20 client.16317 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794c96c0 0x564e794cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 4 client.16317 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e7913b890 con 0x564e794c96c0 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.481+0000 7f146ffff640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794c96c0 0x564e794cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.482+0000 7f144ffff640 10 client.16317 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.482+0000 7f146ffff640 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794c96c0 0x564e794cbae0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34825af4e23a62fb server_cookie=9b7c1b35f3ceeaa4 in_seq=0 out_seq=0 2024-03-24T09:24:21.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f144ffff640 1 -- 172.21.15.116:0/1637122087 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e7913b890 con 0x564e794c96c0 2024-03-24T09:24:21.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f144ffff640 10 client.16317 handle_command_reply: tid=0 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 client.16317 shutdown 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 msgr2=0x564e7944cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 0x564e7944cd30 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x564e79464f10 tx=0x7f1468012e40 comp rx=0 tx=0).stop 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 shutdown_connections 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x564e794c96c0 0x564e794cbae0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e794528f0 0x564e794c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564e79449d10 0x564e7944d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f144ffff640 0 client.16317 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 --2- 172.21.15.116:0/1637122087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e794493f0 0x564e7944cd30 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 >> 172.21.15.116:0/1637122087 conn(0x564e79443b20 msgr2=0x564e7944f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 shutdown_connections 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 1 -- 172.21.15.116:0/1637122087 wait complete. 2024-03-24T09:24:21.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.483+0000 7f1479dce740 20 client.16317 trim_cache size 0 max 16384 2024-03-24T09:24:21.490 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:21.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f145c0a22b0 msgr2=0x7f145c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f145c0a22b0 0x7f145c0a4770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1470069420 tx=0x7f14580098b0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 msgr2=0x7f14701a39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 0x7f14701a39d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f1460009430 tx=0x7f1460004040 comp rx=0 tx=0).stop 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 shutdown_connections 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1470106660 0x7f14701a39d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470100f60 0x7f1470198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f145c0a22b0 0x7f145c0a4770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 --2- 172.21.15.116:0/1260261574 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f14701005a0 0x7f1470198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 >> 172.21.15.116:0/1260261574 conn(0x7f1470077710 msgr2=0x7f14700ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 shutdown_connections 2024-03-24T09:24:21.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.484+0000 7f14771f4640 1 -- 172.21.15.116:0/1260261574 wait complete. 2024-03-24T09:24:21.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:24:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:21 smithi143 ceph-mon[35965]: pgmap v70: 129 pgs: 129 active+clean; 111 MiB data, 753 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 201 op/s 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 -- 172.21.15.116:0/2232383155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480106660 msgr2=0x7ff480106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 --2- 172.21.15.116:0/2232383155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480106660 0x7ff480106a40 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff470009900 tx=0x7ff47002bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 -- 172.21.15.116:0/2232383155 shutdown_connections 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 --2- 172.21.15.116:0/2232383155 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff480100f60 0x7ff48010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 --2- 172.21.15.116:0/2232383155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4801005a0 0x7ff480100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 --2- 172.21.15.116:0/2232383155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480106660 0x7ff480106a40 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 -- 172.21.15.116:0/2232383155 >> 172.21.15.116:0/2232383155 conn(0x7ff480077710 msgr2=0x7ff480077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 -- 172.21.15.116:0/2232383155 shutdown_connections 2024-03-24T09:24:21.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.603+0000 7ff484e83640 1 -- 172.21.15.116:0/2232383155 wait complete. 2024-03-24T09:24:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 Processor -- start 2024-03-24T09:24:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 -- start start 2024-03-24T09:24:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4801005a0 0x7ff48019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 0x7ff48019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff480106660 0x7ff4801979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff480198010 con 0x7ff480100f60 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff480198180 con 0x7ff4801005a0 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff484e83640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff480198340 con 0x7ff480106660 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff47dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 0x7ff48019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff47e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4801005a0 0x7ff48019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.604+0000 7ff47ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff480106660 0x7ff4801979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 0x7ff48019dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44210/0 (socket says 172.21.15.116:44210) 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 -- 172.21.15.116:0/3404468442 learned_addr learned my addr 172.21.15.116:0/3404468442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 -- 172.21.15.116:0/3404468442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff480106660 msgr2=0x7ff4801979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff480106660 0x7ff4801979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 -- 172.21.15.116:0/3404468442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4801005a0 msgr2=0x7ff48019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4801005a0 0x7ff48019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 -- 172.21.15.116:0/3404468442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff470009560 con 0x7ff480100f60 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 0x7ff48019dd50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff47400e9f0 tx=0x7ff47400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff46f7fe640 1 -- 172.21.15.116:0/3404468442 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474009700 con 0x7ff480100f60 2024-03-24T09:24:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff46f7fe640 1 -- 172.21.15.116:0/3404468442 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff47400cd80 con 0x7ff480100f60 2024-03-24T09:24:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff480198620 con 0x7ff480100f60 2024-03-24T09:24:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff46f7fe640 1 -- 172.21.15.116:0/3404468442 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474018840 con 0x7ff480100f60 2024-03-24T09:24:21.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.605+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4801a48b0 con 0x7ff480100f60 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff46f7fe640 1 -- 172.21.15.116:0/3404468442 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7ff4740189a0 con 0x7ff480100f60 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 Processor -- start 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 -- start start 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068cdfd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 0x56068ce063b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068ce068f0 0x56068ce08ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56068cd6b840 con 0x56068cdfdd10 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56068cd68d70 con 0x56068cdfd3f0 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff487a5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56068cd64550 con 0x56068ce068f0 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff46f7fe640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff4680a22b0 0x7ff4680a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff47dd9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068ce068f0 0x56068ce08ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff47e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 0x56068ce063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff47e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 0x56068ce063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44220/0 (socket says 172.21.15.116:44220) 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff47e59c640 1 -- 172.21.15.116:0/1350840844 learned_addr learned my addr 172.21.15.116:0/1350840844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff47dd9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068ce068f0 0x56068ce08ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56512/0 (socket says 172.21.15.116:56512) 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.608+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068cdfd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff4680a22b0 0x7ff4680a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff46f7fe640 1 -- 172.21.15.116:0/3404468442 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff474014070 con 0x7ff480100f60 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47e59c640 1 -- 172.21.15.116:0/1350840844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068ce068f0 msgr2=0x56068ce08ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47e59c640 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068ce068f0 0x56068ce08ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47e59c640 1 -- 172.21.15.116:0/1350840844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 msgr2=0x56068cdfd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47e59c640 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068cdfd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47e59c640 1 -- 172.21.15.116:0/1350840844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56068ccf9de0 con 0x56068cdfdd10 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47e59c640 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 0x56068ce063b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff470009900 tx=0x7ff470005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0fdeda49a04a1ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff4680a22b0 0x7ff4680a4770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff478005c40 tx=0x7ff478009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff46cff9640 1 -- 172.21.15.116:0/1350840844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470004840 con 0x56068cdfdd10 2024-03-24T09:24:21.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff46cff9640 1 -- 172.21.15.116:0/1350840844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4700049a0 con 0x56068cdfdd10 2024-03-24T09:24:21.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.609+0000 7ff46cff9640 1 -- 172.21.15.116:0/1350840844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470003850 con 0x56068cdfdd10 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 -- 172.21.15.116:0/1350840844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 msgr2=0x56068ce063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 0x56068ce063b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff470009900 tx=0x7ff470005e20 comp rx=0 tx=0).stop 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 -- 172.21.15.116:0/1350840844 shutdown_connections 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068ce068f0 0x56068ce08ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068cdfdd10 0x56068ce063b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1350840844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068cdfd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 -- 172.21.15.116:0/1350840844 >> 172.21.15.116:0/1350840844 conn(0x56068cdf7b20 msgr2=0x56068cdf7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 -- 172.21.15.116:0/1350840844 shutdown_connections 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 -- 172.21.15.116:0/1350840844 wait complete. 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 -- start start 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.610+0000 7ff487a5a740 1 Processor -- start 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce03920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068ce03e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce043a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56068ca37430 con 0x56068ce068f0 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56068c9f3e00 con 0x56068cdfdd10 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56068cce9590 con 0x56068cdfd3f0 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 10 client.0 _ll_register_callbacks cb 0x56068cd58fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44232/0 (socket says 172.21.15.116:44232) 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 -- 172.21.15.116:0/1928471780 learned_addr learned my addr 172.21.15.116:0/1928471780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068ce03e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47e59c640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce03920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 -- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 msgr2=0x56068ce03920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce03920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 -- 172.21.15.116:0/1928471780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 msgr2=0x56068ce03e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068ce03e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 -- 172.21.15.116:0/1928471780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48010dd20 con 0x56068ce068f0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068ce03e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce043a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff478011e90 tx=0x7ff478004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88676db05b42cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff465ffb640 1 -- 172.21.15.116:0/1928471780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4780047c0 con 0x56068ce068f0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56068ca637e0 con 0x56068ce068f0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.611+0000 7ff465ffb640 1 -- 172.21.15.116:0/1928471780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff478020440 con 0x56068ce068f0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff465ffb640 1 -- 172.21.15.116:0/1928471780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff478020600 con 0x56068ce068f0 2024-03-24T09:24:21.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 10 client.16329 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56068cd62bd0 con 0x56068ce068f0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff465ffb640 1 -- 172.21.15.116:0/1928471780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56068ca637e0 con 0x56068ce068f0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff465ffb640 1 -- 172.21.15.116:0/1928471780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff4780297f0 con 0x56068ce068f0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 10 client.16329 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 10 client.16329 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 20 client.16329 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce16aa0 0x56068ce18ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 4 client.16329 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56068caa0720 con 0x56068ce16aa0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.612+0000 7ff47e59c640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce16aa0 0x56068ce18ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.613+0000 7ff47e59c640 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce16aa0 0x56068ce18ec0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29772f0bf9876b4d server_cookie=b04251dfe3140cbd in_seq=0 out_seq=0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.613+0000 7ff465ffb640 10 client.16329 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.613+0000 7ff465ffb640 1 -- 172.21.15.116:0/1928471780 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56068caa0720 con 0x56068ce16aa0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.613+0000 7ff465ffb640 10 client.16329 handle_command_reply: tid=0 2024-03-24T09:24:21.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.613+0000 7ff487a5a740 1 client.16329 shutdown 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 msgr2=0x56068ce043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce043a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff478011e90 tx=0x7ff478004a40 comp rx=0 tx=0).stop 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 shutdown_connections 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce043a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce16aa0 0x56068ce18ec0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068ce03e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 --2- 172.21.15.116:0/1928471780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce03920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff465ffb640 0 client.16329 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 >> 172.21.15.116:0/1928471780 conn(0x56068cdf7b20 msgr2=0x56068cdf9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 shutdown_connections 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 1 -- 172.21.15.116:0/1928471780 wait complete. 2024-03-24T09:24:21.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.614+0000 7ff487a5a740 20 client.16329 trim_cache size 0 max 16384 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 Processor -- start 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 -- start start 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce12120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068cdf6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce041c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56068cdf3250 con 0x56068ce068f0 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56068cde8e20 con 0x56068cdfdd10 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.627+0000 7ff487a5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56068cdf8b20 con 0x56068cdfd3f0 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce12120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce12120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56518/0 (socket says 172.21.15.116:56518) 2024-03-24T09:24:21.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 -- 172.21.15.116:0/3015310289 learned_addr learned my addr 172.21.15.116:0/3015310289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068cdf6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 -- 172.21.15.116:0/3015310289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 msgr2=0x56068cdf6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068cdf6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 -- 172.21.15.116:0/3015310289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 msgr2=0x56068ce041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 -- 172.21.15.116:0/3015310289 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56068ccfec80 con 0x56068cdfd3f0 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff47e59c640 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce12120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff470009900 tx=0x7ff470041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65af69f6a7ef754d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff467fff640 1 -- 172.21.15.116:0/3015310289 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff470055070 con 0x56068cdfd3f0 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff467fff640 1 -- 172.21.15.116:0/3015310289 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff470004740 con 0x56068cdfd3f0 2024-03-24T09:24:21.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.628+0000 7ff467fff640 1 -- 172.21.15.116:0/3015310289 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4700512c0 con 0x56068cdfd3f0 2024-03-24T09:24:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 -- 172.21.15.116:0/3015310289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 msgr2=0x56068ce12120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce12120 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff470009900 tx=0x7ff470041040 comp rx=0 tx=0).stop 2024-03-24T09:24:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 -- 172.21.15.116:0/3015310289 shutdown_connections 2024-03-24T09:24:21.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfdd10 0x56068cdf6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 --2- 172.21.15.116:0/3015310289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfd3f0 0x56068ce12120 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 -- 172.21.15.116:0/3015310289 >> 172.21.15.116:0/3015310289 conn(0x56068cdf7b20 msgr2=0x56068cdf7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 -- 172.21.15.116:0/3015310289 shutdown_connections 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.629+0000 7ff487a5a740 1 -- 172.21.15.116:0/3015310289 wait complete. 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 -- start start 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 Processor -- start 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068ce0ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfdd10 0x56068ce09e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce0a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56068ce0aa90 con 0x56068ce068f0 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56068ce0ac00 con 0x56068cdfd3f0 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56068ce7be00 con 0x56068cdfdd10 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff487a5a740 10 client.0 _ll_register_callbacks cb 0x7ff478011d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff47ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce0a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff47ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce0a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44244/0 (socket says 172.21.15.116:44244) 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff47e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068ce0ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff47ed9d640 1 -- 172.21.15.116:0/2466613684 learned_addr learned my addr 172.21.15.116:0/2466613684 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.630+0000 7ff47dd9b640 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfdd10 0x56068ce09e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff47ed9d640 1 -- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfdd10 msgr2=0x56068ce09e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfdd10 0x56068ce09e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff47ed9d640 1 -- 172.21.15.116:0/2466613684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 msgr2=0x56068ce0ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068ce0ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff47ed9d640 1 -- 172.21.15.116:0/2466613684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56068cd62bd0 con 0x56068ce068f0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff47ed9d640 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce0a390 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff478011e60 tx=0x7ff4780049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b11f13f21a4af48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff465ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff465ffb640 1 -- 172.21.15.116:0/2466613684 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff478004400 con 0x56068ce068f0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff465ffb640 1 -- 172.21.15.116:0/2466613684 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff478029650 con 0x56068ce068f0 2024-03-24T09:24:21.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4700049a0 con 0x56068ce068f0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff465ffb640 1 -- 172.21.15.116:0/2466613684 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff478029810 con 0x56068ce068f0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 10 client.16335 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56068ccf9de0 con 0x56068ce068f0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff465ffb640 1 -- 172.21.15.116:0/2466613684 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff4700049a0 con 0x56068ce068f0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff465ffb640 1 -- 172.21.15.116:0/2466613684 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff47803f070 con 0x56068ce068f0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 10 client.16335 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 10 client.16335 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 20 client.16335 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce819a0 0x56068ce83dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 4 client.16335 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.631+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56068caef890 con 0x56068ce819a0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff47e59c640 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce819a0 0x56068ce83dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff465ffb640 10 client.16335 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff47e59c640 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce819a0 0x56068ce83dc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=df62a5b60d53bfe2 server_cookie=6674484b1e9e405 in_seq=0 out_seq=0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff465ffb640 1 -- 172.21.15.116:0/2466613684 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56068caef890 con 0x56068ce819a0 2024-03-24T09:24:21.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff465ffb640 10 client.16335 handle_command_reply: tid=0 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 client.16335 shutdown 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 msgr2=0x56068ce0a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce0a390 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff478011e60 tx=0x7ff4780049d0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 shutdown_connections 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56068ce068f0 0x56068ce0a390 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56068cdfdd10 0x56068ce09e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56068ce819a0 0x56068ce83dc0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff465ffb640 0 client.16335 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 --2- 172.21.15.116:0/2466613684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56068cdfd3f0 0x56068ce0ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 >> 172.21.15.116:0/2466613684 conn(0x56068cdf7b20 msgr2=0x56068cdf6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 shutdown_connections 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.632+0000 7ff487a5a740 1 -- 172.21.15.116:0/2466613684 wait complete. 2024-03-24T09:24:21.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff487a5a740 20 client.16335 trim_cache size 0 max 16384 2024-03-24T09:24:21.639 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:21.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff4680a22b0 msgr2=0x7ff4680a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff4680a22b0 0x7ff4680a4770 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff478005c40 tx=0x7ff478009500 comp rx=0 tx=0).stop 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 msgr2=0x7ff48019dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 0x7ff48019dd50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff47400e9f0 tx=0x7ff47400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 shutdown_connections 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff4680a22b0 0x7ff4680a4770 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff480106660 0x7ff4801979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff480100f60 0x7ff48019dd50 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 --2- 172.21.15.116:0/3404468442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4801005a0 0x7ff48019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 >> 172.21.15.116:0/3404468442 conn(0x7ff480077710 msgr2=0x7ff480104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 shutdown_connections 2024-03-24T09:24:21.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.633+0000 7ff484e83640 1 -- 172.21.15.116:0/3404468442 wait complete. 2024-03-24T09:24:21.644 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.751+0000 7fa044b6c640 1 -- 172.21.15.116:0/3569533335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa040100f60 msgr2=0x7fa04010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.751+0000 7fa044b6c640 1 --2- 172.21.15.116:0/3569533335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa040100f60 0x7fa04010d420 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fa038009900 tx=0x7fa03802bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.751+0000 7fa044b6c640 1 -- 172.21.15.116:0/3569533335 shutdown_connections 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.751+0000 7fa044b6c640 1 --2- 172.21.15.116:0/3569533335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa040100f60 0x7fa04010d420 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.751+0000 7fa044b6c640 1 --2- 172.21.15.116:0/3569533335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0401005a0 0x7fa040100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 --2- 172.21.15.116:0/3569533335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa040106660 0x7fa040106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- 172.21.15.116:0/3569533335 >> 172.21.15.116:0/3569533335 conn(0x7fa040077710 msgr2=0x7fa040077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- 172.21.15.116:0/3569533335 shutdown_connections 2024-03-24T09:24:21.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- 172.21.15.116:0/3569533335 wait complete. 2024-03-24T09:24:21.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 Processor -- start 2024-03-24T09:24:21.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- start start 2024-03-24T09:24:21.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 0x7fa040198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa040100f60 0x7fa040198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa040106660 0x7fa0401a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa040199730 con 0x7fa0401005a0 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401a3f50 con 0x7fa040106660 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.752+0000 7fa044b6c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa0401a4090 con 0x7fa040100f60 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 0x7fa040198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 0x7fa040198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44260/0 (socket says 172.21.15.116:44260) 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 -- 172.21.15.116:0/2674378086 learned_addr learned my addr 172.21.15.116:0/2674378086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa040100f60 0x7fa040198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03ed9d640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa040106660 0x7fa0401a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 -- 172.21.15.116:0/2674378086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa040100f60 msgr2=0x7fa040198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa040100f60 0x7fa040198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 -- 172.21.15.116:0/2674378086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa040106660 msgr2=0x7fa0401a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa040106660 0x7fa0401a3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 -- 172.21.15.116:0/2674378086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa038009560 con 0x7fa0401005a0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa03e59c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 0x7fa040198990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa03000ebc0 tx=0x7fa03000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa02f7fe640 1 -- 172.21.15.116:0/2674378086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300099b0 con 0x7fa0401005a0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0401967e0 con 0x7fa0401005a0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa02f7fe640 1 -- 172.21.15.116:0/2674378086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa03000eec0 con 0x7fa0401005a0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa02f7fe640 1 -- 172.21.15.116:0/2674378086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030018a80 con 0x7fa0401005a0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.753+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa040196d40 con 0x7fa0401005a0 2024-03-24T09:24:21.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.756+0000 7fa02f7fe640 1 -- 172.21.15.116:0/2674378086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fa030018be0 con 0x7fa0401005a0 2024-03-24T09:24:21.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.756+0000 7fa047829740 1 Processor -- start 2024-03-24T09:24:21.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 -- start start 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7c3f0 0x560845a7c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a87ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5608459ea840 con 0x560845a7cd10 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5608459e7d70 con 0x560845a858f0 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa047829740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5608459e3550 con 0x560845a7c3f0 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a87ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7c3f0 0x560845a7c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a853b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44270/0 (socket says 172.21.15.116:44270) 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7c3f0 0x560845a7c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56540/0 (socket says 172.21.15.116:56540) 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a87ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34830/0 (socket says 172.21.15.116:34830) 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 -- 172.21.15.116:0/141055844 learned_addr learned my addr 172.21.15.116:0/141055844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa02f7fe640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0280a2230 0x7fa0280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03ed9d640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0280a2230 0x7fa0280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 -- 172.21.15.116:0/141055844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7c3f0 msgr2=0x560845a7c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7c3f0 0x560845a7c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 -- 172.21.15.116:0/141055844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 msgr2=0x560845a87ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a87ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa02f7fe640 1 -- 172.21.15.116:0/2674378086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa030014070 con 0x7fa0401005a0 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 -- 172.21.15.116:0/141055844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560845978de0 con 0x560845a7cd10 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a87ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.757+0000 7fa03e59c640 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a853b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa0300cbcc0 tx=0x7fa030000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3600429fbe57753 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa02cff9640 1 -- 172.21.15.116:0/141055844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03009b0a0 con 0x560845a7cd10 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa03ed9d640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0280a2230 0x7fa0280a46f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa038005ab0 tx=0x7fa03802bc30 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa02cff9640 1 -- 172.21.15.116:0/141055844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa03009f140 con 0x560845a7cd10 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa02cff9640 1 -- 172.21.15.116:0/141055844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300cc170 con 0x560845a7cd10 2024-03-24T09:24:21.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa047829740 1 -- 172.21.15.116:0/141055844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 msgr2=0x560845a853b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa047829740 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a853b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fa0300cbcc0 tx=0x7fa030000f40 comp rx=0 tx=0).stop 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa047829740 1 -- 172.21.15.116:0/141055844 shutdown_connections 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa047829740 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a87ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa047829740 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a853b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.758+0000 7fa047829740 1 --2- 172.21.15.116:0/141055844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7c3f0 0x560845a7c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- 172.21.15.116:0/141055844 >> 172.21.15.116:0/141055844 conn(0x560845a76b20 msgr2=0x560845a76f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- 172.21.15.116:0/141055844 shutdown_connections 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- 172.21.15.116:0/141055844 wait complete. 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- start start 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 Processor -- start 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a82f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a83440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5608456b6430 con 0x560845a7cd10 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560845672e00 con 0x560845a7c3f0 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560845968590 con 0x560845a858f0 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa047829740 10 client.0 _ll_register_callbacks cb 0x5608459d7fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a82f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa03ed9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a83440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a82f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44278/0 (socket says 172.21.15.116:44278) 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.759+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1401586225 learned_addr learned my addr 172.21.15.116:0/1401586225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 msgr2=0x560845a83440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a83440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1401586225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 msgr2=0x560845a829c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a829c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1401586225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa040196d40 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a82f00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa0340126e0 tx=0x7fa034012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8dfadcc86bb4f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa025ffb640 1 -- 172.21.15.116:0/1401586225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034017070 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa025ffb640 1 -- 172.21.15.116:0/1401586225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa034004050 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5608456e27e0 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa025ffb640 1 -- 172.21.15.116:0/1401586225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa034004210 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 10 client.16347 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5608459e1bd0 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa025ffb640 1 -- 172.21.15.116:0/1401586225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5608456e27e0 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa025ffb640 1 -- 172.21.15.116:0/1401586225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa0340039a0 con 0x560845a7cd10 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 10 client.16347 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 10 client.16347 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 20 client.16347 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845a95a90 0x560845a97ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 4 client.16347 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.760+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56084571f720 con 0x560845a95a90 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.761+0000 7fa03e59c640 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845a95a90 0x560845a97ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.761+0000 7fa025ffb640 10 client.16347 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.761+0000 7fa03e59c640 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845a95a90 0x560845a97ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aeed0becdda27a21 server_cookie=b13551bb823f55a9 in_seq=0 out_seq=0 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa025ffb640 1 -- 172.21.15.116:0/1401586225 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56084571f720 con 0x560845a95a90 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa025ffb640 10 client.16347 handle_command_reply: tid=0 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 client.16347 shutdown 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 msgr2=0x560845a82f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a82f00 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa0340126e0 tx=0x7fa034012ba0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 shutdown_connections 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845a95a90 0x560845a97ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a83440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a82f00 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa025ffb640 0 client.16347 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 --2- 172.21.15.116:0/1401586225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a829c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 >> 172.21.15.116:0/1401586225 conn(0x560845a76b20 msgr2=0x560845a78870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 shutdown_connections 2024-03-24T09:24:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 1 -- 172.21.15.116:0/1401586225 wait complete. 2024-03-24T09:24:21.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.762+0000 7fa047829740 20 client.16347 trim_cache size 0 max 16384 2024-03-24T09:24:21.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.776+0000 7fa047829740 1 Processor -- start 2024-03-24T09:24:21.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.776+0000 7fa047829740 1 -- start start 2024-03-24T09:24:21.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa047829740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 0x560845a91120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa047829740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7cd10 0x560845a83260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa047829740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a75b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa047829740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560845a67e20 con 0x560845a7c3f0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa047829740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560845a77b20 con 0x560845a858f0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa047829740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560845a78e30 con 0x560845a7cd10 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 0x560845a91120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 0x560845a91120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44288/0 (socket says 172.21.15.116:44288) 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 -- 172.21.15.116:0/3990946693 learned_addr learned my addr 172.21.15.116:0/3990946693 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7cd10 0x560845a83260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03ed9d640 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a75b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 -- 172.21.15.116:0/3990946693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7cd10 msgr2=0x560845a83260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7cd10 0x560845a83260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 -- 172.21.15.116:0/3990946693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 msgr2=0x560845a75b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a75b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 -- 172.21.15.116:0/3990946693 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56084597dc80 con 0x560845a7c3f0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa03e59c640 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 0x560845a91120 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa0300cbe00 tx=0x7fa03009ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4fc6f83fe3d0d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa027fff640 1 -- 172.21.15.116:0/3990946693 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300ceab0 con 0x560845a7c3f0 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa027fff640 1 -- 172.21.15.116:0/3990946693 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa030018e90 con 0x560845a7c3f0 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.777+0000 7fa027fff640 1 -- 172.21.15.116:0/3990946693 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0300564a0 con 0x560845a7c3f0 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 -- 172.21.15.116:0/3990946693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 msgr2=0x560845a91120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 0x560845a91120 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa0300cbe00 tx=0x7fa03009ad10 comp rx=0 tx=0).stop 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 -- 172.21.15.116:0/3990946693 shutdown_connections 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a858f0 0x560845a75b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a7cd10 0x560845a83260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 --2- 172.21.15.116:0/3990946693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7c3f0 0x560845a91120 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 -- 172.21.15.116:0/3990946693 >> 172.21.15.116:0/3990946693 conn(0x560845a76b20 msgr2=0x560845a75430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 -- 172.21.15.116:0/3990946693 shutdown_connections 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 -- 172.21.15.116:0/3990946693 wait complete. 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 -- start start 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.778+0000 7fa047829740 1 Processor -- start 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a88fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a894f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a89a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560845a8df30 con 0x560845a7cd10 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560845afae10 con 0x560845a7c3f0 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560845afaf50 con 0x560845a858f0 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 10 client.0 _ll_register_callbacks cb 0x7fa034009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a894f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44298/0 (socket says 172.21.15.116:44298) 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1584375225 learned_addr learned my addr 172.21.15.116:0/1584375225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03e59c640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a88fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03ed9d640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a89a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 msgr2=0x560845a89a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a89a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1584375225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 msgr2=0x560845a88fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a88fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 -- 172.21.15.116:0/1584375225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560845978de0 con 0x560845a7cd10 2024-03-24T09:24:21.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03ed9d640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a89a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa03dd9b640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a894f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x560845a82220 tx=0x7fa034005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66691efde5405f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa025ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa025ffb640 1 -- 172.21.15.116:0/1584375225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa034017070 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa025ffb640 1 -- 172.21.15.116:0/1584375225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa034003900 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa03009f140 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa025ffb640 1 -- 172.21.15.116:0/1584375225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa034003ac0 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 10 client.16353 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.779+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5608459a3920 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa025ffb640 1 -- 172.21.15.116:0/1584375225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa03009f140 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa025ffb640 1 -- 172.21.15.116:0/1584375225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa034032440 con 0x560845a7cd10 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa047829740 10 client.16353 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa047829740 10 client.16353 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa047829740 20 client.16353 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa047829740 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845afc750 0x560845afeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa047829740 4 client.16353 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56084576e890 con 0x560845afc750 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa03e59c640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845afc750 0x560845afeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa025ffb640 10 client.16353 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.780+0000 7fa03e59c640 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845afc750 0x560845afeb70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a33e2a17ff8734d server_cookie=4ee9bc286ada74c0 in_seq=0 out_seq=0 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa025ffb640 1 -- 172.21.15.116:0/1584375225 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56084576e890 con 0x560845afc750 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa025ffb640 10 client.16353 handle_command_reply: tid=0 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 client.16353 shutdown 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 msgr2=0x560845a894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a894f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x560845a82220 tx=0x7fa034005ab0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 shutdown_connections 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x560845afc750 0x560845afeb70 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560845a858f0 0x560845a89a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa025ffb640 0 client.16353 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560845a7cd10 0x560845a894f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 --2- 172.21.15.116:0/1584375225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560845a7c3f0 0x560845a88fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 >> 172.21.15.116:0/1584375225 conn(0x560845a76b20 msgr2=0x560845a75cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 shutdown_connections 2024-03-24T09:24:21.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 1 -- 172.21.15.116:0/1584375225 wait complete. 2024-03-24T09:24:21.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa047829740 20 client.16353 trim_cache size 0 max 16384 2024-03-24T09:24:21.788 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0280a2230 msgr2=0x7fa0280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa044b6c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0280a2230 0x7fa0280a46f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa038005ab0 tx=0x7fa03802bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 msgr2=0x7fa040198990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.781+0000 7fa044b6c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 0x7fa040198990 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa03000ebc0 tx=0x7fa03000c6a0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 shutdown_connections 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa040106660 0x7fa0401a3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0280a2230 0x7fa0280a46f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa040100f60 0x7fa040198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 --2- 172.21.15.116:0/2674378086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0401005a0 0x7fa040198990 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 >> 172.21.15.116:0/2674378086 conn(0x7fa040077710 msgr2=0x7fa0400ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 shutdown_connections 2024-03-24T09:24:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.782+0000 7fa044b6c640 1 -- 172.21.15.116:0/2674378086 wait complete. 2024-03-24T09:24:21.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 -- 172.21.15.116:0/2487064116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 msgr2=0x7fe0a0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 --2- 172.21.15.116:0/2487064116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0106a40 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009900 tx=0x7fe09c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 -- 172.21.15.116:0/2487064116 shutdown_connections 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 --2- 172.21.15.116:0/2487064116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a0100f60 0x7fe0a010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 --2- 172.21.15.116:0/2487064116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 0x7fe0a0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 --2- 172.21.15.116:0/2487064116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0106a40 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 -- 172.21.15.116:0/2487064116 >> 172.21.15.116:0/2487064116 conn(0x7fe0a0077710 msgr2=0x7fe0a0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 -- 172.21.15.116:0/2487064116 shutdown_connections 2024-03-24T09:24:21.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.900+0000 7fe0a8084640 1 -- 172.21.15.116:0/2487064116 wait complete. 2024-03-24T09:24:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 Processor -- start 2024-03-24T09:24:21.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 -- start start 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 0x7fe0a019d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a0100f60 0x7fe0a019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a0198080 con 0x7fe0a0106660 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01981f0 con 0x7fe0a01005a0 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a8084640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01983b0 con 0x7fe0a0100f60 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 0x7fe0a019d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44310/0 (socket says 172.21.15.116:44310) 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 0x7fe0a019d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34896/0 (socket says 172.21.15.116:34896) 2024-03-24T09:24:21.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 -- 172.21.15.116:0/287402626 learned_addr learned my addr 172.21.15.116:0/287402626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a561f640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a0100f60 0x7fe0a019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 -- 172.21.15.116:0/287402626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a0100f60 msgr2=0x7fe0a019dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a0100f60 0x7fe0a019dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 -- 172.21.15.116:0/287402626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 msgr2=0x7fe0a019d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 0x7fe0a019d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.901+0000 7fe0a6621640 1 -- 172.21.15.116:0/287402626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09c009560 con 0x7fe0a0106660 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.902+0000 7fe0a6621640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0197a30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe09800eb10 tx=0x7fe09800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.902+0000 7fe096ffd640 1 -- 172.21.15.116:0/287402626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098004b20 con 0x7fe0a0106660 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.902+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a0198690 con 0x7fe0a0106660 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.902+0000 7fe096ffd640 1 -- 172.21.15.116:0/287402626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe098004c80 con 0x7fe0a0106660 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.902+0000 7fe096ffd640 1 -- 172.21.15.116:0/287402626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098010690 con 0x7fe0a0106660 2024-03-24T09:24:21.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.902+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a0102320 con 0x7fe0a0106660 2024-03-24T09:24:21.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.904+0000 7fe096ffd640 1 -- 172.21.15.116:0/287402626 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe098005020 con 0x7fe0a0106660 2024-03-24T09:24:21.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.904+0000 7fe0aac66740 1 Processor -- start 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.904+0000 7fe0aac66740 1 -- start start 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.904+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.904+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 0x55fddd8d93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.904+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d98f0 0x55fddd8dbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0aac66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fddd83e840 con 0x55fddd8d0d10 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe096ffd640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe08c0a2230 0x7fe08c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0aac66740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fddd83bd70 con 0x55fddd8d98f0 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0aac66740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fddd837550 con 0x55fddd8d03f0 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 0x55fddd8d93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a6621640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe08c0a2230 0x7fe08c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 0x55fddd8d93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44316/0 (socket says 172.21.15.116:44316) 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 -- 172.21.15.116:0/1157973217 learned_addr learned my addr 172.21.15.116:0/1157973217 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a561f640 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d98f0 0x55fddd8dbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a561f640 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe096ffd640 1 -- 172.21.15.116:0/287402626 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe098014070 con 0x7fe0a0106660 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 -- 172.21.15.116:0/1157973217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 msgr2=0x55fddd8d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 -- 172.21.15.116:0/1157973217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d98f0 msgr2=0x55fddd8dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d98f0 0x55fddd8dbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 -- 172.21.15.116:0/1157973217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fddd7ccde0 con 0x55fddd8d0d10 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a561f640 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d98f0 0x55fddd8dbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a6621640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe08c0a2230 0x7fe08c0a46f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe09809ad50 tx=0x7fe098009200 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 0x55fddd8d93b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009230 tx=0x7fe09c005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4830f247ed5f02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe08bfff640 1 -- 172.21.15.116:0/1157973217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c004880 con 0x55fddd8d0d10 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe08bfff640 1 -- 172.21.15.116:0/1157973217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe09c0049e0 con 0x55fddd8d0d10 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.905+0000 7fe08bfff640 1 -- 172.21.15.116:0/1157973217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c003850 con 0x55fddd8d0d10 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 -- 172.21.15.116:0/1157973217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 msgr2=0x55fddd8d93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 0x55fddd8d93b0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009230 tx=0x7fe09c005df0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 -- 172.21.15.116:0/1157973217 shutdown_connections 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d98f0 0x55fddd8dbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d0d10 0x55fddd8d93b0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 --2- 172.21.15.116:0/1157973217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 -- 172.21.15.116:0/1157973217 >> 172.21.15.116:0/1157973217 conn(0x55fddd8cab20 msgr2=0x55fddd8caf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 -- 172.21.15.116:0/1157973217 shutdown_connections 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.906+0000 7fe0aac66740 1 -- 172.21.15.116:0/1157973217 wait complete. 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 -- start start 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 Processor -- start 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8d6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8d73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fddd50a430 con 0x55fddd8d98f0 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fddd4c6e00 con 0x55fddd8d0d10 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fddd7bc590 con 0x55fddd8d03f0 2024-03-24T09:24:21.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0aac66740 10 client.0 _ll_register_callbacks cb 0x55fddd82bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8d73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8d6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8d73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44318/0 (socket says 172.21.15.116:44318) 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 -- 172.21.15.116:0/2461079875 learned_addr learned my addr 172.21.15.116:0/2461079875 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 -- 172.21.15.116:0/2461079875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 msgr2=0x55fddd8d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 -- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 msgr2=0x55fddd8d6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.907+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8d6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0a561f640 1 -- 172.21.15.116:0/2461079875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a010dd20 con 0x55fddd8d98f0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8d73b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe090022880 tx=0x7fe0900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46bb7e8a642df25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0897fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0897fa640 1 -- 172.21.15.116:0/2461079875 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090015070 con 0x55fddd8d98f0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0897fa640 1 -- 172.21.15.116:0/2461079875 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe090004250 con 0x55fddd8d98f0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fddd5367e0 con 0x55fddd8d98f0 2024-03-24T09:24:21.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0897fa640 1 -- 172.21.15.116:0/2461079875 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe090004410 con 0x55fddd8d98f0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 10 client.16365 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fddd835bd0 con 0x55fddd8d98f0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0897fa640 1 -- 172.21.15.116:0/2461079875 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fddd5367e0 con 0x55fddd8d98f0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0897fa640 1 -- 172.21.15.116:0/2461079875 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fe0900037b0 con 0x55fddd8d98f0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 10 client.16365 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 10 client.16365 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 20 client.16365 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd8e9aa0 0x55fddd8ebec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 4 client.16365 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.908+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fddd573720 con 0x55fddd8e9aa0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.909+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd8e9aa0 0x55fddd8ebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.909+0000 7fe0897fa640 10 client.16365 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.909+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd8e9aa0 0x55fddd8ebec0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30d55feecdc5bd7a server_cookie=7d27ba5481d6e28c in_seq=0 out_seq=0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0897fa640 1 -- 172.21.15.116:0/2461079875 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fddd573720 con 0x55fddd8e9aa0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0897fa640 10 client.16365 handle_command_reply: tid=0 2024-03-24T09:24:21.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 client.16365 shutdown 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 msgr2=0x55fddd8d73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8d73b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe090022880 tx=0x7fe0900046c0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 shutdown_connections 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8d73b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd8e9aa0 0x55fddd8ebec0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8d6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2461079875 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8d6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0897fa640 0 client.16365 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 >> 172.21.15.116:0/2461079875 conn(0x55fddd8cab20 msgr2=0x55fddd8cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 shutdown_connections 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 1 -- 172.21.15.116:0/2461079875 wait complete. 2024-03-24T09:24:21.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.911+0000 7fe0aac66740 20 client.16365 trim_cache size 0 max 16384 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.925+0000 7fe0aac66740 1 Processor -- start 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.925+0000 7fe0aac66740 1 -- start start 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8c9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8e5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8f1770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0aac66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fddd8c6240 con 0x55fddd8d98f0 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0aac66740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fddd8e6460 con 0x55fddd8d0d10 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0aac66740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fddd8c63e0 con 0x55fddd8d03f0 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8c9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8c9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56628/0 (socket says 172.21.15.116:56628) 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 -- 172.21.15.116:0/684215335 learned_addr learned my addr 172.21.15.116:0/684215335 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 -- 172.21.15.116:0/684215335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 msgr2=0x55fddd8e5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8e5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 -- 172.21.15.116:0/684215335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 msgr2=0x55fddd8f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 -- 172.21.15.116:0/684215335 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fddd8ca3c0 con 0x55fddd8d03f0 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe0a561f640 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8c9cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe090009f80 tx=0x7fe090000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=350a2db40a550bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.926+0000 7fe08b7fe640 1 -- 172.21.15.116:0/684215335 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090015040 con 0x55fddd8d03f0 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe08b7fe640 1 -- 172.21.15.116:0/684215335 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe09003d070 con 0x55fddd8d03f0 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe08b7fe640 1 -- 172.21.15.116:0/684215335 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090004990 con 0x55fddd8d03f0 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 -- 172.21.15.116:0/684215335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 msgr2=0x55fddd8c9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8c9cc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe090009f80 tx=0x7fe090000bc0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 -- 172.21.15.116:0/684215335 shutdown_connections 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8e5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 --2- 172.21.15.116:0/684215335 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8c9cc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 -- 172.21.15.116:0/684215335 >> 172.21.15.116:0/684215335 conn(0x55fddd8cab20 msgr2=0x55fddd8d68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 -- 172.21.15.116:0/684215335 shutdown_connections 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.927+0000 7fe0aac66740 1 -- 172.21.15.116:0/684215335 wait complete. 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 -- start start 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 Processor -- start 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8e1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8ddca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8de1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fddd8de7f0 con 0x55fddd8d98f0 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fddd8de960 con 0x55fddd8d0d10 2024-03-24T09:24:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fddd8deb20 con 0x55fddd8d03f0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0aac66740 10 client.0 _ll_register_callbacks cb 0x7fe0900224e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8de1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8ddca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a561f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8e1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8ddca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34934/0 (socket says 172.21.15.116:34934) 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 -- 172.21.15.116:0/2207799514 learned_addr learned my addr 172.21.15.116:0/2207799514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 -- 172.21.15.116:0/2207799514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 msgr2=0x55fddd8e1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8e1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 -- 172.21.15.116:0/2207799514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 msgr2=0x55fddd8de1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8de1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a5e20640 1 -- 172.21.15.116:0/2207799514 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fddd835bd0 con 0x55fddd8d0d10 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.928+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8de1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0a5e20640 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8ddca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009230 tx=0x7fe09c02c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6714f61585fdd4c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0897fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:24:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0897fa640 1 -- 172.21.15.116:0/2207799514 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c04f040 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe09c0049e0 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0897fa640 1 -- 172.21.15.116:0/2207799514 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe09c052ca0 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0897fa640 1 -- 172.21.15.116:0/2207799514 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe09c052e60 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 10 client.26201 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fddd7d1c80 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0897fa640 1 -- 172.21.15.116:0/2207799514 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe09c0049e0 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0897fa640 1 -- 172.21.15.116:0/2207799514 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fe09c052050 con 0x55fddd8d0d10 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 10 client.26201 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 10 client.26201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 20 client.26201 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd951260 0x55fddd953680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 4 client.26201 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.929+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fddd5c2890 con 0x55fddd951260 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd951260 0x55fddd953680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0897fa640 10 client.26201 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0a561f640 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd951260 0x55fddd953680 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e4ccfa9f08cd96 server_cookie=7aa1cb4b3dce2c5c in_seq=0 out_seq=0 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0897fa640 1 -- 172.21.15.116:0/2207799514 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fddd5c2890 con 0x55fddd951260 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0897fa640 10 client.26201 handle_command_reply: tid=0 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 client.26201 shutdown 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 msgr2=0x55fddd8ddca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8ddca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009230 tx=0x7fe09c02c7a0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 shutdown_connections 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fddd8d98f0 0x55fddd8de1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55fddd951260 0x55fddd953680 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fddd8d0d10 0x55fddd8ddca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 --2- 172.21.15.116:0/2207799514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fddd8d03f0 0x55fddd8e1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0897fa640 0 client.26201 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 >> 172.21.15.116:0/2207799514 conn(0x55fddd8cab20 msgr2=0x55fddd8c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 shutdown_connections 2024-03-24T09:24:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.930+0000 7fe0aac66740 1 -- 172.21.15.116:0/2207799514 wait complete. 2024-03-24T09:24:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0aac66740 20 client.26201 trim_cache size 0 max 16384 2024-03-24T09:24:21.938 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe08c0a2230 msgr2=0x7fe08c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe08c0a2230 0x7fe08c0a46f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe09809ad50 tx=0x7fe098009200 comp rx=0 tx=0).stop 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 msgr2=0x7fe0a0197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0197a30 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe09800eb10 tx=0x7fe09800c2d0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 shutdown_connections 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0a0106660 0x7fe0a0197a30 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe08c0a2230 0x7fe08c0a46f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0a0100f60 0x7fe0a019dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 --2- 172.21.15.116:0/287402626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0a01005a0 0x7fe0a019d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 >> 172.21.15.116:0/287402626 conn(0x7fe0a0077710 msgr2=0x7fe0a00ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 shutdown_connections 2024-03-24T09:24:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:21.931+0000 7fe0a8084640 1 -- 172.21.15.116:0/287402626 wait complete. 2024-03-24T09:24:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:22 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/635846811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:24:22.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:22 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/635846811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:24:22.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:22 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/635846811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:24:22.942 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:24:22.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:24:22.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 -- 172.21.15.116:0/654612223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc0ffc10 msgr2=0x7f68fc100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 --2- 172.21.15.116:0/654612223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc0ffc10 0x7f68fc100070 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009930 tx=0x7f68f002bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 -- 172.21.15.116:0/654612223 shutdown_connections 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 --2- 172.21.15.116:0/654612223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68fc1005b0 0x7f68fc10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 --2- 172.21.15.116:0/654612223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc0ffc10 0x7f68fc100070 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 --2- 172.21.15.116:0/654612223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc1076d0 0x7f68fc107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 -- 172.21.15.116:0/654612223 >> 172.21.15.116:0/654612223 conn(0x7f68fc076bd0 msgr2=0x7f68fc0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 -- 172.21.15.116:0/654612223 shutdown_connections 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.058+0000 7f690135c640 1 -- 172.21.15.116:0/654612223 wait complete. 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 Processor -- start 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 -- start start 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc0ffc10 0x7f68fc19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 0x7f68fc19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68fc1076d0 0x7f68fc197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc1982b0 con 0x7f68fc1005b0 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc198420 con 0x7f68fc0ffc10 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f690135c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc1985e0 con 0x7f68fc1076d0 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 0x7f68fc19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 0x7f68fc19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44352/0 (socket says 172.21.15.116:44352) 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f68fa7fc640 1 -- 172.21.15.116:0/250544175 learned_addr learned my addr 172.21.15.116:0/250544175 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f68faffd640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc0ffc10 0x7f68fc19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.059+0000 7f68fb7fe640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68fc1076d0 0x7f68fc197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68fa7fc640 1 -- 172.21.15.116:0/250544175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68fc1076d0 msgr2=0x7f68fc197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68fc1076d0 0x7f68fc197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68fa7fc640 1 -- 172.21.15.116:0/250544175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc0ffc10 msgr2=0x7f68fc19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc0ffc10 0x7f68fc19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68fa7fc640 1 -- 172.21.15.116:0/250544175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f0009560 con 0x7f68fc1005b0 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68faffd640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc0ffc10 0x7f68fc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 0x7f68fc19dfa0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009230 tx=0x7f68f0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68e3fff640 1 -- 172.21.15.116:0/250544175 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0004440 con 0x7f68fc1005b0 2024-03-24T09:24:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68fc198860 con 0x7f68fc1005b0 2024-03-24T09:24:23.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68e3fff640 1 -- 172.21.15.116:0/250544175 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68f002de90 con 0x7f68fc1005b0 2024-03-24T09:24:23.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f68e3fff640 1 -- 172.21.15.116:0/250544175 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00052e0 con 0x7f68fc1005b0 2024-03-24T09:24:23.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.060+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68fc1a49e0 con 0x7f68fc1005b0 2024-03-24T09:24:23.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.062+0000 7f68e3fff640 1 -- 172.21.15.116:0/250544175 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f68f0003cf0 con 0x7f68fc1005b0 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 Processor -- start 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 -- start start 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68e3fff640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68e40a2230 0x7f68e40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ac7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9a35530 con 0x55d7f9ac8080 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9a32a60 con 0x55d7f9ac7760 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f690400e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9a2e240 con 0x55d7f9ad0c60 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68fb7fe640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68e40a2230 0x7f68e40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68e3fff640 1 -- 172.21.15.116:0/250544175 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f68f00f02e0 con 0x7f68fc1005b0 2024-03-24T09:24:23.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ac7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68fb7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ac7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34962/0 (socket says 172.21.15.116:34962) 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 -- 172.21.15.116:0/4029968159 learned_addr learned my addr 172.21.15.116:0/4029968159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 -- 172.21.15.116:0/4029968159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 msgr2=0x55d7f9ad3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 -- 172.21.15.116:0/4029968159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 msgr2=0x55d7f9ad0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.063+0000 7f68faffd640 1 -- 172.21.15.116:0/4029968159 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f9a2c8c0 con 0x55d7f9ac7760 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.064+0000 7f68fb7fe640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68e40a2230 0x7f68e40a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f68f4005fd0 tx=0x7f68f4009b70 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.064+0000 7f68faffd640 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ac7b40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0114f0 tx=0x7f68ec0119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34c5327e29d7cbd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.064+0000 7f68e17fa640 1 -- 172.21.15.116:0/4029968159 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec00f540 con 0x55d7f9ac7760 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.064+0000 7f68e17fa640 1 -- 172.21.15.116:0/4029968159 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68ec00fb50 con 0x55d7f9ac7760 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.064+0000 7f68e17fa640 1 -- 172.21.15.116:0/4029968159 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec019b20 con 0x55d7f9ac7760 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 -- 172.21.15.116:0/4029968159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 msgr2=0x55d7f9ac7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ac7b40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0114f0 tx=0x7f68ec0119b0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 -- 172.21.15.116:0/4029968159 shutdown_connections 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 --2- 172.21.15.116:0/4029968159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ac7b40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 -- 172.21.15.116:0/4029968159 >> 172.21.15.116:0/4029968159 conn(0x55d7f9ac1e90 msgr2=0x55d7f9ac2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 -- 172.21.15.116:0/4029968159 shutdown_connections 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 -- 172.21.15.116:0/4029968159 wait complete. 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 -- start start 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.065+0000 7f690400e740 1 Processor -- start 2024-03-24T09:24:23.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9acdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9ace1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ace700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7f99d2f80 con 0x55d7f9ac7760 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7f97014b0 con 0x55d7f9ac8080 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7f99b3ad0 con 0x55d7f9ad0c60 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9ace1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f690400e740 10 client.0 _ll_register_callbacks cb 0x55d7f99e6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9ace1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34972/0 (socket says 172.21.15.116:34972) 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 -- 172.21.15.116:0/4043368248 learned_addr learned my addr 172.21.15.116:0/4043368248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fb7fe640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ace700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68faffd640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9acdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 -- 172.21.15.116:0/4043368248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 msgr2=0x55d7f9ace700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ace700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 -- 172.21.15.116:0/4043368248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 msgr2=0x55d7f9acdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9acdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 -- 172.21.15.116:0/4043368248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ec009cc0 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9ace1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f68f00ed7f0 tx=0x7f68f00bc390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486fa8c993a737a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.066+0000 7f68deff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f68deff5640 1 -- 172.21.15.116:0/4043368248 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f00bce90 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d7f9a527a0 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f68deff5640 1 -- 172.21.15.116:0/4043368248 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f68f00c0b80 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f68deff5640 1 -- 172.21.15.116:0/4043368248 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f68f00f0170 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 10 client.26213 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d7f99e46d0 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f68deff5640 1 -- 172.21.15.116:0/4043368248 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68f00d9080 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f68deff5640 1 -- 172.21.15.116:0/4043368248 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f68f003c070 con 0x55d7f9ac8080 2024-03-24T09:24:23.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 10 client.26213 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 20 client.26213 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.067+0000 7f690400e740 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9ae0c40 0x55d7f9ae3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.068+0000 7f690400e740 4 client.26213 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.068+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7f9a241f0 con 0x55d7f9ae0c40 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.068+0000 7f68faffd640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9ae0c40 0x55d7f9ae3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.068+0000 7f68deff5640 10 client.26213 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.068+0000 7f68faffd640 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9ae0c40 0x55d7f9ae3080 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5978fb42c58ed6b server_cookie=51c1955b316417c3 in_seq=0 out_seq=0 2024-03-24T09:24:23.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f68deff5640 1 -- 172.21.15.116:0/4043368248 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d7f9a241f0 con 0x55d7f9ae0c40 2024-03-24T09:24:23.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f68deff5640 10 client.26213 handle_command_reply: tid=0 2024-03-24T09:24:23.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 client.26213 shutdown 2024-03-24T09:24:23.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 msgr2=0x55d7f9ace1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9ace1c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f68f00ed7f0 tx=0x7f68f00bc390 comp rx=0 tx=0).stop 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 shutdown_connections 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ace700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9ae0c40 0x55d7f9ae3080 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9ace1c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 --2- 172.21.15.116:0/4043368248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9acdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f68deff5640 0 client.26213 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 >> 172.21.15.116:0/4043368248 conn(0x55d7f9ac1e90 msgr2=0x55d7f9ac3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 shutdown_connections 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 1 -- 172.21.15.116:0/4043368248 wait complete. 2024-03-24T09:24:23.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.071+0000 7f690400e740 20 client.26213 trim_cache size 0 max 16384 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 Processor -- start 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 -- start start 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9adc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ae8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9ac2e90 con 0x55d7f9ac7760 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9adca60 con 0x55d7f9ac8080 2024-03-24T09:24:23.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f690400e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9acefd0 con 0x55d7f9ad0c60 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.089+0000 7f68faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68fb7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ae8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44376/0 (socket says 172.21.15.116:44376) 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9adc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 -- 172.21.15.116:0/2286865685 learned_addr learned my addr 172.21.15.116:0/2286865685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 -- 172.21.15.116:0/2286865685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 msgr2=0x55d7f9ae8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ae8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 -- 172.21.15.116:0/2286865685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 msgr2=0x55d7f9adc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9adc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 -- 172.21.15.116:0/2286865685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f99c8c60 con 0x55d7f9ac7760 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68fb7fe640 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ae8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:23.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9adc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68faffd640 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad03b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0114f0 tx=0x7f68ec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3c6e1875d0b7f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68e0ff9640 1 -- 172.21.15.116:0/2286865685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec009b50 con 0x55d7f9ac7760 2024-03-24T09:24:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68e0ff9640 1 -- 172.21.15.116:0/2286865685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68ec019570 con 0x55d7f9ac7760 2024-03-24T09:24:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.090+0000 7f68e0ff9640 1 -- 172.21.15.116:0/2286865685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec005260 con 0x55d7f9ac7760 2024-03-24T09:24:23.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 -- 172.21.15.116:0/2286865685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 msgr2=0x55d7f9ad03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad03b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0114f0 tx=0x7f68ec014000 comp rx=0 tx=0).stop 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 -- 172.21.15.116:0/2286865685 shutdown_connections 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ae8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac8080 0x55d7f9adc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 --2- 172.21.15.116:0/2286865685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad03b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 -- 172.21.15.116:0/2286865685 >> 172.21.15.116:0/2286865685 conn(0x55d7f9ac1e90 msgr2=0x55d7f9ac3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 -- 172.21.15.116:0/2286865685 shutdown_connections 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.091+0000 7f690400e740 1 -- 172.21.15.116:0/2286865685 wait complete. 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 -- start start 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 Processor -- start 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9ad6960 con 0x55d7f9ac8080 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9b46f90 con 0x55d7f9ac7760 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d7f9b470d0 con 0x55d7f9ad0c60 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f68faffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f68fa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44390/0 (socket says 172.21.15.116:44390) 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f690400e740 10 client.0 _ll_register_callbacks cb 0x7f68f00bc570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f68fa7fc640 1 -- 172.21.15.116:0/1290138018 learned_addr learned my addr 172.21.15.116:0/1290138018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.092+0000 7f68fb7fe640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68faffd640 1 -- 172.21.15.116:0/1290138018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 msgr2=0x55d7f9ad6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68faffd640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68faffd640 1 -- 172.21.15.116:0/1290138018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 msgr2=0x55d7f9ad5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68faffd640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68faffd640 1 -- 172.21.15.116:0/1290138018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f99e46d0 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68faffd640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad8fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f68ec02c000 tx=0x7f68ec009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca13c4cc876ee862 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68deff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68deff5640 1 -- 172.21.15.116:0/1290138018 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec009b50 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68ec00fb50 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68deff5640 1 -- 172.21.15.116:0/1290138018 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f68ec011170 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68deff5640 1 -- 172.21.15.116:0/1290138018 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f68ec00f540 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f690400e740 10 client.26219 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f68f401a040 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.093+0000 7f68deff5640 1 -- 172.21.15.116:0/1290138018 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f68ec00fb50 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f68deff5640 1 -- 172.21.15.116:0/1290138018 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f68ec019750 con 0x55d7f9ac7760 2024-03-24T09:24:23.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f690400e740 10 client.26219 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f690400e740 10 client.26219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f690400e740 20 client.26219 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f690400e740 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9b48690 0x55d7f9b4aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f690400e740 4 client.26219 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d7f972d860 con 0x55d7f9b48690 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9b48690 0x55d7f9b4aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f68deff5640 10 client.26219 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.094+0000 7f68fa7fc640 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9b48690 0x55d7f9b4aab0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c95ade11b406fb9 server_cookie=b399051b380cd1 in_seq=0 out_seq=0 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f68deff5640 1 -- 172.21.15.116:0/1290138018 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d7f972d860 con 0x55d7f9b48690 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f68deff5640 10 client.26219 handle_command_reply: tid=0 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 client.26219 shutdown 2024-03-24T09:24:23.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 msgr2=0x55d7f9ad8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad8fc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f68ec02c000 tx=0x7f68ec009390 comp rx=0 tx=0).stop 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 shutdown_connections 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d7f9ad0c60 0x55d7f9ad6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d7f9ac8080 0x55d7f9ad5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55d7f9b48690 0x55d7f9b4aab0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 --2- 172.21.15.116:0/1290138018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d7f9ac7760 0x55d7f9ad8fc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 >> 172.21.15.116:0/1290138018 conn(0x55d7f9ac1e90 msgr2=0x55d7f9acdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f68deff5640 0 client.26219 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 shutdown_connections 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 1 -- 172.21.15.116:0/1290138018 wait complete. 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.096+0000 7f690400e740 20 client.26219 trim_cache size 0 max 16384 2024-03-24T09:24:23.103 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:24:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:24:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "8e5afa01-c677-43d6-aa97-b44989ce332a", 2024-03-24T09:24:23.104 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:24:23.104 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68e40a2230 msgr2=0x7f68e40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68e40a2230 0x7f68e40a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f68f4005fd0 tx=0x7f68f4009b70 comp rx=0 tx=0).stop 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 msgr2=0x7f68fc19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 0x7f68fc19dfa0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f68f0009230 tx=0x7f68f0004040 comp rx=0 tx=0).stop 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 shutdown_connections 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f68e40a2230 0x7f68e40a46f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f68fc1076d0 0x7f68fc197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68fc1005b0 0x7f68fc19dfa0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 --2- 172.21.15.116:0/250544175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68fc0ffc10 0x7f68fc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 >> 172.21.15.116:0/250544175 conn(0x7f68fc076bd0 msgr2=0x7f68fc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 shutdown_connections 2024-03-24T09:24:23.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.097+0000 7f690135c640 1 -- 172.21.15.116:0/250544175 wait complete. 2024-03-24T09:24:23.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:24:23.110 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:24:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 -- 172.21.15.116:0/3865484900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76810e120 msgr2=0x7fe768110510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3865484900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76810e120 0x7fe768110510 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe760009be0 tx=0x7fe76002bef0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 -- 172.21.15.116:0/3865484900 shutdown_connections 2024-03-24T09:24:23.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3865484900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe76810e120 0x7fe768110510 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3865484900 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe768101a00 0x7fe76810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3865484900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7681010e0 0x7fe7681014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.234+0000 7fe76f24e640 1 -- 172.21.15.116:0/3865484900 >> 172.21.15.116:0/3865484900 conn(0x7fe7680776a0 msgr2=0x7fe768077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.235+0000 7fe76f24e640 1 -- 172.21.15.116:0/3865484900 shutdown_connections 2024-03-24T09:24:23.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.235+0000 7fe76f24e640 1 -- 172.21.15.116:0/3865484900 wait complete. 2024-03-24T09:24:23.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.235+0000 7fe76f24e640 1 Processor -- start 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.235+0000 7fe76f24e640 1 -- start start 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76f24e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 0x7fe768199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76f24e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe768101a00 0x7fe76819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76f24e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe76810e120 0x7fe76819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76f24e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe76819ed50 con 0x7fe7681010e0 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76f24e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe76819eec0 con 0x7fe768101a00 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76f24e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe76819f080 con 0x7fe76810e120 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 0x7fe768199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 0x7fe768199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44408/0 (socket says 172.21.15.116:44408) 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 -- 172.21.15.116:0/3936667149 learned_addr learned my addr 172.21.15.116:0/3936667149 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe767fff640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe768101a00 0x7fe76819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe76810e120 0x7fe76819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 -- 172.21.15.116:0/3936667149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe76810e120 msgr2=0x7fe76819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe76810e120 0x7fe76819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 -- 172.21.15.116:0/3936667149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe768101a00 msgr2=0x7fe76819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe768101a00 0x7fe76819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 -- 172.21.15.116:0/3936667149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe760009840 con 0x7fe7681010e0 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.236+0000 7fe76cfea640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 0x7fe768199ee0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00e9c0 tx=0x7fe75c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.237+0000 7fe765ffb640 1 -- 172.21.15.116:0/3936667149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c00ccc0 con 0x7fe7681010e0 2024-03-24T09:24:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.237+0000 7fe765ffb640 1 -- 172.21.15.116:0/3936667149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe75c00ce20 con 0x7fe7681010e0 2024-03-24T09:24:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.237+0000 7fe765ffb640 1 -- 172.21.15.116:0/3936667149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c004e10 con 0x7fe7681010e0 2024-03-24T09:24:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.237+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe76819f360 con 0x7fe7681010e0 2024-03-24T09:24:23.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.237+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7681a6be0 con 0x7fe7681010e0 2024-03-24T09:24:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.240+0000 7fe765ffb640 1 -- 172.21.15.116:0/3936667149 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe75c004f70 con 0x7fe7681010e0 2024-03-24T09:24:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.241+0000 7fe771e2b740 1 Processor -- start 2024-03-24T09:24:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.241+0000 7fe771e2b740 1 -- start start 2024-03-24T09:24:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.241+0000 7fe765ffb640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe7580a2380 0x7fe7580a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.241+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe7580a2380 0x7fe7580a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.241+0000 7fe765ffb640 1 -- 172.21.15.116:0/3936667149 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe75c014070 con 0x7fe7681010e0 2024-03-24T09:24:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3c1d0 0x55b203f3c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f3caf0 0x55b203f45190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 0x55b203f47ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe771e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b203e60f30 con 0x55b203f456d0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe771e2b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b203eaa810 con 0x55b203f3caf0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe7580a2380 0x7fe7580a4840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe760004af0 tx=0x7fe760005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe771e2b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b203ea7d40 con 0x55b203f3c1d0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe76d7eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 0x55b203f47ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3c1d0 0x55b203f3c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe76d7eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 0x55b203f47ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44424/0 (socket says 172.21.15.116:44424) 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe76d7eb640 1 -- 172.21.15.116:0/1741677691 learned_addr learned my addr 172.21.15.116:0/1741677691 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.242+0000 7fe767fff640 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f3caf0 0x55b203f45190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe76d7eb640 1 -- 172.21.15.116:0/1741677691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3c1d0 msgr2=0x55b203f3c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3c1d0 0x55b203f3c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe76d7eb640 1 -- 172.21.15.116:0/1741677691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f3caf0 msgr2=0x55b203f45190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f3caf0 0x55b203f45190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe76d7eb640 1 -- 172.21.15.116:0/1741677691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b203e64ea0 con 0x55b203f456d0 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 0x55b203f47ac0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe760035000 tx=0x7fe76003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9fa2f3a28bc9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe752ffd640 1 -- 172.21.15.116:0/1741677691 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760033810 con 0x55b203f456d0 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe752ffd640 1 -- 172.21.15.116:0/1741677691 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe760003e20 con 0x55b203f456d0 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.243+0000 7fe752ffd640 1 -- 172.21.15.116:0/1741677691 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760034430 con 0x55b203f456d0 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 -- 172.21.15.116:0/1741677691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 msgr2=0x55b203f47ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 0x55b203f47ac0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fe760035000 tx=0x7fe76003f000 comp rx=0 tx=0).stop 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 -- 172.21.15.116:0/1741677691 shutdown_connections 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f456d0 0x55b203f47ac0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f3caf0 0x55b203f45190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 --2- 172.21.15.116:0/1741677691 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3c1d0 0x55b203f3c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 -- 172.21.15.116:0/1741677691 >> 172.21.15.116:0/1741677691 conn(0x55b203f36900 msgr2=0x55b203f36ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 -- 172.21.15.116:0/1741677691 shutdown_connections 2024-03-24T09:24:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.244+0000 7fe771e2b740 1 -- 172.21.15.116:0/1741677691 wait complete. 2024-03-24T09:24:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.245+0000 7fe771e2b740 1 -- start start 2024-03-24T09:24:23.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.245+0000 7fe771e2b740 1 Processor -- start 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.245+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.245+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f42c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f43140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f426c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44426/0 (socket says 172.21.15.116:44426) 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 -- 172.21.15.116:0/3087511084 learned_addr learned my addr 172.21.15.116:0/3087511084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b203ea3520 con 0x55b203f3c1d0 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b203b76430 con 0x55b203f456d0 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b203b32e00 con 0x55b203f3caf0 2024-03-24T09:24:23.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe771e2b740 10 client.0 _ll_register_callbacks cb 0x55b203e41d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f43140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 -- 172.21.15.116:0/3087511084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 msgr2=0x55b203f42c00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f42c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 -- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 msgr2=0x55b203f43140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f43140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7681110a0 con 0x55b203f3c1d0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe76cfea640 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f426c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0cc0a0 tx=0x7fe75c09b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50d081960ebd8b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.246+0000 7fe747fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe747fff640 1 -- 172.21.15.116:0/3087511084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0b7070 con 0x55b203f3c1d0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe747fff640 1 -- 172.21.15.116:0/3087511084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe75c0a34e0 con 0x55b203f3c1d0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b203ba27e0 con 0x55b203f3c1d0 2024-03-24T09:24:23.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe747fff640 1 -- 172.21.15.116:0/3087511084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe75c0b92b0 con 0x55b203f3c1d0 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 10 client.16401 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b203ea1ba0 con 0x55b203f3c1d0 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe747fff640 1 -- 172.21.15.116:0/3087511084 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b203ba27e0 con 0x55b203f3c1d0 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe747fff640 1 -- 172.21.15.116:0/3087511084 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fe75c01f060 con 0x55b203f3c1d0 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 10 client.16401 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 20 client.16401 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203f55910 0x55b203f57d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 4 client.16401 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.247+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b203bdf720 con 0x55b203f55910 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.248+0000 7fe767fff640 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203f55910 0x55b203f57d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.249+0000 7fe767fff640 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203f55910 0x55b203f57d50 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc6ea4c3d64f5f41 server_cookie=b6829c1f97a80801 in_seq=0 out_seq=0 2024-03-24T09:24:23.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.249+0000 7fe747fff640 10 client.16401 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe747fff640 1 -- 172.21.15.116:0/3087511084 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b203bdf720 con 0x55b203f55910 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe747fff640 10 client.16401 handle_command_reply: tid=0 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 client.16401 shutdown 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 msgr2=0x55b203f426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f426c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0cc0a0 tx=0x7fe75c09b440 comp rx=0 tx=0).stop 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 shutdown_connections 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203f55910 0x55b203f57d50 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f43140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f42c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 --2- 172.21.15.116:0/3087511084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f426c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 >> 172.21.15.116:0/3087511084 conn(0x55b203f36900 msgr2=0x55b203f35b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe747fff640 0 client.16401 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 shutdown_connections 2024-03-24T09:24:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 1 -- 172.21.15.116:0/3087511084 wait complete. 2024-03-24T09:24:23.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.250+0000 7fe771e2b740 20 client.16401 trim_cache size 0 max 16384 2024-03-24T09:24:23.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 Processor -- start 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 -- start start 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f44df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f50de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f5d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b203e28430 con 0x55b203f3c1d0 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b203f52410 con 0x55b203f456d0 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.272+0000 7fe771e2b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b203f51320 con 0x55b203f3caf0 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f44df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f44df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44434/0 (socket says 172.21.15.116:44434) 2024-03-24T09:24:23.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 -- 172.21.15.116:0/155642035 learned_addr learned my addr 172.21.15.116:0/155642035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe767fff640 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f50de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f5d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 -- 172.21.15.116:0/155642035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 msgr2=0x55b203f50de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f50de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 -- 172.21.15.116:0/155642035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 msgr2=0x55b203f5d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f5d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 -- 172.21.15.116:0/155642035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b203e3db20 con 0x55b203f3c1d0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe76cfea640 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f44df0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe75c09af00 tx=0x7fe75c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83429685046e9c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe7527fc640 1 -- 172.21.15.116:0/155642035 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0b7020 con 0x55b203f3c1d0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe7527fc640 1 -- 172.21.15.116:0/155642035 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe75c0b92b0 con 0x55b203f3c1d0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.273+0000 7fe7527fc640 1 -- 172.21.15.116:0/155642035 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0a34e0 con 0x55b203f3c1d0 2024-03-24T09:24:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 -- 172.21.15.116:0/155642035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 msgr2=0x55b203f44df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f44df0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe75c09af00 tx=0x7fe75c00cf90 comp rx=0 tx=0).stop 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 -- 172.21.15.116:0/155642035 shutdown_connections 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f5d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f50de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 --2- 172.21.15.116:0/155642035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f44df0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 -- 172.21.15.116:0/155642035 >> 172.21.15.116:0/155642035 conn(0x55b203f36900 msgr2=0x55b203f3da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 -- 172.21.15.116:0/155642035 shutdown_connections 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 -- 172.21.15.116:0/155642035 wait complete. 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 -- start start 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.274+0000 7fe771e2b740 1 Processor -- start 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f4d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f4d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f489e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b203f490b0 con 0x55b203f3c1d0 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b203f49220 con 0x55b203f456d0 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b203f493e0 con 0x55b203f3caf0 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe771e2b740 10 client.0 _ll_register_callbacks cb 0x7fe75c09a8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f4d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f4d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44450/0 (socket says 172.21.15.116:44450) 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 -- 172.21.15.116:0/960381352 learned_addr learned my addr 172.21.15.116:0/960381352 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe767fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f4d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76d7eb640 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 -- 172.21.15.116:0/960381352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 msgr2=0x55b203f4d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f4d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 -- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 msgr2=0x55b203f489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.275+0000 7fe76cfea640 1 -- 172.21.15.116:0/960381352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b203ea1ba0 con 0x55b203f3c1d0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe76cfea640 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f4d330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00ef90 tx=0x7fe75c005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c162bf5b6e7a3828 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe747fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe747fff640 1 -- 172.21.15.116:0/960381352 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0b7070 con 0x55b203f3c1d0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe747fff640 1 -- 172.21.15.116:0/960381352 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fe75c0a34e0 con 0x55b203f3c1d0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b203c2e890 con 0x55b203f3c1d0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe747fff640 1 -- 172.21.15.116:0/960381352 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe75c0b92b0 con 0x55b203f3c1d0 2024-03-24T09:24:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 10 client.16407 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b203e63700 con 0x55b203f3c1d0 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe747fff640 1 -- 172.21.15.116:0/960381352 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b203c2e890 con 0x55b203f3c1d0 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe747fff640 1 -- 172.21.15.116:0/960381352 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fe75c01f070 con 0x55b203f3c1d0 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 10 client.16407 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 20 client.16407 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203fbbbe0 0x55b203fbdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 4 client.16407 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.276+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b203b4e700 con 0x55b203fbbbe0 2024-03-24T09:24:23.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.277+0000 7fe767fff640 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203fbbbe0 0x55b203fbdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe747fff640 10 client.16407 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe767fff640 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203fbbbe0 0x55b203fbdfd0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87be29bf8c5766a3 server_cookie=7df8711e8bb5596c in_seq=0 out_seq=0 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe747fff640 1 -- 172.21.15.116:0/960381352 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b203b4e700 con 0x55b203fbbbe0 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe747fff640 10 client.16407 handle_command_reply: tid=0 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 client.16407 shutdown 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 msgr2=0x55b203f4d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f4d330 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00ef90 tx=0x7fe75c005750 comp rx=0 tx=0).stop 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 shutdown_connections 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b203fbbbe0 0x55b203fbdfd0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe747fff640 0 client.16407 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b203f456d0 0x55b203f489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b203f3caf0 0x55b203f4d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 --2- 172.21.15.116:0/960381352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b203f3c1d0 0x55b203f4d330 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 >> 172.21.15.116:0/960381352 conn(0x55b203f36900 msgr2=0x55b203f37d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 shutdown_connections 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 1 -- 172.21.15.116:0/960381352 wait complete. 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.278+0000 7fe771e2b740 20 client.16407 trim_cache size 0 max 16384 2024-03-24T09:24:23.287 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: "8e5afa01-c677-43d6-aa97-b44989ce332a": { 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "8e5afa01-c677-43d6-aa97-b44989ce332a", 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:24:23.288 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:24:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe7580a2380 msgr2=0x7fe7580a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe7580a2380 0x7fe7580a4840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe760004af0 tx=0x7fe760005fd0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 msgr2=0x7fe768199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 0x7fe768199ee0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00e9c0 tx=0x7fe75c00edc0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 shutdown_connections 2024-03-24T09:24:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe7580a2380 0x7fe7580a4840 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe76810e120 0x7fe76819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe768101a00 0x7fe76819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 --2- 172.21.15.116:0/3936667149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe7681010e0 0x7fe768199ee0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:23.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 >> 172.21.15.116:0/3936667149 conn(0x7fe7680776a0 msgr2=0x7fe76810f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:23.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 shutdown_connections 2024-03-24T09:24:23.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:23.279+0000 7fe76f24e640 1 -- 172.21.15.116:0/3936667149 wait complete. 2024-03-24T09:24:23.291 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:23 smithi116 ceph-mon[32207]: pgmap v71: 129 pgs: 129 active+clean; 121 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 286 op/s 2024-03-24T09:24:23.291 INFO:tasks.cephfs.filesystem:scrub status for tag:8e5afa01-c677-43d6-aa97-b44989ce332a - {'path': '/', 'tag': '8e5afa01-c677-43d6-aa97-b44989ce332a', 'options': 'recursive,force'} 2024-03-24T09:24:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:23 smithi117 ceph-mon[35788]: pgmap v71: 129 pgs: 129 active+clean; 121 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 286 op/s 2024-03-24T09:24:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:23 smithi143 ceph-mon[35965]: pgmap v71: 129 pgs: 129 active+clean; 121 MiB data, 812 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.7 MiB/s wr, 286 op/s 2024-03-24T09:24:24.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:24 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:24 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:24:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:24 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:25 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:25 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:25 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:25.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:25 smithi117 ceph-mon[35788]: pgmap v72: 129 pgs: 129 active+clean; 121 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 203 op/s 2024-03-24T09:24:25.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:25 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:25.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:25 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:25 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:25 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:25 smithi116 ceph-mon[32207]: pgmap v72: 129 pgs: 129 active+clean; 121 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 203 op/s 2024-03-24T09:24:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:25 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:25 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:25 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:25 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:25 smithi143 ceph-mon[35965]: pgmap v72: 129 pgs: 129 active+clean; 121 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 203 op/s 2024-03-24T09:24:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:25 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:27 smithi117 ceph-mon[35788]: pgmap v73: 129 pgs: 129 active+clean; 121 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 187 op/s 2024-03-24T09:24:27.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:27 smithi116 ceph-mon[32207]: pgmap v73: 129 pgs: 129 active+clean; 121 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 187 op/s 2024-03-24T09:24:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:27 smithi143 ceph-mon[35965]: pgmap v73: 129 pgs: 129 active+clean; 121 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 187 op/s 2024-03-24T09:24:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:29 smithi117 ceph-mon[35788]: pgmap v74: 129 pgs: 129 active+clean; 132 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 277 op/s 2024-03-24T09:24:29.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:29 smithi116 ceph-mon[32207]: pgmap v74: 129 pgs: 129 active+clean; 132 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 277 op/s 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.355 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T09:24:29.356 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-03-24T09:24:29.357 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-24T09:24:29.367 INFO:tasks.workunit.client.0.smithi116.stderr:+ make -j4 2024-03-24T09:24:29.392 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize -cfi 2024-03-24T09:24:29.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:29 smithi143 ceph-mon[35965]: pgmap v74: 129 pgs: 129 active+clean; 132 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 277 op/s 2024-03-24T09:24:29.803 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-24T09:24:29.803 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file './config.guess' 2024-03-24T09:24:29.812 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file './config.sub' 2024-03-24T09:24:29.824 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file './install-sh' 2024-03-24T09:24:29.835 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file './ltmain.sh' 2024-03-24T09:24:29.891 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-24T09:24:29.891 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-24T09:24:29.915 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-24T09:24:29.937 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-24T09:24:29.958 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-24T09:24:29.979 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-24T09:24:29.987 INFO:tasks.workunit.client.0.smithi116.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-24T09:24:29.989 INFO:tasks.workunit.client.0.smithi116.stdout:cp include/install-sh . 2024-03-24T09:24:29.999 INFO:tasks.workunit.client.0.smithi116.stdout:aclocal -I m4 2024-03-24T09:24:31.264 INFO:tasks.workunit.client.0.smithi116.stdout:autoheader 2024-03-24T09:24:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:31 smithi117 ceph-mon[35788]: pgmap v75: 129 pgs: 129 active+clean; 132 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-03-24T09:24:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:31 smithi116 ceph-mon[32207]: pgmap v75: 129 pgs: 129 active+clean; 132 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-03-24T09:24:31.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:31 smithi143 ceph-mon[35965]: pgmap v75: 129 pgs: 129 active+clean; 132 MiB data, 818 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-03-24T09:24:31.699 INFO:tasks.workunit.client.0.smithi116.stdout:autoconf 2024-03-24T09:24:32.012 INFO:tasks.workunit.client.0.smithi116.stdout:./configure \ 2024-03-24T09:24:32.012 INFO:tasks.workunit.client.0.smithi116.stdout: --libexecdir=/usr/lib \ 2024-03-24T09:24:32.012 INFO:tasks.workunit.client.0.smithi116.stdout: --exec_prefix=/var/lib 2024-03-24T09:24:32.211 INFO:tasks.workunit.client.0.smithi116.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-24T09:24:32.211 INFO:tasks.workunit.client.0.smithi116.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-24T09:24:32.212 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to print strings... printf 2024-03-24T09:24:32.213 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... gcc 2024-03-24T09:24:32.352 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the C compiler works... yes 2024-03-24T09:24:32.352 INFO:tasks.workunit.client.0.smithi116.stdout:checking for C compiler default output file name... a.out 2024-03-24T09:24:32.417 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of executables... 2024-03-24T09:24:32.475 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are cross compiling... no 2024-03-24T09:24:32.537 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of object files... o 2024-03-24T09:24:32.579 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are using the GNU C compiler... yes 2024-03-24T09:24:32.612 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... yes 2024-03-24T09:24:32.701 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-24T09:24:32.707 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-24T09:24:32.712 INFO:tasks.workunit.client.0.smithi116.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-24T09:24:32.714 INFO:tasks.workunit.client.0.smithi116.stdout:checking for egrep... /usr/bin/grep -E 2024-03-24T09:24:32.715 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-24T09:24:32.719 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-24T09:24:32.722 INFO:tasks.workunit.client.0.smithi116.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-24T09:24:32.734 INFO:tasks.workunit.client.0.smithi116.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-24T09:24:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:33 smithi117 ceph-mon[35788]: pgmap v76: 129 pgs: 129 active+clean; 140 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-24T09:24:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:33 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:24:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:33 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:24:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:33 smithi116 ceph-mon[32207]: pgmap v76: 129 pgs: 129 active+clean; 140 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-24T09:24:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:33 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:24:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:33 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:24:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:33 smithi143 ceph-mon[35965]: pgmap v76: 129 pgs: 129 active+clean; 140 MiB data, 846 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 248 op/s 2024-03-24T09:24:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:33 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:24:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:33 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:24:33.698 INFO:tasks.workunit.client.0.smithi116.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-24T09:24:33.698 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether ln -s works... yes 2024-03-24T09:24:33.708 INFO:tasks.workunit.client.0.smithi116.stdout:checking the maximum length of command line arguments... 1572864 2024-03-24T09:24:33.711 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-24T09:24:33.712 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-24T09:24:33.712 INFO:tasks.workunit.client.0.smithi116.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-24T09:24:33.713 INFO:tasks.workunit.client.0.smithi116.stdout:checking for objdump... objdump 2024-03-24T09:24:33.714 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to recognize dependent libraries... pass_all 2024-03-24T09:24:33.715 INFO:tasks.workunit.client.0.smithi116.stdout:checking for dlltool... no 2024-03-24T09:24:33.715 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-24T09:24:33.717 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ar... ar 2024-03-24T09:24:33.842 INFO:tasks.workunit.client.0.smithi116.stdout:checking for archiver @FILE support... @ 2024-03-24T09:24:33.843 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strip... strip 2024-03-24T09:24:33.843 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ranlib... ranlib 2024-03-24T09:24:33.844 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gawk... gawk 2024-03-24T09:24:34.026 INFO:tasks.workunit.client.0.smithi116.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-24T09:24:34.027 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sysroot... no 2024-03-24T09:24:34.053 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a working dd... /usr/bin/dd 2024-03-24T09:24:34.075 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-24T09:24:34.132 INFO:tasks.workunit.client.0.smithi116.stdout:checking for mt... no 2024-03-24T09:24:34.158 INFO:tasks.workunit.client.0.smithi116.stdout:checking if : is a manifest tool... no 2024-03-24T09:24:34.292 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to run the C preprocessor... gcc -E 2024-03-24T09:24:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:24:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:24:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:24:34.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:24:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:24:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:24:34.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:24:34.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:24:34.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:24:34.654 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ANSI C header files... yes 2024-03-24T09:24:34.714 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/types.h... yes 2024-03-24T09:24:34.792 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/stat.h... yes 2024-03-24T09:24:34.862 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdlib.h... yes 2024-03-24T09:24:35.022 INFO:tasks.workunit.client.0.smithi116.stdout:checking for string.h... yes 2024-03-24T09:24:35.103 INFO:tasks.workunit.client.0.smithi116.stdout:checking for memory.h... yes 2024-03-24T09:24:35.172 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strings.h... yes 2024-03-24T09:24:35.248 INFO:tasks.workunit.client.0.smithi116.stdout:checking for inttypes.h... yes 2024-03-24T09:24:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:35 smithi117 ceph-mon[35788]: pgmap v77: 129 pgs: 129 active+clean; 141 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-03-24T09:24:35.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:35 smithi116 ceph-mon[32207]: pgmap v77: 129 pgs: 129 active+clean; 141 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-03-24T09:24:35.326 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdint.h... yes 2024-03-24T09:24:35.401 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... yes 2024-03-24T09:24:35.470 INFO:tasks.workunit.client.0.smithi116.stdout:checking for dlfcn.h... yes 2024-03-24T09:24:35.484 INFO:tasks.workunit.client.0.smithi116.stdout:checking for objdir... .libs 2024-03-24T09:24:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:35 smithi143 ceph-mon[35965]: pgmap v77: 129 pgs: 129 active+clean; 141 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-03-24T09:24:35.670 INFO:tasks.workunit.client.0.smithi116.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-24T09:24:35.670 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-24T09:24:35.744 INFO:tasks.workunit.client.0.smithi116.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-24T09:24:35.808 INFO:tasks.workunit.client.0.smithi116.stdout:checking if gcc static flag -static works... no 2024-03-24T09:24:35.914 INFO:tasks.workunit.client.0.smithi116.stdout:checking if gcc supports -c -o file.o... yes 2024-03-24T09:24:35.914 INFO:tasks.workunit.client.0.smithi116.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-24T09:24:35.922 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-24T09:24:36.014 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether -lc should be explicitly linked in... no 2024-03-24T09:24:36.114 INFO:tasks.workunit.client.0.smithi116.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-24T09:24:36.114 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to hardcode library paths into programs... immediate 2024-03-24T09:24:36.116 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether stripping libraries is possible... yes 2024-03-24T09:24:36.116 INFO:tasks.workunit.client.0.smithi116.stdout:checking if libtool supports shared libraries... yes 2024-03-24T09:24:36.116 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether to build shared libraries... yes 2024-03-24T09:24:36.117 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether to build static libraries... yes 2024-03-24T09:24:36.125 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... (cached) gcc 2024-03-24T09:24:36.185 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-24T09:24:36.185 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-24T09:24:36.185 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-24T09:24:36.186 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gmake... /usr/bin/gmake 2024-03-24T09:24:36.186 INFO:tasks.workunit.client.0.smithi116.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-03-24T09:24:36.187 INFO:tasks.workunit.client.0.smithi116.stdout:checking for tar... /bin/tar 2024-03-24T09:24:36.187 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gzip... /bin/gzip 2024-03-24T09:24:36.215 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc -MM is supported... yes 2024-03-24T09:24:36.215 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sort... /bin/sort 2024-03-24T09:24:36.216 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether ln -s works... yes 2024-03-24T09:24:36.217 INFO:tasks.workunit.client.0.smithi116.stdout:checking for rpm... /bin/rpm 2024-03-24T09:24:36.221 INFO:tasks.workunit.client.0.smithi116.stdout:checking for rpmbuild... no 2024-03-24T09:24:36.221 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ANSI C header files... (cached) yes 2024-03-24T09:24:36.319 INFO:tasks.workunit.client.0.smithi116.stdout:checking assert.h usability... yes 2024-03-24T09:24:36.353 INFO:tasks.workunit.client.0.smithi116.stdout:checking assert.h presence... yes 2024-03-24T09:24:36.353 INFO:tasks.workunit.client.0.smithi116.stdout:checking for assert.h... yes 2024-03-24T09:24:36.452 INFO:tasks.workunit.client.0.smithi116.stdout:checking bstring.h usability... no 2024-03-24T09:24:36.503 INFO:tasks.workunit.client.0.smithi116.stdout:checking bstring.h presence... no 2024-03-24T09:24:36.503 INFO:tasks.workunit.client.0.smithi116.stdout:checking for bstring.h... no 2024-03-24T09:24:36.564 INFO:tasks.workunit.client.0.smithi116.stdout:checking libgen.h usability... yes 2024-03-24T09:24:36.588 INFO:tasks.workunit.client.0.smithi116.stdout:checking libgen.h presence... yes 2024-03-24T09:24:36.588 INFO:tasks.workunit.client.0.smithi116.stdout:checking for libgen.h... yes 2024-03-24T09:24:36.681 INFO:tasks.workunit.client.0.smithi116.stdout:checking dirent.h usability... yes 2024-03-24T09:24:36.711 INFO:tasks.workunit.client.0.smithi116.stdout:checking dirent.h presence... yes 2024-03-24T09:24:36.711 INFO:tasks.workunit.client.0.smithi116.stdout:checking for dirent.h... yes 2024-03-24T09:24:36.791 INFO:tasks.workunit.client.0.smithi116.stdout:checking errno.h usability... yes 2024-03-24T09:24:36.814 INFO:tasks.workunit.client.0.smithi116.stdout:checking errno.h presence... yes 2024-03-24T09:24:36.814 INFO:tasks.workunit.client.0.smithi116.stdout:checking for errno.h... yes 2024-03-24T09:24:36.893 INFO:tasks.workunit.client.0.smithi116.stdout:checking malloc.h usability... yes 2024-03-24T09:24:36.922 INFO:tasks.workunit.client.0.smithi116.stdout:checking malloc.h presence... yes 2024-03-24T09:24:36.922 INFO:tasks.workunit.client.0.smithi116.stdout:checking for malloc.h... yes 2024-03-24T09:24:37.016 INFO:tasks.workunit.client.0.smithi116.stdout:checking uuid.h usability... no 2024-03-24T09:24:37.063 INFO:tasks.workunit.client.0.smithi116.stdout:checking uuid.h presence... no 2024-03-24T09:24:37.063 INFO:tasks.workunit.client.0.smithi116.stdout:checking for uuid.h... no 2024-03-24T09:24:37.149 INFO:tasks.workunit.client.0.smithi116.stdout:checking uuid/uuid.h usability... yes 2024-03-24T09:24:37.174 INFO:tasks.workunit.client.0.smithi116.stdout:checking uuid/uuid.h presence... yes 2024-03-24T09:24:37.175 INFO:tasks.workunit.client.0.smithi116.stdout:checking for uuid/uuid.h... yes 2024-03-24T09:24:37.272 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/uuid.h usability... no 2024-03-24T09:24:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:37 smithi117 ceph-mon[35788]: pgmap v78: 129 pgs: 129 active+clean; 141 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-03-24T09:24:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:37 smithi116 ceph-mon[32207]: pgmap v78: 129 pgs: 129 active+clean; 141 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-03-24T09:24:37.326 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/uuid.h presence... no 2024-03-24T09:24:37.326 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/uuid.h... no 2024-03-24T09:24:37.398 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/file.h usability... yes 2024-03-24T09:24:37.423 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/file.h presence... yes 2024-03-24T09:24:37.423 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/file.h... yes 2024-03-24T09:24:37.512 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fcntl.h usability... yes 2024-03-24T09:24:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:37 smithi143 ceph-mon[35965]: pgmap v78: 129 pgs: 129 active+clean; 141 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-03-24T09:24:37.538 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fcntl.h presence... yes 2024-03-24T09:24:37.572 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/fcntl.h... yes 2024-03-24T09:24:37.629 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/syssgi.h usability... no 2024-03-24T09:24:37.681 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/syssgi.h presence... no 2024-03-24T09:24:37.682 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/syssgi.h... no 2024-03-24T09:24:37.819 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/param.h usability... yes 2024-03-24T09:24:37.849 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/param.h presence... yes 2024-03-24T09:24:37.849 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/param.h... yes 2024-03-24T09:24:37.876 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/stat.h... (cached) yes 2024-03-24T09:24:37.966 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/statvfs.h usability... yes 2024-03-24T09:24:38.025 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/statvfs.h presence... yes 2024-03-24T09:24:38.026 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/statvfs.h... yes 2024-03-24T09:24:38.142 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/time.h usability... yes 2024-03-24T09:24:38.167 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/time.h presence... yes 2024-03-24T09:24:38.167 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/time.h... yes 2024-03-24T09:24:38.249 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/ioctl.h usability... yes 2024-03-24T09:24:38.274 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/ioctl.h presence... yes 2024-03-24T09:24:38.274 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/ioctl.h... yes 2024-03-24T09:24:38.370 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/wait.h usability... yes 2024-03-24T09:24:38.397 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/wait.h presence... yes 2024-03-24T09:24:38.397 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/wait.h... yes 2024-03-24T09:24:38.425 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/types.h... (cached) yes 2024-03-24T09:24:38.444 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strings.h... (cached) yes 2024-03-24T09:24:38.521 INFO:tasks.workunit.client.0.smithi116.stdout:checking err.h usability... yes 2024-03-24T09:24:38.548 INFO:tasks.workunit.client.0.smithi116.stdout:checking err.h presence... yes 2024-03-24T09:24:38.548 INFO:tasks.workunit.client.0.smithi116.stdout:checking for err.h... yes 2024-03-24T09:24:38.631 INFO:tasks.workunit.client.0.smithi116.stdout:checking linux/falloc.h usability... yes 2024-03-24T09:24:38.656 INFO:tasks.workunit.client.0.smithi116.stdout:checking linux/falloc.h presence... yes 2024-03-24T09:24:38.656 INFO:tasks.workunit.client.0.smithi116.stdout:checking for linux/falloc.h... yes 2024-03-24T09:24:38.749 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-24T09:24:38.796 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-24T09:24:38.797 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-24T09:24:38.867 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-24T09:24:38.909 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-24T09:24:38.909 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-24T09:24:38.976 INFO:tasks.workunit.client.0.smithi116.stdout:checking xfs/platform_defs.h usability... no 2024-03-24T09:24:39.023 INFO:tasks.workunit.client.0.smithi116.stdout:checking xfs/platform_defs.h presence... no 2024-03-24T09:24:39.024 INFO:tasks.workunit.client.0.smithi116.stdout:checking for xfs/platform_defs.h... no 2024-03-24T09:24:39.090 INFO:tasks.workunit.client.0.smithi116.stdout:checking btrfs/ioctl.h usability... no 2024-03-24T09:24:39.138 INFO:tasks.workunit.client.0.smithi116.stdout:checking btrfs/ioctl.h presence... no 2024-03-24T09:24:39.138 INFO:tasks.workunit.client.0.smithi116.stdout:checking for btrfs/ioctl.h... no 2024-03-24T09:24:39.204 INFO:tasks.workunit.client.0.smithi116.stdout:checking cifs/ioctl.h usability... no 2024-03-24T09:24:39.254 INFO:tasks.workunit.client.0.smithi116.stdout:checking cifs/ioctl.h presence... no 2024-03-24T09:24:39.254 INFO:tasks.workunit.client.0.smithi116.stdout:checking for cifs/ioctl.h... no 2024-03-24T09:24:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:39 smithi117 ceph-mon[35788]: pgmap v79: 129 pgs: 129 active+clean; 144 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 210 op/s 2024-03-24T09:24:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:39 smithi116 ceph-mon[32207]: pgmap v79: 129 pgs: 129 active+clean; 144 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 210 op/s 2024-03-24T09:24:39.324 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/mman.h usability... yes 2024-03-24T09:24:39.352 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/mman.h presence... yes 2024-03-24T09:24:39.352 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/mman.h... yes 2024-03-24T09:24:39.421 INFO:tasks.workunit.client.0.smithi116.stdout:checking for xfs/xfs_log_format.h... no 2024-03-24T09:24:39.511 INFO:tasks.workunit.client.0.smithi116.stdout:checking for uuid_compare... no 2024-03-24T09:24:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:39 smithi143 ceph-mon[35965]: pgmap v79: 129 pgs: 129 active+clean; 144 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 210 op/s 2024-03-24T09:24:39.578 INFO:tasks.workunit.client.0.smithi116.stdout:checking for uuid_compare in -luuid... yes 2024-03-24T09:24:39.665 INFO:tasks.workunit.client.0.smithi116.stdout:checking for xfs/xfs.h... yes 2024-03-24T09:24:39.713 INFO:tasks.workunit.client.0.smithi116.stdout:checking for xfs/libxfs.h... no 2024-03-24T09:24:39.762 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-24T09:24:39.815 INFO:tasks.workunit.client.0.smithi116.stdout:checking for xfs/xqm.h... yes 2024-03-24T09:24:39.893 INFO:tasks.workunit.client.0.smithi116.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-24T09:24:39.942 INFO:tasks.workunit.client.0.smithi116.stdout:checking xfs/handle.h usability... yes 2024-03-24T09:24:39.968 INFO:tasks.workunit.client.0.smithi116.stdout:checking xfs/handle.h presence... yes 2024-03-24T09:24:39.968 INFO:tasks.workunit.client.0.smithi116.stdout:checking for xfs/handle.h... yes 2024-03-24T09:24:40.045 INFO:tasks.workunit.client.0.smithi116.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-24T09:24:40.120 INFO:tasks.workunit.client.0.smithi116.stdout:checking attr/attributes.h usability... yes 2024-03-24T09:24:40.144 INFO:tasks.workunit.client.0.smithi116.stdout:checking attr/attributes.h presence... yes 2024-03-24T09:24:40.144 INFO:tasks.workunit.client.0.smithi116.stdout:checking for attr/attributes.h... yes 2024-03-24T09:24:40.234 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/attributes.h usability... no 2024-03-24T09:24:40.281 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/attributes.h presence... no 2024-03-24T09:24:40.281 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/attributes.h... no 2024-03-24T09:24:40.364 INFO:tasks.workunit.client.0.smithi116.stdout:checking for attr_list in -lattr... yes 2024-03-24T09:24:40.430 INFO:tasks.workunit.client.0.smithi116.stdout:checking for attr_set in -lattr... yes 2024-03-24T09:24:40.486 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/acl.h usability... yes 2024-03-24T09:24:40.516 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/acl.h presence... yes 2024-03-24T09:24:40.516 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/acl.h... yes 2024-03-24T09:24:40.567 INFO:tasks.workunit.client.0.smithi116.stdout:checking acl/libacl.h usability... yes 2024-03-24T09:24:40.591 INFO:tasks.workunit.client.0.smithi116.stdout:checking acl/libacl.h presence... yes 2024-03-24T09:24:40.592 INFO:tasks.workunit.client.0.smithi116.stdout:checking for acl/libacl.h... yes 2024-03-24T09:24:40.659 INFO:tasks.workunit.client.0.smithi116.stdout:checking for acl_init in -lacl... yes 2024-03-24T09:24:40.720 INFO:tasks.workunit.client.0.smithi116.stdout:checking gdbm-ndbm.h usability... no 2024-03-24T09:24:40.763 INFO:tasks.workunit.client.0.smithi116.stdout:checking gdbm-ndbm.h presence... no 2024-03-24T09:24:40.764 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gdbm-ndbm.h... no 2024-03-24T09:24:40.817 INFO:tasks.workunit.client.0.smithi116.stdout:checking gdbm/ndbm.h usability... no 2024-03-24T09:24:40.860 INFO:tasks.workunit.client.0.smithi116.stdout:checking gdbm/ndbm.h presence... no 2024-03-24T09:24:40.860 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gdbm/ndbm.h... no 2024-03-24T09:24:40.915 INFO:tasks.workunit.client.0.smithi116.stdout:checking ndbm.h usability... no 2024-03-24T09:24:40.963 INFO:tasks.workunit.client.0.smithi116.stdout:checking ndbm.h presence... no 2024-03-24T09:24:40.963 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ndbm.h... no 2024-03-24T09:24:41.004 INFO:tasks.workunit.client.0.smithi116.stdout:checking libaio.h usability... yes 2024-03-24T09:24:41.032 INFO:tasks.workunit.client.0.smithi116.stdout:checking libaio.h presence... yes 2024-03-24T09:24:41.032 INFO:tasks.workunit.client.0.smithi116.stdout:checking for libaio.h... yes 2024-03-24T09:24:41.121 INFO:tasks.workunit.client.0.smithi116.stdout:checking for dm_make_handle in -ldm... no 2024-03-24T09:24:41.122 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-24T09:24:41.122 INFO:tasks.workunit.client.0.smithi116.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-24T09:24:41.122 INFO:tasks.workunit.client.0.smithi116.stdout:If you want DMAPI support please install the data migration 2024-03-24T09:24:41.122 INFO:tasks.workunit.client.0.smithi116.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-24T09:24:41.122 INFO:tasks.workunit.client.0.smithi116.stdout:from the dmapi source. 2024-03-24T09:24:41.122 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-03-24T09:24:41.167 INFO:tasks.workunit.client.0.smithi116.stdout:checking linux/fiemap.h usability... yes 2024-03-24T09:24:41.194 INFO:tasks.workunit.client.0.smithi116.stdout:checking linux/fiemap.h presence... yes 2024-03-24T09:24:41.194 INFO:tasks.workunit.client.0.smithi116.stdout:checking for linux/fiemap.h... yes 2024-03-24T09:24:41.262 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fallocate... yes 2024-03-24T09:24:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:41 smithi117 ceph-mon[35788]: pgmap v80: 129 pgs: 129 active+clean; 144 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-03-24T09:24:41.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:41 smithi116 ceph-mon[32207]: pgmap v80: 129 pgs: 129 active+clean; 144 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-03-24T09:24:41.350 INFO:tasks.workunit.client.0.smithi116.stdout:checking for open_by_handle_at... yes 2024-03-24T09:24:41.431 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/prctl.h usability... yes 2024-03-24T09:24:41.456 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/prctl.h presence... yes 2024-03-24T09:24:41.456 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/prctl.h... yes 2024-03-24T09:24:41.516 INFO:tasks.workunit.client.0.smithi116.stdout:checking linux/fs.h usability... yes 2024-03-24T09:24:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:41 smithi143 ceph-mon[35965]: pgmap v80: 129 pgs: 129 active+clean; 144 MiB data, 865 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-03-24T09:24:41.543 INFO:tasks.workunit.client.0.smithi116.stdout:checking linux/fs.h presence... yes 2024-03-24T09:24:41.543 INFO:tasks.workunit.client.0.smithi116.stdout:checking for linux/fs.h... yes 2024-03-24T09:24:41.612 INFO:tasks.workunit.client.0.smithi116.stdout:checking btrfsutil.h usability... no 2024-03-24T09:24:41.658 INFO:tasks.workunit.client.0.smithi116.stdout:checking btrfsutil.h presence... no 2024-03-24T09:24:41.658 INFO:tasks.workunit.client.0.smithi116.stdout:checking for btrfsutil.h... no 2024-03-24T09:24:41.719 INFO:tasks.workunit.client.0.smithi116.stdout:checking for copy_file_range... yes 2024-03-24T09:24:41.795 INFO:tasks.workunit.client.0.smithi116.stdout:checking for renameat2... yes 2024-03-24T09:24:41.826 INFO:tasks.workunit.client.0.smithi116.stdout:configure: creating ./config.status 2024-03-24T09:24:42.253 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating include/builddefs 2024-03-24T09:24:42.298 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating include/config.h 2024-03-24T09:24:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:43 smithi117 ceph-mon[35788]: pgmap v81: 129 pgs: 129 active+clean; 148 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-24T09:24:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:43 smithi116 ceph-mon[32207]: pgmap v81: 129 pgs: 129 active+clean; 148 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-24T09:24:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:43 smithi143 ceph-mon[35965]: pgmap v81: 129 pgs: 129 active+clean; 148 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-24T09:24:44.383 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: executing libtool commands 2024-03-24T09:24:44.572 INFO:tasks.workunit.client.0.smithi116.stdout:Building include 2024-03-24T09:24:44.572 INFO:tasks.workunit.client.0.smithi116.stdout:Building lib 2024-03-24T09:24:44.572 INFO:tasks.workunit.client.0.smithi116.stdout:Building tests 2024-03-24T09:24:44.592 INFO:tasks.workunit.client.0.smithi116.stdout: [LTDEP] 2024-03-24T09:24:44.924 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] dataascii.lo 2024-03-24T09:24:44.924 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] databin.lo 2024-03-24T09:24:44.924 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] datapid.lo 2024-03-24T09:24:44.925 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] file_lock.lo 2024-03-24T09:24:45.128 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] forker.lo 2024-03-24T09:24:45.139 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] pattern.lo 2024-03-24T09:24:45.182 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] open_flags.lo 2024-03-24T09:24:45.212 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] random_range.lo 2024-03-24T09:24:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:45 smithi117 ceph-mon[35788]: pgmap v82: 129 pgs: 129 active+clean; 148 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 807 KiB/s wr, 121 op/s 2024-03-24T09:24:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:45 smithi116 ceph-mon[32207]: pgmap v82: 129 pgs: 129 active+clean; 148 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 807 KiB/s wr, 121 op/s 2024-03-24T09:24:45.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:45 smithi143 ceph-mon[35965]: pgmap v82: 129 pgs: 129 active+clean; 148 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 807 KiB/s wr, 121 op/s 2024-03-24T09:24:45.328 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] string_to_tokens.lo 2024-03-24T09:24:45.373 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] str_to_bytes.lo 2024-03-24T09:24:45.439 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] tlibio.lo 2024-03-24T09:24:45.493 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] write_log.lo 2024-03-24T09:24:45.560 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] random.lo 2024-03-24T09:24:45.842 INFO:tasks.workunit.client.0.smithi116.stdout: [LD] 2024-03-24T09:24:46.074 INFO:tasks.workunit.client.0.smithi116.stdout:Building src 2024-03-24T09:24:46.074 INFO:tasks.workunit.client.0.smithi116.stdout:Building ltp 2024-03-24T09:24:46.074 INFO:tasks.workunit.client.0.smithi116.stdout:Building m4 2024-03-24T09:24:46.075 INFO:tasks.workunit.client.0.smithi116.stdout:Building common 2024-03-24T09:24:46.098 INFO:tasks.workunit.client.0.smithi116.stdout: [DEP] 2024-03-24T09:24:46.128 INFO:tasks.workunit.client.0.smithi116.stdout: [DEP] 2024-03-24T09:24:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:47 smithi116 ceph-mon[32207]: pgmap v83: 129 pgs: 129 active+clean; 148 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 735 KiB/s wr, 116 op/s 2024-03-24T09:24:47.421 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] fsstress 2024-03-24T09:24:47.422 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] doio 2024-03-24T09:24:47.422 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] fsx 2024-03-24T09:24:47.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:47 smithi143 ceph-mon[35965]: pgmap v83: 129 pgs: 129 active+clean; 148 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 735 KiB/s wr, 116 op/s 2024-03-24T09:24:47.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:47 smithi117 ceph-mon[35788]: pgmap v83: 129 pgs: 129 active+clean; 148 MiB data, 885 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 735 KiB/s wr, 116 op/s 2024-03-24T09:24:47.794 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c: In function 'attr_list_path': 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~ 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr:In file included from ../src/global.h:25, 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr: from fsstress.c:12: 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~ 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-24T09:24:47.795 INFO:tasks.workunit.client.0.smithi116.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~ 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:In file included from ../src/global.h:25, 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: from fsstress.c:12: 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~ 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c: In function 'attr_set_path': 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~ 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:In file included from ../src/global.h:25, 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr: from fsstress.c:12: 2024-03-24T09:24:47.796 INFO:tasks.workunit.client.0.smithi116.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-24T09:24:47.797 INFO:tasks.workunit.client.0.smithi116.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-03-24T09:24:47.797 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-03-24T09:24:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c: In function 'allocsp_f': 2024-03-24T09:24:47.806 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-24T09:24:47.807 INFO:tasks.workunit.client.0.smithi116.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-03-24T09:24:47.807 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~ 2024-03-24T09:24:47.807 INFO:tasks.workunit.client.0.smithi116.stderr: | XFS_IOC_RESVSP64 2024-03-24T09:24:47.807 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-03-24T09:24:47.827 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c: In function 'freesp_f': 2024-03-24T09:24:47.827 INFO:tasks.workunit.client.0.smithi116.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-24T09:24:47.827 INFO:tasks.workunit.client.0.smithi116.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-03-24T09:24:47.827 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~ 2024-03-24T09:24:47.827 INFO:tasks.workunit.client.0.smithi116.stderr: | XFS_IOC_RESVSP64 2024-03-24T09:24:47.865 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[3]: *** [Makefile:46: fsstress] Error 1 2024-03-24T09:24:47.865 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-24T09:24:48.260 INFO:tasks.workunit.client.0.smithi116.stderr:doio.c: In function 'parse_cmdline': 2024-03-24T09:24:48.260 INFO:tasks.workunit.client.0.smithi116.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-03-24T09:24:48.261 INFO:tasks.workunit.client.0.smithi116.stderr: 3113 | fprintf(stderr, 2024-03-24T09:24:48.261 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~ 2024-03-24T09:24:48.261 INFO:tasks.workunit.client.0.smithi116.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-03-24T09:24:48.261 INFO:tasks.workunit.client.0.smithi116.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-24T09:24:48.261 INFO:tasks.workunit.client.0.smithi116.stderr: 3115 | Prog, TagName, tok); 2024-03-24T09:24:48.261 INFO:tasks.workunit.client.0.smithi116.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-03-24T09:24:48.298 INFO:tasks.workunit.client.0.smithi116.stderr:fsx.c: In function 'main': 2024-03-24T09:24:48.298 INFO:tasks.workunit.client.0.smithi116.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-24T09:24:48.298 INFO:tasks.workunit.client.0.smithi116.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-24T09:24:48.298 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-24T09:24:48.299 INFO:tasks.workunit.client.0.smithi116.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-24T09:24:48.299 INFO:tasks.workunit.client.0.smithi116.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-03-24T09:24:48.299 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-24T09:24:48.433 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-03-24T09:24:48.434 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-03-24T09:24:49.368 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:49 smithi117 ceph-mon[35788]: pgmap v84: 129 pgs: 129 active+clean; 150 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 974 KiB/s wr, 161 op/s 2024-03-24T09:24:49.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:49 smithi143 ceph-mon[35965]: pgmap v84: 129 pgs: 129 active+clean; 150 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 974 KiB/s wr, 161 op/s 2024-03-24T09:24:49.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:49 smithi116 ceph-mon[32207]: pgmap v84: 129 pgs: 129 active+clean; 150 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 974 KiB/s wr, 161 op/s 2024-03-24T09:24:50.212 INFO:tasks.workunit.client.0.smithi116.stdout: [DEP] 2024-03-24T09:24:50.212 INFO:tasks.workunit.client.0.smithi116.stdout: [DEP] 2024-03-24T09:24:50.216 INFO:tasks.workunit.client.0.smithi116.stdout: [DEP] 2024-03-24T09:24:50.789 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] dirstress 2024-03-24T09:24:50.803 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] fill 2024-03-24T09:24:50.803 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] fill2 2024-03-24T09:24:50.807 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] getpagesize 2024-03-24T09:24:51.138 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] holes 2024-03-24T09:24:51.145 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] lstat64 2024-03-24T09:24:51.292 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] nametest 2024-03-24T09:24:51.308 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] permname 2024-03-24T09:24:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:51 smithi143 ceph-mon[35965]: pgmap v85: 129 pgs: 129 active+clean; 150 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 655 KiB/s wr, 112 op/s 2024-03-24T09:24:51.529 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] randholes 2024-03-24T09:24:51.546 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] runas 2024-03-24T09:24:51.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:51 smithi117 ceph-mon[35788]: pgmap v85: 129 pgs: 129 active+clean; 150 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 655 KiB/s wr, 112 op/s 2024-03-24T09:24:51.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:51 smithi116 ceph-mon[32207]: pgmap v85: 129 pgs: 129 active+clean; 150 MiB data, 890 MiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 655 KiB/s wr, 112 op/s 2024-03-24T09:24:51.721 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] truncfile 2024-03-24T09:24:51.755 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] usemem 2024-03-24T09:24:51.937 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] mmapcat 2024-03-24T09:24:52.016 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] append_reader 2024-03-24T09:24:52.078 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] append_writer 2024-03-24T09:24:52.099 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] dirperf 2024-03-24T09:24:52.268 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] metaperf 2024-03-24T09:24:52.339 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] devzero 2024-03-24T09:24:52.415 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] feature 2024-03-24T09:24:52.546 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] alloc 2024-03-24T09:24:52.698 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] fault 2024-03-24T09:24:52.759 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] fstest 2024-03-24T09:24:52.818 INFO:tasks.workunit.client.0.smithi116.stdout: [CC] t_access_root 2024-03-24T09:24:52.854 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c: In function 'main': 2024-03-24T09:24:52.854 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr: 154 | bozo! 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~ 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr: 154 | bozo! 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-03-24T09:24:52.855 INFO:tasks.workunit.client.0.smithi116.stderr: | ; 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr: 184 | rflag = 1; 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~ 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr: | nflag 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr: 320 | bozo! 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr: | ; 2024-03-24T09:24:52.856 INFO:tasks.workunit.client.0.smithi116.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-03-24T09:24:52.857 INFO:tasks.workunit.client.0.smithi116.stderr: 128 | struct flock64 f; 2024-03-24T09:24:52.857 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-03-24T09:24:52.859 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-03-24T09:24:52.859 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-24T09:24:53.060 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c: In function 'run_child': 2024-03-24T09:24:53.060 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~ 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~ 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-24T09:24:53.061 INFO:tasks.workunit.client.0.smithi116.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~ 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~ 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-24T09:24:53.062 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-24T09:24:53.156 INFO:tasks.workunit.client.0.smithi116.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-03-24T09:24:53.156 INFO:tasks.workunit.client.0.smithi116.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-03-24T09:24:53.156 INFO:tasks.workunit.client.0.smithi116.stderr:make: *** [Makefile:51: default] Error 2 2024-03-24T09:24:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-24T09:24:53.158 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-24T09:24:53.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T09:24:53.292 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:24:53.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:24:53.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 -- 172.21.15.116:0/1578826922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa994106660 msgr2=0x7fa994106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 --2- 172.21.15.116:0/1578826922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa994106660 0x7fa994106a40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa984009930 tx=0x7fa98402bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 -- 172.21.15.116:0/1578826922 shutdown_connections 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 --2- 172.21.15.116:0/1578826922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994100f60 0x7fa99410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 --2- 172.21.15.116:0/1578826922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9941005a0 0x7fa994100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 --2- 172.21.15.116:0/1578826922 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa994106660 0x7fa994106a40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 -- 172.21.15.116:0/1578826922 >> 172.21.15.116:0/1578826922 conn(0x7fa994077710 msgr2=0x7fa994077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 -- 172.21.15.116:0/1578826922 shutdown_connections 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.399+0000 7fa998b19640 1 -- 172.21.15.116:0/1578826922 wait complete. 2024-03-24T09:24:53.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 Processor -- start 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 -- start start 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9941005a0 0x7fa9941025f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa994100f60 0x7fa994102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 0x7fa994103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa9941a5fa0 con 0x7fa994106660 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa9941a6110 con 0x7fa994100f60 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa998b19640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa9941a62d0 con 0x7fa9941005a0 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 0x7fa994103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 0x7fa994103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37260/0 (socket says 172.21.15.116:37260) 2024-03-24T09:24:53.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 -- 172.21.15.116:0/3910968512 learned_addr learned my addr 172.21.15.116:0/3910968512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa99259c640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9941005a0 0x7fa9941025f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa991d9b640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa994100f60 0x7fa994102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 -- 172.21.15.116:0/3910968512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9941005a0 msgr2=0x7fa9941025f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9941005a0 0x7fa9941025f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 -- 172.21.15.116:0/3910968512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa994100f60 msgr2=0x7fa994102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.400+0000 7fa992d9d640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa994100f60 0x7fa994102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa992d9d640 1 -- 172.21.15.116:0/3910968512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c009840 con 0x7fa994106660 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa992d9d640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 0x7fa994103070 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00efd0 tx=0x7fa98c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa9837fe640 1 -- 172.21.15.116:0/3910968512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c009b40 con 0x7fa994106660 2024-03-24T09:24:53.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa9837fe640 1 -- 172.21.15.116:0/3910968512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa98c003b70 con 0x7fa994106660 2024-03-24T09:24:53.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa984009560 con 0x7fa994106660 2024-03-24T09:24:53.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa9837fe640 1 -- 172.21.15.116:0/3910968512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0107e0 con 0x7fa994106660 2024-03-24T09:24:53.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.401+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9941a68e0 con 0x7fa994106660 2024-03-24T09:24:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.403+0000 7fa9837fe640 1 -- 172.21.15.116:0/3910968512 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fa98c003ce0 con 0x7fa994106660 2024-03-24T09:24:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.403+0000 7fa9837fe640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa97c0a2260 0x7fa97c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.404+0000 7fa99259c640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa97c0a2260 0x7fa97c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.404+0000 7fa9837fe640 1 -- 172.21.15.116:0/3910968512 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa98c014070 con 0x7fa994106660 2024-03-24T09:24:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.404+0000 7fa99259c640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa97c0a2260 0x7fa97c0a4720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa984005ab0 tx=0x7fa984005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.404+0000 7fa99b7ce740 1 Processor -- start 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.404+0000 7fa99b7ce740 1 -- start start 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb5e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5eaf0 0x56073fb67190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb676d0 0x56073fb69ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56073fa82f30 con 0x56073fb5e1d0 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56073facc810 con 0x56073fb676d0 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56073fac9d40 con 0x56073fb5eaf0 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb5e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb5e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37270/0 (socket says 172.21.15.116:37270) 2024-03-24T09:24:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb676d0 0x56073fb69ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99259c640 1 -- 172.21.15.116:0/589065902 learned_addr learned my addr 172.21.15.116:0/589065902 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa991d9b640 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5eaf0 0x56073fb67190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 -- 172.21.15.116:0/589065902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5eaf0 msgr2=0x56073fb67190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5eaf0 0x56073fb67190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 -- 172.21.15.116:0/589065902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 msgr2=0x56073fb5e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb5e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 -- 172.21.15.116:0/589065902 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56073fa86ea0 con 0x56073fb676d0 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa99259c640 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb5e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa992d9d640 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb676d0 0x56073fb69ac0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0cbcd0 tx=0x7fa98c09a8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90b5c9735aa4b5de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.405+0000 7fa980ff9640 1 -- 172.21.15.116:0/589065902 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c09f0c0 con 0x56073fb676d0 2024-03-24T09:24:53.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa980ff9640 1 -- 172.21.15.116:0/589065902 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa98c0a3710 con 0x56073fb676d0 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa980ff9640 1 -- 172.21.15.116:0/589065902 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c09a620 con 0x56073fb676d0 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 -- 172.21.15.116:0/589065902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb676d0 msgr2=0x56073fb69ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb676d0 0x56073fb69ac0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0cbcd0 tx=0x7fa98c09a8d0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 -- 172.21.15.116:0/589065902 shutdown_connections 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb676d0 0x56073fb69ac0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5eaf0 0x56073fb67190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/589065902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb5e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 -- 172.21.15.116:0/589065902 >> 172.21.15.116:0/589065902 conn(0x56073fb58900 msgr2=0x56073fb58ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 -- 172.21.15.116:0/589065902 shutdown_connections 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.406+0000 7fa99b7ce740 1 -- 172.21.15.116:0/589065902 wait complete. 2024-03-24T09:24:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 -- start start 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 Processor -- start 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb646d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb64c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb676d0 0x56073fb65150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56073fac5520 con 0x56073fb5e1d0 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56073f798430 con 0x56073fb5eaf0 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56073f754e00 con 0x56073fb676d0 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99b7ce740 10 client.0 _ll_register_callbacks cb 0x56073fa63d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb646d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb646d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37278/0 (socket says 172.21.15.116:37278) 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 -- 172.21.15.116:0/303194487 learned_addr learned my addr 172.21.15.116:0/303194487 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa991d9b640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb64c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa992d9d640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb676d0 0x56073fb65150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 -- 172.21.15.116:0/303194487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb676d0 msgr2=0x56073fb65150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb676d0 0x56073fb65150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 -- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 msgr2=0x56073fb64c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb64c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.407+0000 7fa99259c640 1 -- 172.21.15.116:0/303194487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9941a68e0 con 0x56073fb5e1d0 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa99259c640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb646d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa984009230 tx=0x7fa984009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d03032b8d7fe523 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa979ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa979ffb640 1 -- 172.21.15.116:0/303194487 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9840035f0 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa979ffb640 1 -- 172.21.15.116:0/303194487 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa984047040 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56073f7c47e0 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa979ffb640 1 -- 172.21.15.116:0/303194487 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa984003750 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa99b7ce740 10 client.16419 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56073fac3ba0 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa979ffb640 1 -- 172.21.15.116:0/303194487 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56073f7c47e0 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa979ffb640 1 -- 172.21.15.116:0/303194487 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa9840052f0 con 0x56073fb5e1d0 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.408+0000 7fa99b7ce740 10 client.16419 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa99b7ce740 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa99b7ce740 20 client.16419 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fb79fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa99b7ce740 4 client.16419 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56073f801720 con 0x56073fb77ba0 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa991d9b640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fb79fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa979ffb640 10 client.16419 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.409+0000 7fa991d9b640 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fb79fc0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73e6b0b68d1e3762 server_cookie=81bbff88341cc5ee in_seq=0 out_seq=0 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa979ffb640 1 -- 172.21.15.116:0/303194487 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56073f801720 con 0x56073fb77ba0 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa979ffb640 10 client.16419 handle_command_reply: tid=0 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 client.16419 shutdown 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 msgr2=0x56073fb646d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb646d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa984009230 tx=0x7fa984009260 comp rx=0 tx=0).stop 2024-03-24T09:24:53.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 shutdown_connections 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb676d0 0x56073fb65150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fb79fc0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb64c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa979ffb640 0 client.16419 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/303194487 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb5e1d0 0x56073fb646d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 >> 172.21.15.116:0/303194487 conn(0x56073fb58900 msgr2=0x56073fb57b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 shutdown_connections 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 1 -- 172.21.15.116:0/303194487 wait complete. 2024-03-24T09:24:53.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.410+0000 7fa99b7ce740 20 client.16419 trim_cache size 0 max 16384 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 Processor -- start 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 -- start start 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb72f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb579e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fb7f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56073fa4a430 con 0x56073fb676d0 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56073fb74540 con 0x56073fb5eaf0 2024-03-24T09:24:53.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.442+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56073fb69ac0 con 0x56073fb5e1d0 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fb7f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb72f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa991d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fb7f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37286/0 (socket says 172.21.15.116:37286) 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa992d9d640 1 -- 172.21.15.116:0/605134035 learned_addr learned my addr 172.21.15.116:0/605134035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 -- 172.21.15.116:0/605134035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 msgr2=0x56073fb579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 -- 172.21.15.116:0/605134035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 msgr2=0x56073fb7f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fb7f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 -- 172.21.15.116:0/605134035 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56073fa5fb20 con 0x56073fb5e1d0 2024-03-24T09:24:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa99259c640 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb72f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa9840056d0 tx=0x7fa98402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e02271666a18917 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa97bfff640 1 -- 172.21.15.116:0/605134035 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984004dc0 con 0x56073fb5e1d0 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa97bfff640 1 -- 172.21.15.116:0/605134035 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa98402de90 con 0x56073fb5e1d0 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.443+0000 7fa97bfff640 1 -- 172.21.15.116:0/605134035 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98404e760 con 0x56073fb5e1d0 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 -- 172.21.15.116:0/605134035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 msgr2=0x56073fb72f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb72f00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa9840056d0 tx=0x7fa98402bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 -- 172.21.15.116:0/605134035 shutdown_connections 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fb7f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/605134035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb72f00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 -- 172.21.15.116:0/605134035 >> 172.21.15.116:0/605134035 conn(0x56073fb58900 msgr2=0x56073fb572d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 -- 172.21.15.116:0/605134035 shutdown_connections 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.444+0000 7fa99b7ce740 1 -- 172.21.15.116:0/605134035 wait complete. 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 -- start start 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 Processor -- start 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb6a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb6ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fbdbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56073fbdc430 con 0x56073fb676d0 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56073fbdc570 con 0x56073fb5eaf0 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56073fbdc6b0 con 0x56073fb5e1d0 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99b7ce740 10 client.0 _ll_register_callbacks cb 0x7fa98402c8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:53.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fbdbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb6a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fbdbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37298/0 (socket says 172.21.15.116:37298) 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb6a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:46426/0 (socket says 172.21.15.116:46426) 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 -- 172.21.15.116:0/2048105804 learned_addr learned my addr 172.21.15.116:0/2048105804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa991d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb6ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 -- 172.21.15.116:0/2048105804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 msgr2=0x56073fb6a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb6a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 -- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 msgr2=0x56073fb6ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb6ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa992d9d640 1 -- 172.21.15.116:0/2048105804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56073fac3ba0 con 0x56073fb676d0 2024-03-24T09:24:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.445+0000 7fa99259c640 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb6a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa992d9d640 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fbdbef0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0973d0 tx=0x7fa98c09aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f661f8adda96bb29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa979ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa979ffb640 1 -- 172.21.15.116:0/2048105804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0b8070 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa979ffb640 1 -- 172.21.15.116:0/2048105804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa98c09f0c0 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa98c0a3710 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa979ffb640 1 -- 172.21.15.116:0/2048105804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa98c099270 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 10 client.16425 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56073fa85700 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa979ffb640 1 -- 172.21.15.116:0/2048105804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa98c0a3710 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa979ffb640 1 -- 172.21.15.116:0/2048105804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa98c096cc0 con 0x56073fb676d0 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 10 client.16425 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 20 client.16425 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fbdfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 4 client.16425 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.446+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56073f850890 con 0x56073fb77ba0 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99259c640 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fbdfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa979ffb640 10 client.16425 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99259c640 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fbdfd20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a63236adbcaeb3 server_cookie=a6cd00db9eb54a3c in_seq=0 out_seq=0 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa979ffb640 1 -- 172.21.15.116:0/2048105804 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56073f850890 con 0x56073fb77ba0 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa979ffb640 10 client.16425 handle_command_reply: tid=0 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 client.16425 shutdown 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 msgr2=0x56073fbdbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fbdbef0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0973d0 tx=0x7fa98c09aa00 comp rx=0 tx=0).stop 2024-03-24T09:24:53.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 shutdown_connections 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56073fb676d0 0x56073fbdbef0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56073fb77ba0 0x56073fbdfd20 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56073fb5eaf0 0x56073fb6ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 --2- 172.21.15.116:0/2048105804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56073fb5e1d0 0x56073fb6a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 >> 172.21.15.116:0/2048105804 conn(0x56073fb58900 msgr2=0x56073fb57b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa979ffb640 0 client.16425 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 shutdown_connections 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 1 -- 172.21.15.116:0/2048105804 wait complete. 2024-03-24T09:24:53.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.447+0000 7fa99b7ce740 20 client.16425 trim_cache size 0 max 16384 2024-03-24T09:24:53.458 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:24:53.458 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:24:53.458 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:24:53.458 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa97c0a2260 msgr2=0x7fa97c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa97c0a2260 0x7fa97c0a4720 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa984005ab0 tx=0x7fa984005fd0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 msgr2=0x7fa994103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 0x7fa994103070 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00efd0 tx=0x7fa98c00c6a0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 shutdown_connections 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa994106660 0x7fa994103070 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa994100f60 0x7fa994102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa97c0a2260 0x7fa97c0a4720 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 --2- 172.21.15.116:0/3910968512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9941005a0 0x7fa9941025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 >> 172.21.15.116:0/3910968512 conn(0x7fa994077710 msgr2=0x7fa994104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 shutdown_connections 2024-03-24T09:24:53.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.448+0000 7fa998b19640 1 -- 172.21.15.116:0/3910968512 wait complete. 2024-03-24T09:24:53.460 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:53 smithi116 ceph-mon[32207]: pgmap v86: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 705 KiB/s wr, 128 op/s 2024-03-24T09:24:53.461 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:24:53.461 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:24:53.463 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi116 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-24T09:24:53.697 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3dbc45f72e5443bdb48dee0d180b8ca2 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi116 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-24T09:24:53.699 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-24T09:24:53.709 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-24T09:24:53.710 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T09:24:53.710 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:24:53.710 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-24T09:24:53.715 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:53 smithi117 ceph-mon[35788]: pgmap v86: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 705 KiB/s wr, 128 op/s 2024-03-24T09:24:53.715 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:53 smithi143 ceph-mon[35965]: pgmap v86: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 705 KiB/s wr, 128 op/s 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/2292496792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 msgr2=0x7f7c3406a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/2292496792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 0x7f7c3406a260 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c28009ca0 tx=0x7f7c2802c180 comp rx=0 tx=0).stop 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/2292496792 shutdown_connections 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/2292496792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3406a7a0 0x7f7c3410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/2292496792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 0x7f7c3406a260 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/2292496792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c34107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/2292496792 >> 172.21.15.116:0/2292496792 conn(0x7f7c340776b0 msgr2=0x7f7c34077ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/2292496792 shutdown_connections 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/2292496792 wait complete. 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.604+0000 7f7c3a1a7640 1 Processor -- start 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 -- start start 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 0x7f7c34100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3406a7a0 0x7f7c34100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c341010c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c341a5ff0 con 0x7f7c34107710 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7c341a6160 con 0x7f7c34069e00 2024-03-24T09:24:53.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c3a1a7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7c341a6320 con 0x7f7c3406a7a0 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c341010c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c337fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 0x7f7c34100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c341010c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37310/0 (socket says 172.21.15.116:37310) 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 -- 172.21.15.116:0/3948323767 learned_addr learned my addr 172.21.15.116:0/3948323767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c32ffd640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3406a7a0 0x7f7c34100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 -- 172.21.15.116:0/3948323767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3406a7a0 msgr2=0x7f7c34100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3406a7a0 0x7f7c34100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 -- 172.21.15.116:0/3948323767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 msgr2=0x7f7c34100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 0x7f7c34100640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 -- 172.21.15.116:0/3948323767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c28009900 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.605+0000 7f7c33fff640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c341010c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00e970 tx=0x7f7c2c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.606+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c009720 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.606+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c341a6600 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.606+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7c2c0054d0 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.606+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c010430 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.606+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c341a6ae0 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.607+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c34102e30 con 0x7f7c34107710 2024-03-24T09:24:53.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.608+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7c2c004f10 con 0x7f7c34107710 2024-03-24T09:24:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.608+0000 7f7c30ff9640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7c1c0a2230 0x7f7c1c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.608+0000 7f7c337fe640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7c1c0a2230 0x7f7c1c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.608+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7c2c014070 con 0x7f7c34107710 2024-03-24T09:24:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.609+0000 7f7c337fe640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7c1c0a2230 0x7f7c1c0a46f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009b50 tx=0x7f7c24005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.613+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7c2c097080 con 0x7f7c34107710 2024-03-24T09:24:53.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-24T09:24:53.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.824+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c34109bb0 con 0x7f7c34107710 2024-03-24T09:24:53.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.825+0000 7f7c30ff9640 1 -- 172.21.15.116:0/3948323767 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7f7c2c09a740 con 0x7f7c34107710 2024-03-24T09:24:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:24:53.826 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:24:53.826 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:24:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7c1c0a2230 msgr2=0x7f7c1c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7c1c0a2230 0x7f7c1c0a46f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009b50 tx=0x7f7c24005f90 comp rx=0 tx=0).stop 2024-03-24T09:24:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 msgr2=0x7f7c341010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c341010c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c00e970 tx=0x7f7c2c00ed70 comp rx=0 tx=0).stop 2024-03-24T09:24:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 shutdown_connections 2024-03-24T09:24:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c34107710 0x7f7c341010c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7c1c0a2230 0x7f7c1c0a46f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c3406a7a0 0x7f7c34100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 --2- 172.21.15.116:0/3948323767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c34069e00 0x7f7c34100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 >> 172.21.15.116:0/3948323767 conn(0x7f7c340776b0 msgr2=0x7f7c340ff330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 shutdown_connections 2024-03-24T09:24:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.827+0000 7f7c3a1a7640 1 -- 172.21.15.116:0/3948323767 wait complete. 2024-03-24T09:24:53.840 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:24:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 -- 172.21.15.116:0/2553334603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 msgr2=0x7fa3d0100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2553334603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0100a20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009900 tx=0x7fa3cc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 -- 172.21.15.116:0/2553334603 shutdown_connections 2024-03-24T09:24:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2553334603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0100f60 0x7fa3d010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2553334603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0100a20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2553334603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3d0106660 0x7fa3d0106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.984+0000 7fa3d7607640 1 -- 172.21.15.116:0/2553334603 >> 172.21.15.116:0/2553334603 conn(0x7fa3d0077710 msgr2=0x7fa3d0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 -- 172.21.15.116:0/2553334603 shutdown_connections 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 -- 172.21.15.116:0/2553334603 wait complete. 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 Processor -- start 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 -- start start 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0100f60 0x7fa3d0198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3d0106660 0x7fa3d01a3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d01997a0 con 0x7fa3d01005a0 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d0199910 con 0x7fa3d0106660 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d7607640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d01a4090 con 0x7fa3d0100f60 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37336/0 (socket says 172.21.15.116:37336) 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d53a3640 1 -- 172.21.15.116:0/2897215221 learned_addr learned my addr 172.21.15.116:0/2897215221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.985+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0100f60 0x7fa3d0198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3d0106660 0x7fa3d01a3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d53a3640 1 -- 172.21.15.116:0/2897215221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0100f60 msgr2=0x7fa3d0198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0100f60 0x7fa3d0198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d53a3640 1 -- 172.21.15.116:0/2897215221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3d0106660 msgr2=0x7fa3d01a3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3d0106660 0x7fa3d01a3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d53a3640 1 -- 172.21.15.116:0/2897215221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c0009490 con 0x7fa3d01005a0 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0198a50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000ea60 tx=0x7fa3c000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3c67fc640 1 -- 172.21.15.116:0/2897215221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000cde0 con 0x7fa3d01005a0 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3c67fc640 1 -- 172.21.15.116:0/2897215221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3c00044d0 con 0x7fa3d01005a0 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3cc009560 con 0x7fa3d01005a0 2024-03-24T09:24:53.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3c67fc640 1 -- 172.21.15.116:0/2897215221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0010690 con 0x7fa3d01005a0 2024-03-24T09:24:53.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.986+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d01a46c0 con 0x7fa3d01005a0 2024-03-24T09:24:53.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.988+0000 7fa3c67fc640 1 -- 172.21.15.116:0/2897215221 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fa3c0003e40 con 0x7fa3d01005a0 2024-03-24T09:24:53.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.988+0000 7fa3da1e9740 1 Processor -- start 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 -- start start 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f916639840 con 0x55f9166cbd10 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f916636d70 con 0x55f9166d48f0 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f916632550 con 0x55f9166cb3f0 2024-03-24T09:24:53.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3c67fc640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3bc0a2230 0x7fa3bc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d5ba4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37338/0 (socket says 172.21.15.116:37338) 2024-03-24T09:24:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 -- 172.21.15.116:0/925348117 learned_addr learned my addr 172.21.15.116:0/925348117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:53.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3bc0a2230 0x7fa3bc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3c67fc640 1 -- 172.21.15.116:0/2897215221 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa3c0014070 con 0x7fa3d01005a0 2024-03-24T09:24:54.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 -- 172.21.15.116:0/925348117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 msgr2=0x55f9166cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 -- 172.21.15.116:0/925348117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 msgr2=0x55f9166d6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 -- 172.21.15.116:0/925348117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9165c7de0 con 0x55f9166cbd10 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d43b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0000c00 tx=0x7fa3c006df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=720e694fd8101e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.989+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3bc0a2230 0x7fa3bc0a46f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005fd0 tx=0x7fa3c8009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3bb7fe640 1 -- 172.21.15.116:0/925348117 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c009eff0 con 0x55f9166cbd10 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3bb7fe640 1 -- 172.21.15.116:0/925348117 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3c00a3640 con 0x55f9166cbd10 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3bb7fe640 1 -- 172.21.15.116:0/925348117 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0097510 con 0x55f9166cbd10 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 -- 172.21.15.116:0/925348117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 msgr2=0x55f9166d43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d43b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0000c00 tx=0x7fa3c006df10 comp rx=0 tx=0).stop 2024-03-24T09:24:54.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 -- 172.21.15.116:0/925348117 shutdown_connections 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d43b0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/925348117 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 -- 172.21.15.116:0/925348117 >> 172.21.15.116:0/925348117 conn(0x55f9166c5b20 msgr2=0x55f9166c5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 -- 172.21.15.116:0/925348117 shutdown_connections 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.990+0000 7fa3da1e9740 1 -- 172.21.15.116:0/925348117 wait complete. 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 -- start start 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 Processor -- start 2024-03-24T09:24:54.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f916305430 con 0x55f9166cbd10 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f9162c1e00 con 0x55f9166d48f0 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f9165b7590 con 0x55f9166cb3f0 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3da1e9740 10 client.0 _ll_register_callbacks cb 0x55f916626fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3d4ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3d4ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37352/0 (socket says 172.21.15.116:37352) 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3d4ba2640 1 -- 172.21.15.116:0/2813903012 learned_addr learned my addr 172.21.15.116:0/2813903012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.991+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3d4ba2640 1 -- 172.21.15.116:0/2813903012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 msgr2=0x55f9166d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3d4ba2640 1 -- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 msgr2=0x55f9166d2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3d4ba2640 1 -- 172.21.15.116:0/2813903012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d01a46c0 con 0x55f9166cbd10 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d1ec0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009900 tx=0x7fa3cc02c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfffe739934ea8f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/2813903012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc02cb40 con 0x55f9166cbd10 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/2813903012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa3cc004250 con 0x55f9166cbd10 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9163317e0 con 0x55f9166cbd10 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/2813903012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa3cc004410 con 0x55f9166cbd10 2024-03-24T09:24:54.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 10 client.16443 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f916630bd0 con 0x55f9166cbd10 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/2813903012 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f9163317e0 con 0x55f9166cbd10 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/2813903012 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa3cc03c040 con 0x55f9166cbd10 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 10 client.16443 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 10 client.16443 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 20 client.16443 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f9166e4ae0 0x55f9166e6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 4 client.16443 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:54.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.992+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f91636e720 con 0x55f9166e4ae0 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.993+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f9166e4ae0 0x55f9166e6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.993+0000 7fa3b8ff9640 10 client.16443 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.993+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f9166e4ae0 0x55f9166e6f00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=37dacc1c9bb25254 server_cookie=b619a20541546e12 in_seq=0 out_seq=0 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.993+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/2813903012 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f91636e720 con 0x55f9166e4ae0 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3b8ff9640 10 client.16443 handle_command_reply: tid=0 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 client.16443 shutdown 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 msgr2=0x55f9166d1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d1ec0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc009900 tx=0x7fa3cc02c9d0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 shutdown_connections 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166d48f0 0x55f9166d2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f9166e4ae0 0x55f9166e6f00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cbd10 0x55f9166d1ec0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3b8ff9640 0 client.16443 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/2813903012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cb3f0 0x55f9166d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 >> 172.21.15.116:0/2813903012 conn(0x55f9166c5b20 msgr2=0x55f9166c7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 shutdown_connections 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 1 -- 172.21.15.116:0/2813903012 wait complete. 2024-03-24T09:24:54.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:53.994+0000 7fa3da1e9740 20 client.16443 trim_cache size 0 max 16384 2024-03-24T09:24:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 Processor -- start 2024-03-24T09:24:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 -- start start 2024-03-24T09:24:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 0x55f9166c0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cbd10 0x55f9166e0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166d48f0 0x55f9166d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9166cb050 con 0x55f9166cb3f0 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f9166c6b20 con 0x55f9166cbd10 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f9166b6e20 con 0x55f9166d48f0 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 0x55f9166c0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 0x55f9166c0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37364/0 (socket says 172.21.15.116:37364) 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 -- 172.21.15.116:0/3594360671 learned_addr learned my addr 172.21.15.116:0/3594360671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cbd10 0x55f9166e0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166d48f0 0x55f9166d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 -- 172.21.15.116:0/3594360671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166d48f0 msgr2=0x55f9166d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166d48f0 0x55f9166d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 -- 172.21.15.116:0/3594360671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cbd10 msgr2=0x55f9166e0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.007+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cbd10 0x55f9166e0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3d53a3640 1 -- 172.21.15.116:0/3594360671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9165ccc80 con 0x55f9166cb3f0 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 0x55f9166c0b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00cc200 tx=0x7fa3c00098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac706bcd5fc6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3baffd640 1 -- 172.21.15.116:0/3594360671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00b7070 con 0x55f9166cb3f0 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3baffd640 1 -- 172.21.15.116:0/3594360671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3c0097140 con 0x55f9166cb3f0 2024-03-24T09:24:54.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3baffd640 1 -- 172.21.15.116:0/3594360671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0003940 con 0x55f9166cb3f0 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 -- 172.21.15.116:0/3594360671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 msgr2=0x55f9166c0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 0x55f9166c0b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00cc200 tx=0x7fa3c00098b0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 -- 172.21.15.116:0/3594360671 shutdown_connections 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166d48f0 0x55f9166d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cbd10 0x55f9166e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/3594360671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166cb3f0 0x55f9166c0b50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 -- 172.21.15.116:0/3594360671 >> 172.21.15.116:0/3594360671 conn(0x55f9166c5b20 msgr2=0x55f9166c77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 -- 172.21.15.116:0/3594360671 shutdown_connections 2024-03-24T09:24:54.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.008+0000 7fa3da1e9740 1 -- 172.21.15.116:0/3594360671 wait complete. 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 -- start start 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 Processor -- start 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cb3f0 0x55f9166d8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cbd10 0x55f9166d8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 0x55f916743e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9167443a0 con 0x55f9166d48f0 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f916744510 con 0x55f9166cb3f0 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f9167446d0 con 0x55f9166cbd10 2024-03-24T09:24:54.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3da1e9740 10 client.0 _ll_register_callbacks cb 0x7fa3cc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 0x55f916743e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 0x55f916743e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37372/0 (socket says 172.21.15.116:37372) 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 -- 172.21.15.116:0/1351400416 learned_addr learned my addr 172.21.15.116:0/1351400416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cb3f0 0x55f9166d8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d4ba2640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cbd10 0x55f9166d8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 -- 172.21.15.116:0/1351400416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cbd10 msgr2=0x55f9166d8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cbd10 0x55f9166d8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 -- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cb3f0 msgr2=0x55f9166d8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cb3f0 0x55f9166d8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d5ba4640 1 -- 172.21.15.116:0/1351400416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9165c7de0 con 0x55f9166d48f0 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.009+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cb3f0 0x55f9166d8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3d5ba4640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 0x55f916743e60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa3c801f940 tx=0x7fa3c8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d9b1f444ff6b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/1351400416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c802e070 con 0x55f9166d48f0 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/1351400416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa3c8004b20 con 0x55f9166d48f0 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3c00a3640 con 0x55f9166d48f0 2024-03-24T09:24:54.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/1351400416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa3c8025040 con 0x55f9166d48f0 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 10 client.16449 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f9165f2920 con 0x55f9166d48f0 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/1351400416 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3c00a3640 con 0x55f9166d48f0 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/1351400416 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fa3c8031030 con 0x55f9166d48f0 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 10 client.16449 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 10 client.16449 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 20 client.16449 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f91674c010 0x55f91674e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 4 client.16449 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.010+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f9163bd890 con 0x55f91674c010 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f91674c010 0x55f91674e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3b8ff9640 10 client.16449 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3d53a3640 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f91674c010 0x55f91674e430 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69c9b7514d1f252b server_cookie=82fbc41e32245c92 in_seq=0 out_seq=0 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3b8ff9640 1 -- 172.21.15.116:0/1351400416 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f9163bd890 con 0x55f91674c010 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3b8ff9640 10 client.16449 handle_command_reply: tid=0 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 client.16449 shutdown 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 msgr2=0x55f916743e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 0x55f916743e60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa3c801f940 tx=0x7fa3c8004ab0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 shutdown_connections 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f91674c010 0x55f91674e430 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f9166d48f0 0x55f916743e60 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3b8ff9640 0 client.16449 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f9166cbd10 0x55f9166d8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 --2- 172.21.15.116:0/1351400416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f9166cb3f0 0x55f9166d8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 >> 172.21.15.116:0/1351400416 conn(0x55f9166c5b20 msgr2=0x55f9166c4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 shutdown_connections 2024-03-24T09:24:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 1 -- 172.21.15.116:0/1351400416 wait complete. 2024-03-24T09:24:54.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.011+0000 7fa3da1e9740 20 client.16449 trim_cache size 0 max 16384 2024-03-24T09:24:54.025 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:54.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3bc0a2230 msgr2=0x7fa3bc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3bc0a2230 0x7fa3bc0a46f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8005fd0 tx=0x7fa3c8009500 comp rx=0 tx=0).stop 2024-03-24T09:24:54.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 msgr2=0x7fa3d0198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0198a50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000ea60 tx=0x7fa3c000ef20 comp rx=0 tx=0).stop 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 shutdown_connections 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3d0106660 0x7fa3d01a3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3bc0a2230 0x7fa3bc0a46f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa3d0100f60 0x7fa3d0198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 --2- 172.21.15.116:0/2897215221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d01005a0 0x7fa3d0198a50 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 >> 172.21.15.116:0/2897215221 conn(0x7fa3d0077710 msgr2=0x7fa3d00ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 shutdown_connections 2024-03-24T09:24:54.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.012+0000 7fa3d7607640 1 -- 172.21.15.116:0/2897215221 wait complete. 2024-03-24T09:24:54.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:24:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.135+0000 7fc774743640 1 -- 172.21.15.116:0/704848488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 msgr2=0x7fc76c107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.135+0000 7fc774743640 1 --2- 172.21.15.116:0/704848488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c107a70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc768009cd0 tx=0x7fc76802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- 172.21.15.116:0/704848488 shutdown_connections 2024-03-24T09:24:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 --2- 172.21.15.116:0/704848488 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c0692b0 0x7fc76c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 --2- 172.21.15.116:0/704848488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c0688f0 0x7fc76c068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 --2- 172.21.15.116:0/704848488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c107a70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- 172.21.15.116:0/704848488 >> 172.21.15.116:0/704848488 conn(0x7fc76c077710 msgr2=0x7fc76c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- 172.21.15.116:0/704848488 shutdown_connections 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- 172.21.15.116:0/704848488 wait complete. 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 Processor -- start 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- start start 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c0688f0 0x7fc76c19d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c0692b0 0x7fc76c19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1980e0 con 0x7fc76c107690 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c198250 con 0x7fc76c0692b0 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc774743640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c198410 con 0x7fc76c0688f0 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc772ce0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc772ce0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37396/0 (socket says 172.21.15.116:37396) 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc772ce0640 1 -- 172.21.15.116:0/109635111 learned_addr learned my addr 172.21.15.116:0/109635111 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc771cde640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c0692b0 0x7fc76c19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.136+0000 7fc7724df640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c0688f0 0x7fc76c19d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc772ce0640 1 -- 172.21.15.116:0/109635111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c0688f0 msgr2=0x7fc76c19d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc772ce0640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c0688f0 0x7fc76c19d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc772ce0640 1 -- 172.21.15.116:0/109635111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c0692b0 msgr2=0x7fc76c19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc772ce0640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c0692b0 0x7fc76c19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc772ce0640 1 -- 172.21.15.116:0/109635111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc768009900 con 0x7fc76c107690 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc772ce0640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c197a90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc76400e970 tx=0x7fc76400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc7637fe640 1 -- 172.21.15.116:0/109635111 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764009720 con 0x7fc76c107690 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c1986f0 con 0x7fc76c107690 2024-03-24T09:24:54.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc7637fe640 1 -- 172.21.15.116:0/109635111 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc7640054d0 con 0x7fc76c107690 2024-03-24T09:24:54.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc7637fe640 1 -- 172.21.15.116:0/109635111 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764010430 con 0x7fc76c107690 2024-03-24T09:24:54.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.137+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c1a4980 con 0x7fc76c107690 2024-03-24T09:24:54.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.139+0000 7fc7637fe640 1 -- 172.21.15.116:0/109635111 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc764004f10 con 0x7fc76c107690 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.139+0000 7fc7773f0740 1 Processor -- start 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 -- start start 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491bb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491c48f0 0x557b491c6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b49129840 con 0x557b491c48f0 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b49126d70 con 0x557b491bb3f0 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7637fe640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7580a2230 0x7fc7580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7773f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b49122550 con 0x557b491bbd10 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc771cde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491c48f0 0x557b491c6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc772ce0640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7580a2230 0x7fc7580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:46500/0 (socket says 172.21.15.116:46500) 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 -- 172.21.15.116:0/731508238 learned_addr learned my addr 172.21.15.116:0/731508238 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc771cde640 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491bb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7637fe640 1 -- 172.21.15.116:0/109635111 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc764014070 con 0x7fc76c107690 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 -- 172.21.15.116:0/731508238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 msgr2=0x557b491bb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491bb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 -- 172.21.15.116:0/731508238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491c48f0 msgr2=0x557b491c6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491c48f0 0x557b491c6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 -- 172.21.15.116:0/731508238 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b490b7de0 con 0x557b491bbd10 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc771cde640 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491bb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc772ce0640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7580a2230 0x7fc7580a46f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc76409aa40 tx=0x7fc764004850 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.140+0000 7fc7724df640 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c43b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc768009ca0 tx=0x7fc768005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81a35d0d11d8dd47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc760ff9640 1 -- 172.21.15.116:0/731508238 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768032c90 con 0x557b491bbd10 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc760ff9640 1 -- 172.21.15.116:0/731508238 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc768032df0 con 0x557b491bbd10 2024-03-24T09:24:54.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc760ff9640 1 -- 172.21.15.116:0/731508238 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768032910 con 0x557b491bbd10 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 -- 172.21.15.116:0/731508238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 msgr2=0x557b491c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c43b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc768009ca0 tx=0x7fc768005f80 comp rx=0 tx=0).stop 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 -- 172.21.15.116:0/731508238 shutdown_connections 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491c48f0 0x557b491c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c43b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 --2- 172.21.15.116:0/731508238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491bb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 -- 172.21.15.116:0/731508238 >> 172.21.15.116:0/731508238 conn(0x557b491b5b20 msgr2=0x557b491b5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.141+0000 7fc7773f0740 1 -- 172.21.15.116:0/731508238 shutdown_connections 2024-03-24T09:24:54.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 -- 172.21.15.116:0/731508238 wait complete. 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 -- start start 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 Processor -- start 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491c1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491c48f0 0x557b491c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b48df5430 con 0x557b491bbd10 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b48db1e00 con 0x557b491bb3f0 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b490a7590 con 0x557b491c48f0 2024-03-24T09:24:54.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7773f0740 10 client.0 _ll_register_callbacks cb 0x557b49116fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491c1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491c1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37416/0 (socket says 172.21.15.116:37416) 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 -- 172.21.15.116:0/3281851555 learned_addr learned my addr 172.21.15.116:0/3281851555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc771cde640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491c48f0 0x557b491c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc771cde640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 -- 172.21.15.116:0/3281851555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491c48f0 msgr2=0x557b491c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491c48f0 0x557b491c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 -- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 msgr2=0x557b491c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc7724df640 1 -- 172.21.15.116:0/3281851555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c10ee50 con 0x557b491bbd10 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.142+0000 7fc771cde640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7724df640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491c1e70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc768000f80 tx=0x7fc768032960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da66f797d292ec0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc755ffb640 1 -- 172.21.15.116:0/3281851555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768032c90 con 0x557b491bbd10 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc755ffb640 1 -- 172.21.15.116:0/3281851555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc768034c70 con 0x557b491bbd10 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557b48e217e0 con 0x557b491bbd10 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc755ffb640 1 -- 172.21.15.116:0/3281851555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc768034790 con 0x557b491bbd10 2024-03-24T09:24:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 10 client.16461 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557b49120bd0 con 0x557b491bbd10 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc755ffb640 1 -- 172.21.15.116:0/3281851555 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x557b48e217e0 con 0x557b491bbd10 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc755ffb640 1 -- 172.21.15.116:0/3281851555 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc7680042b0 con 0x557b491bbd10 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 10 client.16461 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 10 client.16461 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 20 client.16461 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b491d4750 0x557b491d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 4 client.16461 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.143+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b48e5e720 con 0x557b491d4750 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.144+0000 7fc771cde640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b491d4750 0x557b491d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.144+0000 7fc755ffb640 10 client.16461 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.144+0000 7fc771cde640 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b491d4750 0x557b491d6b90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5b43321dc4133af server_cookie=ea1467879ae405e9 in_seq=0 out_seq=0 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.144+0000 7fc755ffb640 1 -- 172.21.15.116:0/3281851555 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b48e5e720 con 0x557b491d4750 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.144+0000 7fc755ffb640 10 client.16461 handle_command_reply: tid=0 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 client.16461 shutdown 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 msgr2=0x557b491c1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491c1e70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc768000f80 tx=0x7fc768032960 comp rx=0 tx=0).stop 2024-03-24T09:24:54.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 shutdown_connections 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b491d4750 0x557b491d6b90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491c48f0 0x557b491c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491c1e70 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc755ffb640 0 client.16461 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 --2- 172.21.15.116:0/3281851555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491bb3f0 0x557b491c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 >> 172.21.15.116:0/3281851555 conn(0x557b491b5b20 msgr2=0x557b491b7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 shutdown_connections 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 1 -- 172.21.15.116:0/3281851555 wait complete. 2024-03-24T09:24:54.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.145+0000 7fc7773f0740 20 client.16461 trim_cache size 0 max 16384 2024-03-24T09:24:54.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 Processor -- start 2024-03-24T09:24:54.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 -- start start 2024-03-24T09:24:54.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bb3f0 0x557b491d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491dc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b491b5650 con 0x557b491bbd10 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b491c1720 con 0x557b491c48f0 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7773f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b491b6b20 con 0x557b491bb3f0 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491b4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37420/0 (socket says 172.21.15.116:37420) 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 -- 172.21.15.116:0/2792560398 learned_addr learned my addr 172.21.15.116:0/2792560398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc771cde640 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bb3f0 0x557b491d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc771cde640 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491dc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 -- 172.21.15.116:0/2792560398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bb3f0 msgr2=0x557b491d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bb3f0 0x557b491d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 -- 172.21.15.116:0/2792560398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 msgr2=0x557b491dc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491dc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 -- 172.21.15.116:0/2792560398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b490bcc80 con 0x557b491bbd10 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc771cde640 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491dc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.158+0000 7fc7724df640 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491b4b40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc76802f020 tx=0x7fc768032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d61d3d12dace5db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc757fff640 1 -- 172.21.15.116:0/2792560398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768032c90 con 0x557b491bbd10 2024-03-24T09:24:54.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc757fff640 1 -- 172.21.15.116:0/2792560398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc768034750 con 0x557b491bbd10 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc757fff640 1 -- 172.21.15.116:0/2792560398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76803fd20 con 0x557b491bbd10 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 -- 172.21.15.116:0/2792560398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 msgr2=0x557b491b4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491b4b40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc76802f020 tx=0x7fc768032fc0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 -- 172.21.15.116:0/2792560398 shutdown_connections 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491dc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bbd10 0x557b491b4b40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2792560398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bb3f0 0x557b491d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 -- 172.21.15.116:0/2792560398 >> 172.21.15.116:0/2792560398 conn(0x557b491b5b20 msgr2=0x557b491b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 -- 172.21.15.116:0/2792560398 shutdown_connections 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 -- 172.21.15.116:0/2792560398 wait complete. 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.159+0000 7fc7773f0740 1 -- start start 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 Processor -- start 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 0x557b491c8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557b49239ad0 con 0x557b491bb3f0 2024-03-24T09:24:54.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557b49239c10 con 0x557b491c48f0 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557b49239d50 con 0x557b491bbd10 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 10 client.0 _ll_register_callbacks cb 0x7fc75c009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 0x557b491c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7724df640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 0x557b491c8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37434/0 (socket says 172.21.15.116:37434) 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 -- 172.21.15.116:0/2622969601 learned_addr learned my addr 172.21.15.116:0/2622969601 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 -- 172.21.15.116:0/2622969601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 msgr2=0x557b491c85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 -- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 msgr2=0x557b491c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 -- 172.21.15.116:0/2622969601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b49120bd0 con 0x557b491bb3f0 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc771cde640 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 0x557b491c8090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000c00 tx=0x7fc75c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3f65e83b79ddbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc755ffb640 1 -- 172.21.15.116:0/2622969601 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c005220 con 0x557b491bb3f0 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc755ffb640 1 -- 172.21.15.116:0/2622969601 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc75c005380 con 0x557b491bb3f0 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.160+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc768032df0 con 0x557b491bb3f0 2024-03-24T09:24:54.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc755ffb640 1 -- 172.21.15.116:0/2622969601 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc75c005540 con 0x557b491bb3f0 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 10 client.16467 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557b490e2920 con 0x557b491bb3f0 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc755ffb640 1 -- 172.21.15.116:0/2622969601 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc75c02d020 con 0x557b491bb3f0 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc755ffb640 1 -- 172.21.15.116:0/2622969601 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc75c021710 con 0x557b491bb3f0 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 10 client.16467 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 10 client.16467 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 20 client.16467 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b4923b5a0 0x557b4923d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 4 client.16467 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b48ead890 con 0x557b4923b5a0 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7724df640 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b4923b5a0 0x557b4923d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc7724df640 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b4923b5a0 0x557b4923d9c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=caf58f34534661b4 server_cookie=f71522fcbd0763b7 in_seq=0 out_seq=0 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.161+0000 7fc755ffb640 10 client.16467 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc755ffb640 1 -- 172.21.15.116:0/2622969601 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b48ead890 con 0x557b4923b5a0 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc755ffb640 10 client.16467 handle_command_reply: tid=0 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 client.16467 shutdown 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 msgr2=0x557b491c8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 0x557b491c8090 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000c00 tx=0x7fc75c005f90 comp rx=0 tx=0).stop 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 shutdown_connections 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x557b4923b5a0 0x557b4923d9c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557b491c48f0 0x557b491c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557b491bbd10 0x557b491c85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 --2- 172.21.15.116:0/2622969601 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557b491bb3f0 0x557b491c8090 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc755ffb640 0 client.16467 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 >> 172.21.15.116:0/2622969601 conn(0x557b491b5b20 msgr2=0x557b491b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 shutdown_connections 2024-03-24T09:24:54.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 1 -- 172.21.15.116:0/2622969601 wait complete. 2024-03-24T09:24:54.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc7773f0740 20 client.16467 trim_cache size 0 max 16384 2024-03-24T09:24:54.173 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:54.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7580a2230 msgr2=0x7fc7580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7580a2230 0x7fc7580a46f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc76409aa40 tx=0x7fc764004850 comp rx=0 tx=0).stop 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 msgr2=0x7fc76c197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c197a90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc76400e970 tx=0x7fc76400ed70 comp rx=0 tx=0).stop 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 shutdown_connections 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c107690 0x7fc76c197a90 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc76c0692b0 0x7fc76c19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7580a2230 0x7fc7580a46f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 --2- 172.21.15.116:0/109635111 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc76c0688f0 0x7fc76c19d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 >> 172.21.15.116:0/109635111 conn(0x7fc76c077710 msgr2=0x7fc76c0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 shutdown_connections 2024-03-24T09:24:54.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.162+0000 7fc774743640 1 -- 172.21.15.116:0/109635111 wait complete. 2024-03-24T09:24:54.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:24:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 -- 172.21.15.116:0/2594715638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f417810bd20 msgr2=0x7f417810e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 --2- 172.21.15.116:0/2594715638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f417810bd20 0x7f417810e150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4170009ca0 tx=0x7f417002c180 comp rx=0 tx=0).stop 2024-03-24T09:24:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 -- 172.21.15.116:0/2594715638 shutdown_connections 2024-03-24T09:24:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 --2- 172.21.15.116:0/2594715638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f417810bd20 0x7f417810e150 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 --2- 172.21.15.116:0/2594715638 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178103270 0x7f417810b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 --2- 172.21.15.116:0/2594715638 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4178102950 0x7f4178102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 -- 172.21.15.116:0/2594715638 >> 172.21.15.116:0/2594715638 conn(0x7f4178076990 msgr2=0x7f4178076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 -- 172.21.15.116:0/2594715638 shutdown_connections 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.279+0000 7f417f68f640 1 -- 172.21.15.116:0/2594715638 wait complete. 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 Processor -- start 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 -- start start 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178102950 0x7f417819d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 0x7f417819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417810bd20 0x7f4178197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41781980f0 con 0x7f4178103270 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4178198260 con 0x7f4178102950 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417f68f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4178198420 con 0x7f417810bd20 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 0x7f417819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 0x7f417819dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37442/0 (socket says 172.21.15.116:37442) 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417d42b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178102950 0x7f417819d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 -- 172.21.15.116:0/2312854266 learned_addr learned my addr 172.21.15.116:0/2312854266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417dc2c640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417810bd20 0x7f4178197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 -- 172.21.15.116:0/2312854266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417810bd20 msgr2=0x7f4178197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417810bd20 0x7f4178197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 -- 172.21.15.116:0/2312854266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178102950 msgr2=0x7f417819d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178102950 0x7f417819d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417cc2a640 1 -- 172.21.15.116:0/2312854266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4170009900 con 0x7f4178103270 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.280+0000 7f417d42b640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178102950 0x7f417819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.281+0000 7f417cc2a640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 0x7f417819dde0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f416800e9f0 tx=0x7f416800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.281+0000 7f416e7fc640 1 -- 172.21.15.116:0/2312854266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168009700 con 0x7f4178103270 2024-03-24T09:24:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.281+0000 7f416e7fc640 1 -- 172.21.15.116:0/2312854266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f416800cd80 con 0x7f4178103270 2024-03-24T09:24:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.281+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4178198700 con 0x7f4178103270 2024-03-24T09:24:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.281+0000 7f416e7fc640 1 -- 172.21.15.116:0/2312854266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168018840 con 0x7f4178103270 2024-03-24T09:24:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.281+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41781a4940 con 0x7f4178103270 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.283+0000 7f416e7fc640 1 -- 172.21.15.116:0/2312854266 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f41680189a0 con 0x7f4178103270 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.283+0000 7f4182279740 1 Processor -- start 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.283+0000 7f4182279740 1 -- start start 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.283+0000 7f4182279740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.283+0000 7f4182279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bcc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f4182279740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bcece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f4182279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6b31840 con 0x55ecd6bc3d10 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f4182279740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6b2ed70 con 0x55ecd6bc33f0 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f4182279740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6b2a550 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f416e7fc640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41640a22b0 0x7f41640a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417d42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bcc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417d42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bcc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37444/0 (socket says 172.21.15.116:37444) 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417d42b640 1 -- 172.21.15.116:0/1997323972 learned_addr learned my addr 172.21.15.116:0/1997323972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417cc2a640 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bcece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41640a22b0 0x7f41640a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f416e7fc640 1 -- 172.21.15.116:0/2312854266 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4168014070 con 0x7f4178103270 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 -- 172.21.15.116:0/1997323972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 msgr2=0x55ecd6bcece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bcece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 -- 172.21.15.116:0/1997323972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc3d10 msgr2=0x55ecd6bcc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bcc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 -- 172.21.15.116:0/1997323972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecd6abfde0 con 0x55ecd6bc33f0 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc37d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f417002c6c0 tx=0x7f4170034650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7a10cf3de377b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f417dc2c640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41640a22b0 0x7f41640a4770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4170032620 tx=0x7f4170049000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f41637fe640 1 -- 172.21.15.116:0/1997323972 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170048070 con 0x55ecd6bc33f0 2024-03-24T09:24:54.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f41637fe640 1 -- 172.21.15.116:0/1997323972 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4170003e00 con 0x55ecd6bc33f0 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.284+0000 7f41637fe640 1 -- 172.21.15.116:0/1997323972 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170004280 con 0x55ecd6bc33f0 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 -- 172.21.15.116:0/1997323972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc33f0 msgr2=0x55ecd6bc37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc37d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f417002c6c0 tx=0x7f4170034650 comp rx=0 tx=0).stop 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 -- 172.21.15.116:0/1997323972 shutdown_connections 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bcece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bcc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 --2- 172.21.15.116:0/1997323972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc37d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 -- 172.21.15.116:0/1997323972 >> 172.21.15.116:0/1997323972 conn(0x55ecd6bbdb20 msgr2=0x55ecd6bbdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 -- 172.21.15.116:0/1997323972 shutdown_connections 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 -- 172.21.15.116:0/1997323972 wait complete. 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.285+0000 7f4182279740 1 -- start start 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 Processor -- start 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bc9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bca470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ecd67fd430 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ecd67b9e00 con 0x55ecd6bc3d10 2024-03-24T09:24:54.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6aaf590 con 0x55ecd6bc33f0 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f4182279740 10 client.0 _ll_register_callbacks cb 0x55ecd6b1efc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417d42b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37452/0 (socket says 172.21.15.116:37452) 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417d42b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:46572/0 (socket says 172.21.15.116:46572) 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417cc2a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bc9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417d42b640 1 -- 172.21.15.116:0/1744720976 learned_addr learned my addr 172.21.15.116:0/1744720976 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 -- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc33f0 msgr2=0x55ecd6bc99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 -- 172.21.15.116:0/1744720976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 msgr2=0x55ecd6bc9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bc9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.286+0000 7f417dc2c640 1 -- 172.21.15.116:0/1744720976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f417810edd0 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f417dc2c640 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bca470 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4170009ca0 tx=0x7f4170004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369a035d983e8cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4160ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4160ff9640 1 -- 172.21.15.116:0/1744720976 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170048040 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4160ff9640 1 -- 172.21.15.116:0/1744720976 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4170032360 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ecd68297e0 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4160ff9640 1 -- 172.21.15.116:0/1744720976 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4170009580 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 10 client.16479 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecd6b28bd0 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4160ff9640 1 -- 172.21.15.116:0/1744720976 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ecd68297e0 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4160ff9640 1 -- 172.21.15.116:0/1744720976 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f4170052070 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 10 client.16479 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 10 client.16479 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 20 client.16479 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6bdc690 0x55ecd6bdea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 4 client.16479 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecd6866720 con 0x55ecd6bdc690 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.287+0000 7f417d42b640 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6bdc690 0x55ecd6bdea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.288+0000 7f4160ff9640 10 client.16479 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.288+0000 7f417d42b640 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6bdc690 0x55ecd6bdea80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c3d799f1f01c6dae server_cookie=2f7c14693850555b in_seq=0 out_seq=0 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.288+0000 7f4160ff9640 1 -- 172.21.15.116:0/1744720976 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ecd6866720 con 0x55ecd6bdc690 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.288+0000 7f4160ff9640 10 client.16479 handle_command_reply: tid=0 2024-03-24T09:24:54.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.288+0000 7f4182279740 1 client.16479 shutdown 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 msgr2=0x55ecd6bca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bca470 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4170009ca0 tx=0x7f4170004df0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 shutdown_connections 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bca470 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6bdc690 0x55ecd6bdea80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bc9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 --2- 172.21.15.116:0/1744720976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bc99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 >> 172.21.15.116:0/1744720976 conn(0x55ecd6bbdb20 msgr2=0x55ecd6bbf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4160ff9640 0 client.16479 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 shutdown_connections 2024-03-24T09:24:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 1 -- 172.21.15.116:0/1744720976 wait complete. 2024-03-24T09:24:54.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.289+0000 7f4182279740 20 client.16479 trim_cache size 0 max 16384 2024-03-24T09:24:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 Processor -- start 2024-03-24T09:24:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 -- start start 2024-03-24T09:24:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bcc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6be4320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6bbeb20 con 0x55ecd6bc33f0 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6bd8660 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f4182279740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6bbd650 con 0x55ecd6bc3d10 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bcc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bcc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37464/0 (socket says 172.21.15.116:37464) 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417dc2c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6be4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 -- 172.21.15.116:0/3601933310 learned_addr learned my addr 172.21.15.116:0/3601933310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417cc2a640 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 -- 172.21.15.116:0/3601933310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc3d10 msgr2=0x55ecd6bd8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 -- 172.21.15.116:0/3601933310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bcc8f0 msgr2=0x55ecd6be4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6be4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 -- 172.21.15.116:0/3601933310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecd6ac4c80 con 0x55ecd6bc33f0 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417dc2c640 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6be4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.303+0000 7f417d42b640 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bcc0b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4174004180 tx=0x7f4174005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d634d813f5dc1dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4162ffd640 1 -- 172.21.15.116:0/3601933310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174017030 con 0x55ecd6bc33f0 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4162ffd640 1 -- 172.21.15.116:0/3601933310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4174018040 con 0x55ecd6bc33f0 2024-03-24T09:24:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4162ffd640 1 -- 172.21.15.116:0/3601933310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174004a90 con 0x55ecd6bc33f0 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 -- 172.21.15.116:0/3601933310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 msgr2=0x55ecd6bcc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bcc0b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4174004180 tx=0x7f4174005f90 comp rx=0 tx=0).stop 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 -- 172.21.15.116:0/3601933310 shutdown_connections 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6be4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 --2- 172.21.15.116:0/3601933310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bcc0b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 -- 172.21.15.116:0/3601933310 >> 172.21.15.116:0/3601933310 conn(0x55ecd6bbdb20 msgr2=0x55ecd6bbc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 -- 172.21.15.116:0/3601933310 shutdown_connections 2024-03-24T09:24:54.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.304+0000 7f4182279740 1 -- 172.21.15.116:0/3601933310 wait complete. 2024-03-24T09:24:54.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 -- start start 2024-03-24T09:24:54.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 Processor -- start 2024-03-24T09:24:54.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bd0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bd0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6c419f0 con 0x55ecd6bc33f0 2024-03-24T09:24:54.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6c41b60 con 0x55ecd6bc3d10 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ecd6c41d20 con 0x55ecd6bcc8f0 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f4182279740 10 client.0 _ll_register_callbacks cb 0x7f4174009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bd0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bd0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37466/0 (socket says 172.21.15.116:37466) 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 -- 172.21.15.116:0/4210692602 learned_addr learned my addr 172.21.15.116:0/4210692602 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417cc2a640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417dc2c640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bd0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 -- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 msgr2=0x55ecd6bd0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bd0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 -- 172.21.15.116:0/4210692602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 msgr2=0x55ecd6bd0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.305+0000 7f417d42b640 1 -- 172.21.15.116:0/4210692602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecd6b28bd0 con 0x55ecd6bc33f0 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f417d42b640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bd0240 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f417400a010 tx=0x7f4174020410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cb2567f6dd5f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4160ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4160ff9640 1 -- 172.21.15.116:0/4210692602 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174017040 con 0x55ecd6bc33f0 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4160ff9640 1 -- 172.21.15.116:0/4210692602 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f417402f070 con 0x55ecd6bc33f0 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4170003e00 con 0x55ecd6bc33f0 2024-03-24T09:24:54.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4160ff9640 1 -- 172.21.15.116:0/4210692602 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4174005220 con 0x55ecd6bc33f0 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 10 client.16485 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f4174005d00 con 0x55ecd6bc33f0 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4160ff9640 1 -- 172.21.15.116:0/4210692602 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4170003e00 con 0x55ecd6bc33f0 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4160ff9640 1 -- 172.21.15.116:0/4210692602 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f4174026020 con 0x55ecd6bc33f0 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 10 client.16485 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 10 client.16485 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 20 client.16485 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6c434d0 0x55ecd6c458f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 4 client.16485 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:54.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ecd68b5890 con 0x55ecd6c434d0 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.306+0000 7f417cc2a640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6c434d0 0x55ecd6c458f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f417cc2a640 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6c434d0 0x55ecd6c458f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=854ab12ac0424a0d server_cookie=198bb8ad61e48682 in_seq=0 out_seq=0 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4160ff9640 10 client.16485 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4160ff9640 1 -- 172.21.15.116:0/4210692602 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ecd68b5890 con 0x55ecd6c434d0 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4160ff9640 10 client.16485 handle_command_reply: tid=0 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 client.16485 shutdown 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 msgr2=0x55ecd6bd0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bd0240 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f417400a010 tx=0x7f4174020410 comp rx=0 tx=0).stop 2024-03-24T09:24:54.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 shutdown_connections 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ecd6bcc8f0 0x55ecd6bd0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ecd6c434d0 0x55ecd6c458f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ecd6bc3d10 0x55ecd6bd0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 --2- 172.21.15.116:0/4210692602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ecd6bc33f0 0x55ecd6bd0240 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4160ff9640 0 client.16485 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 >> 172.21.15.116:0/4210692602 conn(0x55ecd6bbdb20 msgr2=0x55ecd6bbccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 shutdown_connections 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 1 -- 172.21.15.116:0/4210692602 wait complete. 2024-03-24T09:24:54.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.307+0000 7f4182279740 20 client.16485 trim_cache size 0 max 16384 2024-03-24T09:24:54.318 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41640a22b0 msgr2=0x7f41640a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41640a22b0 0x7f41640a4770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4170032620 tx=0x7f4170049000 comp rx=0 tx=0).stop 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 msgr2=0x7f417819dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 0x7f417819dde0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f416800e9f0 tx=0x7f416800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 shutdown_connections 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41640a22b0 0x7f41640a4770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f417810bd20 0x7f4178197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178103270 0x7f417819dde0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 --2- 172.21.15.116:0/2312854266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4178102950 0x7f417819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 >> 172.21.15.116:0/2312854266 conn(0x7f4178076990 msgr2=0x7f41780fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 shutdown_connections 2024-03-24T09:24:54.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.308+0000 7f417f68f640 1 -- 172.21.15.116:0/2312854266 wait complete. 2024-03-24T09:24:54.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:54 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3948323767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:24:54.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:24:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- 172.21.15.116:0/3255485685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 msgr2=0x7f67d0100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- 172.21.15.116:0/3255485685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0100ab0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009930 tx=0x7f67bc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- 172.21.15.116:0/3255485685 shutdown_connections 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- 172.21.15.116:0/3255485685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67d0100ff0 0x7f67d010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- 172.21.15.116:0/3255485685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0100ab0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- 172.21.15.116:0/3255485685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d0106640 0x7f67d0106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- 172.21.15.116:0/3255485685 >> 172.21.15.116:0/3255485685 conn(0x7f67d0077770 msgr2=0x7f67d0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- 172.21.15.116:0/3255485685 shutdown_connections 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- 172.21.15.116:0/3255485685 wait complete. 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 Processor -- start 2024-03-24T09:24:54.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- start start 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d0100ff0 0x7f67d019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67d0106640 0x7f67d019e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67d019ede0 con 0x7f67d0100630 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f67d019ef50 con 0x7f67d0106640 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.426+0000 7f67d6f93640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f67d019f110 con 0x7f67d0100ff0 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d5530640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67d0106640 0x7f67d019e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37488/0 (socket says 172.21.15.116:37488) 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67cffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d0100ff0 0x7f67d019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1249113119 learned_addr learned my addr 172.21.15.116:0/1249113119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1249113119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d0100ff0 msgr2=0x7f67d019a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d0100ff0 0x7f67d019a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1249113119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67d0106640 msgr2=0x7f67d019e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67d0106640 0x7f67d019e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1249113119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c8009840 con 0x7f67d0100630 2024-03-24T09:24:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0199f20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f67c400e9f0 tx=0x7f67c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67cdffb640 1 -- 172.21.15.116:0/1249113119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4009700 con 0x7f67d0100630 2024-03-24T09:24:54.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc009560 con 0x7f67d0100630 2024-03-24T09:24:54.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67cdffb640 1 -- 172.21.15.116:0/1249113119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67c400ccc0 con 0x7f67d0100630 2024-03-24T09:24:54.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67cdffb640 1 -- 172.21.15.116:0/1249113119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4018790 con 0x7f67d0100630 2024-03-24T09:24:54.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.427+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d01a6a90 con 0x7f67d0100630 2024-03-24T09:24:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.429+0000 7f67cdffb640 1 -- 172.21.15.116:0/1249113119 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f67c4010430 con 0x7f67d0100630 2024-03-24T09:24:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 Processor -- start 2024-03-24T09:24:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 -- start start 2024-03-24T09:24:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cdffb640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67c00a2230 0x7f67c00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294bff77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 0x56294c0003b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294c0008f0 0x56294c002ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56294bf65840 con 0x56294c0008f0 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56294bf62d70 con 0x56294bff7d10 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d9b74740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56294bf5e550 con 0x56294bff73f0 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d5530640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67c00a2230 0x7f67c00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cdffb640 1 -- 172.21.15.116:0/1249113119 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67c4014070 con 0x7f67d0100630 2024-03-24T09:24:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d5530640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294c0008f0 0x56294c002ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 0x56294c0003b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294bff77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d5530640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294c0008f0 0x56294c002ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37490/0 (socket says 172.21.15.116:37490) 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 0x56294c0003b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:39228/0 (socket says 172.21.15.116:39228) 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67d5530640 1 -- 172.21.15.116:0/3048173911 learned_addr learned my addr 172.21.15.116:0/3048173911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cffff640 1 -- 172.21.15.116:0/3048173911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 msgr2=0x56294bff77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cffff640 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294bff77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cffff640 1 -- 172.21.15.116:0/3048173911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294c0008f0 msgr2=0x56294c002ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.430+0000 7f67cffff640 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294c0008f0 0x56294c002ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.431+0000 7f67cffff640 1 -- 172.21.15.116:0/3048173911 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56294bef3de0 con 0x56294bff7d10 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.431+0000 7f67d5530640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67c00a2230 0x7f67c00a46f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f67c8011e90 tx=0x7f67c8011590 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.431+0000 7f67cffff640 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 0x56294c0003b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009230 tx=0x7f67bc005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79d286f671ecebf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.431+0000 7f67baffd640 1 -- 172.21.15.116:0/3048173911 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc004780 con 0x56294bff7d10 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.431+0000 7f67baffd640 1 -- 172.21.15.116:0/3048173911 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67bc02de90 con 0x56294bff7d10 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.431+0000 7f67baffd640 1 -- 172.21.15.116:0/3048173911 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc0035f0 con 0x56294bff7d10 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- 172.21.15.116:0/3048173911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 msgr2=0x56294c0003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 0x56294c0003b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009230 tx=0x7f67bc005d50 comp rx=0 tx=0).stop 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- 172.21.15.116:0/3048173911 shutdown_connections 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294c0008f0 0x56294c002ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294bff7d10 0x56294c0003b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3048173911 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294bff77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- 172.21.15.116:0/3048173911 >> 172.21.15.116:0/3048173911 conn(0x56294bff1b20 msgr2=0x56294bff1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- 172.21.15.116:0/3048173911 shutdown_connections 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- 172.21.15.116:0/3048173911 wait complete. 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- start start 2024-03-24T09:24:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 Processor -- start 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294bffd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294bffde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294bffe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56294bc31430 con 0x56294bff73f0 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56294bbede00 con 0x56294c0008f0 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56294bee3590 con 0x56294bff7d10 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d9b74740 10 client.0 _ll_register_callbacks cb 0x56294bf52fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294bffd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294bffd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37500/0 (socket says 172.21.15.116:37500) 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.432+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1704177964 learned_addr learned my addr 172.21.15.116:0/1704177964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d5530640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294bffe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67cffff640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294bffde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 msgr2=0x56294bffde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294bffde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1704177964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 msgr2=0x56294bffe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294bffe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d4d2f640 1 -- 172.21.15.116:0/1704177964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d010de00 con 0x56294bff73f0 2024-03-24T09:24:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d5530640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294bffe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294bffd920 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f67c40971d0 tx=0x7f67c4010b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6499693d953493b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67affff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67affff640 1 -- 172.21.15.116:0/1704177964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c40a3560 con 0x56294bff73f0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67affff640 1 -- 172.21.15.116:0/1704177964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67c401ae00 con 0x56294bff73f0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56294bc5d7e0 con 0x56294bff73f0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67affff640 1 -- 172.21.15.116:0/1704177964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f67c4097430 con 0x56294bff73f0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d9b74740 10 client.16497 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56294bf5cbd0 con 0x56294bff73f0 2024-03-24T09:24:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67affff640 1 -- 172.21.15.116:0/1704177964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56294bc5d7e0 con 0x56294bff73f0 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.433+0000 7f67affff640 1 -- 172.21.15.116:0/1704177964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f67c40ce3b0 con 0x56294bff73f0 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67d9b74740 10 client.16497 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67d9b74740 10 client.16497 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67d9b74740 20 client.16497 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67d9b74740 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c010960 0x56294c012d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67d9b74740 4 client.16497 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56294bc9a720 con 0x56294c010960 2024-03-24T09:24:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67cffff640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c010960 0x56294c012d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67affff640 10 client.16497 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.434+0000 7f67cffff640 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c010960 0x56294c012d70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73ec10224e8010a0 server_cookie=6b2138dc77b4fb5f in_seq=0 out_seq=0 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67affff640 1 -- 172.21.15.116:0/1704177964 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56294bc9a720 con 0x56294c010960 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67affff640 10 client.16497 handle_command_reply: tid=0 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 client.16497 shutdown 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 msgr2=0x56294bffd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294bffd920 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f67c40971d0 tx=0x7f67c4010b20 comp rx=0 tx=0).stop 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 shutdown_connections 2024-03-24T09:24:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c010960 0x56294c012d70 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294bffe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67affff640 0 client.16497 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294bffde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 --2- 172.21.15.116:0/1704177964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294bffd920 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 >> 172.21.15.116:0/1704177964 conn(0x56294bff1b20 msgr2=0x56294bff3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 shutdown_connections 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 1 -- 172.21.15.116:0/1704177964 wait complete. 2024-03-24T09:24:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.435+0000 7f67d9b74740 20 client.16497 trim_cache size 0 max 16384 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 Processor -- start 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 -- start start 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294c000050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294c018610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56294bfe2e20 con 0x56294bff73f0 2024-03-24T09:24:54.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56294c002ce0 con 0x56294c0008f0 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d9b74740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56294bfec0d0 con 0x56294bff7d10 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294c000050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294c000050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37506/0 (socket says 172.21.15.116:37506) 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d4d2f640 1 -- 172.21.15.116:0/2264918990 learned_addr learned my addr 172.21.15.116:0/2264918990 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67cffff640 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294c018610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.448+0000 7f67d5530640 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d4d2f640 1 -- 172.21.15.116:0/2264918990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 msgr2=0x56294c018610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294c018610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d4d2f640 1 -- 172.21.15.116:0/2264918990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 msgr2=0x56294c008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d4d2f640 1 -- 172.21.15.116:0/2264918990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56294bef8c80 con 0x56294bff73f0 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294c000050 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f67c40cc120 tx=0x7f67c4097110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f74c285574bf39a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67ba7fc640 1 -- 172.21.15.116:0/2264918990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4096e60 con 0x56294bff73f0 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67ba7fc640 1 -- 172.21.15.116:0/2264918990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67c40a3560 con 0x56294bff73f0 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67ba7fc640 1 -- 172.21.15.116:0/2264918990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4010840 con 0x56294bff73f0 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d9b74740 1 -- 172.21.15.116:0/2264918990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 msgr2=0x56294c000050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.449+0000 7f67d9b74740 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294c000050 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f67c40cc120 tx=0x7f67c4097110 comp rx=0 tx=0).stop 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- 172.21.15.116:0/2264918990 shutdown_connections 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c008740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff7d10 0x56294c018610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 --2- 172.21.15.116:0/2264918990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff73f0 0x56294c000050 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- 172.21.15.116:0/2264918990 >> 172.21.15.116:0/2264918990 conn(0x56294bff1b20 msgr2=0x56294bff2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- 172.21.15.116:0/2264918990 shutdown_connections 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- 172.21.15.116:0/2264918990 wait complete. 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- start start 2024-03-24T09:24:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 Processor -- start 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294c004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 0x56294c004560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c004aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56294c075c40 con 0x56294bff7d10 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56294c075d80 con 0x56294c0008f0 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56294c075ec0 con 0x56294bff73f0 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d9b74740 10 client.0 _ll_register_callbacks cb 0x7f67c409a900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 0x56294c004560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 0x56294c004560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37520/0 (socket says 172.21.15.116:37520) 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d4d2f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294c004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67cffff640 1 -- 172.21.15.116:0/3578478308 learned_addr learned my addr 172.21.15.116:0/3578478308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.450+0000 7f67d5530640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c004aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67cffff640 1 -- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 msgr2=0x56294c004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67cffff640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294c004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67cffff640 1 -- 172.21.15.116:0/3578478308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 msgr2=0x56294c004aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67cffff640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c004aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67cffff640 1 -- 172.21.15.116:0/3578478308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56294bf5cbd0 con 0x56294bff7d10 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294c004020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67cffff640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 0x56294c004560 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009230 tx=0x7f67bc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb375bd7523d8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67affff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67affff640 1 -- 172.21.15.116:0/3578478308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc0044d0 con 0x56294bff7d10 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67affff640 1 -- 172.21.15.116:0/3578478308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67bc04e040 con 0x56294bff7d10 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67bc02de90 con 0x56294bff7d10 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67affff640 1 -- 172.21.15.116:0/3578478308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f67bc005850 con 0x56294bff7d10 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67d9b74740 10 client.16503 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56294bf1e920 con 0x56294bff7d10 2024-03-24T09:24:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.451+0000 7f67affff640 1 -- 172.21.15.116:0/3578478308 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67bc02de90 con 0x56294bff7d10 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67affff640 1 -- 172.21.15.116:0/3578478308 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f67bc03c050 con 0x56294bff7d10 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 10 client.16503 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 10 client.16503 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 20 client.16503 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c077430 0x56294c079820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 4 client.16503 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56294bce9890 con 0x56294c077430 2024-03-24T09:24:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c077430 0x56294c079820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d4d2f640 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c077430 0x56294c079820 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b6f304053d50c2f1 server_cookie=f9baadd0b6ec03b6 in_seq=0 out_seq=0 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67affff640 10 client.16503 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67affff640 1 -- 172.21.15.116:0/3578478308 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56294bce9890 con 0x56294c077430 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67affff640 10 client.16503 handle_command_reply: tid=0 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 client.16503 shutdown 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 msgr2=0x56294c004560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 0x56294c004560 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009230 tx=0x7f67bc000bc0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 shutdown_connections 2024-03-24T09:24:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x56294c077430 0x56294c079820 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56294c0008f0 0x56294c004aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56294bff7d10 0x56294c004560 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 --2- 172.21.15.116:0/3578478308 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56294bff73f0 0x56294c004020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67affff640 0 client.16503 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.452+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 >> 172.21.15.116:0/3578478308 conn(0x56294bff1b20 msgr2=0x56294bff3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 shutdown_connections 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d9b74740 1 -- 172.21.15.116:0/3578478308 wait complete. 2024-03-24T09:24:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d9b74740 20 client.16503 trim_cache size 0 max 16384 2024-03-24T09:24:54.464 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67c00a2230 msgr2=0x7f67c00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67c00a2230 0x7f67c00a46f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f67c8011e90 tx=0x7f67c8011590 comp rx=0 tx=0).stop 2024-03-24T09:24:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 msgr2=0x7f67d0199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0199f20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f67c400e9f0 tx=0x7f67c400edf0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 shutdown_connections 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67d0106640 0x7f67d019e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67c00a2230 0x7f67c00a46f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d0100ff0 0x7f67d019a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 --2- 172.21.15.116:0/1249113119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67d0100630 0x7f67d0199f20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 >> 172.21.15.116:0/1249113119 conn(0x7f67d0077770 msgr2=0x7f67d010bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 shutdown_connections 2024-03-24T09:24:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.453+0000 7f67d6f93640 1 -- 172.21.15.116:0/1249113119 wait complete. 2024-03-24T09:24:54.466 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:24:54.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:54 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3948323767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:24:54.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:54 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3948323767' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:24:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 -- 172.21.15.116:0/1578648027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 msgr2=0x7f3e48100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 --2- 172.21.15.116:0/1578648027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e48100a40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009930 tx=0x7f3e3802bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 -- 172.21.15.116:0/1578648027 shutdown_connections 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 --2- 172.21.15.116:0/1578648027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e48100f80 0x7f3e4810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 --2- 172.21.15.116:0/1578648027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e48100a40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 --2- 172.21.15.116:0/1578648027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e48106660 0x7f3e48106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 -- 172.21.15.116:0/1578648027 >> 172.21.15.116:0/1578648027 conn(0x7f3e48076990 msgr2=0x7f3e48076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 -- 172.21.15.116:0/1578648027 shutdown_connections 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.571+0000 7f3e4e817640 1 -- 172.21.15.116:0/1578648027 wait complete. 2024-03-24T09:24:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 Processor -- start 2024-03-24T09:24:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 -- start start 2024-03-24T09:24:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e4819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e48100f80 0x7f3e4819cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e48106660 0x7f3e481969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48196ff0 con 0x7f3e481005c0 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48197160 con 0x7f3e48100f80 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4e817640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48197320 con 0x7f3e48106660 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e4819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e4819c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37542/0 (socket says 172.21.15.116:37542) 2024-03-24T09:24:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 -- 172.21.15.116:0/3784360056 learned_addr learned my addr 172.21.15.116:0/3784360056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e48106660 0x7f3e481969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e477fe640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e48100f80 0x7f3e4819cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 -- 172.21.15.116:0/3784360056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e48106660 msgr2=0x7f3e481969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e48106660 0x7f3e481969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 -- 172.21.15.116:0/3784360056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e48100f80 msgr2=0x7f3e4819cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e48100f80 0x7f3e4819cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.572+0000 7f3e47fff640 1 -- 172.21.15.116:0/3784360056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3c009490 con 0x7f3e481005c0 2024-03-24T09:24:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.573+0000 7f3e47fff640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e4819c7a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00e9c0 tx=0x7f3e3c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.573+0000 7f3e457fa640 1 -- 172.21.15.116:0/3784360056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00cde0 con 0x7f3e481005c0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.573+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e38009560 con 0x7f3e481005c0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.573+0000 7f3e457fa640 1 -- 172.21.15.116:0/3784360056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e3c0108e0 con 0x7f3e481005c0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.573+0000 7f3e457fa640 1 -- 172.21.15.116:0/3784360056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0188e0 con 0x7f3e481005c0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.573+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e481a3660 con 0x7f3e481005c0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.575+0000 7f3e457fa640 1 -- 172.21.15.116:0/3784360056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f3e3c010a50 con 0x7f3e481005c0 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.575+0000 7f3e513f0740 1 Processor -- start 2024-03-24T09:24:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.575+0000 7f3e513f0740 1 -- start start 2024-03-24T09:24:54.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b894ff3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffe5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895000ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e513f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b894f61840 con 0x55b894ff39c0 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e513f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b894f5ed70 con 0x55b894ff30a0 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e513f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b894f5a550 con 0x55b894ffeb00 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e457fa640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e300a2230 0x7f3e300a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffe5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b894ff3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b894ff3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:39290/0 (socket says 172.21.15.116:39290) 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 -- 172.21.15.116:0/3715375407 learned_addr learned my addr 172.21.15.116:0/3715375407 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e477fe640 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895000ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e300a2230 0x7f3e300a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e457fa640 1 -- 172.21.15.116:0/3784360056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3e3c014070 con 0x7f3e481005c0 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 -- 172.21.15.116:0/3715375407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 msgr2=0x55b895000ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895000ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 -- 172.21.15.116:0/3715375407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 msgr2=0x55b894ffe5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffe5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 -- 172.21.15.116:0/3715375407 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b894eefde0 con 0x55b894ff30a0 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e300a2230 0x7f3e300a46f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000fc90 tx=0x7f3e40018000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.576+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b894ff3480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3e40011e00 tx=0x7f3e4000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4793e8aae0d1d0cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e2e7fc640 1 -- 172.21.15.116:0/3715375407 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40011910 con 0x55b894ff30a0 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e2e7fc640 1 -- 172.21.15.116:0/3715375407 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e4000ec50 con 0x55b894ff30a0 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e2e7fc640 1 -- 172.21.15.116:0/3715375407 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40004290 con 0x55b894ff30a0 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 -- 172.21.15.116:0/3715375407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 msgr2=0x55b894ff3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b894ff3480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3e40011e00 tx=0x7f3e4000f790 comp rx=0 tx=0).stop 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 -- 172.21.15.116:0/3715375407 shutdown_connections 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895000ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffe5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 --2- 172.21.15.116:0/3715375407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b894ff3480 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 -- 172.21.15.116:0/3715375407 >> 172.21.15.116:0/3715375407 conn(0x55b894fedb20 msgr2=0x55b894fedf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 -- 172.21.15.116:0/3715375407 shutdown_connections 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.577+0000 7f3e513f0740 1 -- 172.21.15.116:0/3715375407 wait complete. 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 -- start start 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 Processor -- start 2024-03-24T09:24:54.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ffbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ffc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894ffc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b894c2d430 con 0x55b894ff30a0 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b894be9e00 con 0x55b894ff39c0 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b894edf590 con 0x55b894ffeb00 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e513f0740 10 client.0 _ll_register_callbacks cb 0x55b894f4efc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ffbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e477fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ffc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ffbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37566/0 (socket says 172.21.15.116:37566) 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 -- 172.21.15.116:0/2036041661 learned_addr learned my addr 172.21.15.116:0/2036041661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894ffc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 -- 172.21.15.116:0/2036041661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 msgr2=0x55b894ffc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894ffc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 -- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 msgr2=0x55b894ffc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ffc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e47fff640 1 -- 172.21.15.116:0/2036041661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e40019040 con 0x55b894ff30a0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.578+0000 7f3e477fe640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ffc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e47fff640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ffbb40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c000c00 tx=0x7f3e3c0a01d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65cd61f78681a8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e277fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e277fe640 1 -- 172.21.15.116:0/2036041661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c09b5c0 con 0x55b894ff30a0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e277fe640 1 -- 172.21.15.116:0/2036041661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3e3c09b160 con 0x55b894ff30a0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b894c597e0 con 0x55b894ff30a0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e277fe640 1 -- 172.21.15.116:0/2036041661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3e3c09f2b0 con 0x55b894ff30a0 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 10 client.16515 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b894f58bd0 con 0x55b894ff30a0 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e277fe640 1 -- 172.21.15.116:0/2036041661 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b894c597e0 con 0x55b894ff30a0 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e277fe640 1 -- 172.21.15.116:0/2036041661 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f3e3c0b2080 con 0x55b894ff30a0 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 10 client.16515 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 20 client.16515 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b89500ec60 0x55b8950110a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 4 client.16515 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:54.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.579+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b894c96720 con 0x55b89500ec60 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.580+0000 7f3e477fe640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b89500ec60 0x55b8950110a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.580+0000 7f3e277fe640 10 client.16515 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.580+0000 7f3e477fe640 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b89500ec60 0x55b8950110a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8ca9980f7212336 server_cookie=e298270d5bfd7d8 in_seq=0 out_seq=0 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e277fe640 1 -- 172.21.15.116:0/2036041661 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b894c96720 con 0x55b89500ec60 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e277fe640 10 client.16515 handle_command_reply: tid=0 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 client.16515 shutdown 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 msgr2=0x55b894ffbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ffbb40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c000c00 tx=0x7f3e3c0a01d0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 shutdown_connections 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894ffc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b89500ec60 0x55b8950110a0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ffc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2036041661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ffbb40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e277fe640 0 client.16515 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 >> 172.21.15.116:0/2036041661 conn(0x55b894fedb20 msgr2=0x55b894fef870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 shutdown_connections 2024-03-24T09:24:54.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 1 -- 172.21.15.116:0/2036041661 wait complete. 2024-03-24T09:24:54.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.581+0000 7f3e513f0740 20 client.16515 trim_cache size 0 max 16384 2024-03-24T09:24:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 Processor -- start 2024-03-24T09:24:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 -- start start 2024-03-24T09:24:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b89500a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894feccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b894fe9250 con 0x55b894ff39c0 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b89500e960 con 0x55b894ff30a0 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e513f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b894ff2770 con 0x55b894ffeb00 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.594+0000 7f3e477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37580/0 (socket says 172.21.15.116:37580) 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 -- 172.21.15.116:0/146036852 learned_addr learned my addr 172.21.15.116:0/146036852 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e47fff640 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b89500a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894feccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 -- 172.21.15.116:0/146036852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 msgr2=0x55b894feccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894feccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 -- 172.21.15.116:0/146036852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 msgr2=0x55b89500a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b89500a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 -- 172.21.15.116:0/146036852 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b894ef4c80 con 0x55b894ff39c0 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e477fe640 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffc3e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009900 tx=0x7f3e38009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7244308decb7297 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e2dffb640 1 -- 172.21.15.116:0/146036852 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3802c910 con 0x55b894ff39c0 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e2dffb640 1 -- 172.21.15.116:0/146036852 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e3802de90 con 0x55b894ff39c0 2024-03-24T09:24:54.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.595+0000 7f3e2dffb640 1 -- 172.21.15.116:0/146036852 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3804a330 con 0x55b894ff39c0 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- 172.21.15.116:0/146036852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 msgr2=0x55b894ffc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffc3e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009900 tx=0x7f3e38009260 comp rx=0 tx=0).stop 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- 172.21.15.116:0/146036852 shutdown_connections 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b894feccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff39c0 0x55b894ffc3e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- 172.21.15.116:0/146036852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff30a0 0x55b89500a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- 172.21.15.116:0/146036852 >> 172.21.15.116:0/146036852 conn(0x55b894fedb20 msgr2=0x55b894fedf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- 172.21.15.116:0/146036852 shutdown_connections 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- 172.21.15.116:0/146036852 wait complete. 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- start start 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 Processor -- start 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ff8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ff93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895073fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b895074520 con 0x55b894ff30a0 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b895074660 con 0x55b894ff39c0 2024-03-24T09:24:54.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b8950747a0 con 0x55b894ffeb00 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e513f0740 10 client.0 _ll_register_callbacks cb 0x7f3e3c096d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.596+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ff8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ff8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37594/0 (socket says 172.21.15.116:37594) 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 -- 172.21.15.116:0/2210477668 learned_addr learned my addr 172.21.15.116:0/2210477668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e4cdb4640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895073fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e477fe640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ff93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 -- 172.21.15.116:0/2210477668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 msgr2=0x55b895073fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895073fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 -- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 msgr2=0x55b894ff93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ff93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 -- 172.21.15.116:0/2210477668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4810dd90 con 0x55b894ff30a0 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e47fff640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ff8eb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0a33d0 tx=0x7f3e3c010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b32f1bc6e6fc0ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e277fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e277fe640 1 -- 172.21.15.116:0/2210477668 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0a3720 con 0x55b894ff30a0 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e277fe640 1 -- 172.21.15.116:0/2210477668 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3e3c010de0 con 0x55b894ff30a0 2024-03-24T09:24:54.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e4000ec50 con 0x55b894ff30a0 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e277fe640 1 -- 172.21.15.116:0/2210477668 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3e3c0a3470 con 0x55b894ff30a0 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 10 client.16521 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b894f1a920 con 0x55b894ff30a0 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e277fe640 1 -- 172.21.15.116:0/2210477668 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e4000ec50 con 0x55b894ff30a0 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e277fe640 1 -- 172.21.15.116:0/2210477668 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f3e3c09a020 con 0x55b894ff30a0 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 10 client.16521 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 20 client.16521 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.597+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b895075e80 0x55b8950782a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 4 client.16521 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b894ce5890 con 0x55b895075e80 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e477fe640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b895075e80 0x55b8950782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e277fe640 10 client.16521 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e477fe640 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b895075e80 0x55b8950782a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3afbe7def478e343 server_cookie=bea9c87d7eaefd8a in_seq=0 out_seq=0 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e277fe640 1 -- 172.21.15.116:0/2210477668 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b894ce5890 con 0x55b895075e80 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e277fe640 10 client.16521 handle_command_reply: tid=0 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 client.16521 shutdown 2024-03-24T09:24:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 msgr2=0x55b894ff8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ff8eb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0a33d0 tx=0x7f3e3c010660 comp rx=0 tx=0).stop 2024-03-24T09:24:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 shutdown_connections 2024-03-24T09:24:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b894ffeb00 0x55b895073fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b895075e80 0x55b8950782a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b894ff39c0 0x55b894ff93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 --2- 172.21.15.116:0/2210477668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b894ff30a0 0x55b894ff8eb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e277fe640 0 client.16521 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 >> 172.21.15.116:0/2210477668 conn(0x55b894fedb20 msgr2=0x55b894feccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 shutdown_connections 2024-03-24T09:24:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.598+0000 7f3e513f0740 1 -- 172.21.15.116:0/2210477668 wait complete. 2024-03-24T09:24:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e513f0740 20 client.16521 trim_cache size 0 max 16384 2024-03-24T09:24:54.614 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:24:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e300a2230 msgr2=0x7f3e300a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e300a2230 0x7f3e300a46f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000fc90 tx=0x7f3e40018000 comp rx=0 tx=0).stop 2024-03-24T09:24:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 msgr2=0x7f3e4819c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e4819c7a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00e9c0 tx=0x7f3e3c00ee80 comp rx=0 tx=0).stop 2024-03-24T09:24:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 shutdown_connections 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e300a2230 0x7f3e300a46f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e48106660 0x7f3e481969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e48100f80 0x7f3e4819cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 --2- 172.21.15.116:0/3784360056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e481005c0 0x7f3e4819c7a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 >> 172.21.15.116:0/3784360056 conn(0x7f3e48076990 msgr2=0x7f3e480fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 shutdown_connections 2024-03-24T09:24:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:54.599+0000 7f3e4e817640 1 -- 172.21.15.116:0/3784360056 wait complete. 2024-03-24T09:24:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:55 smithi143 ceph-mon[35965]: pgmap v87: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 328 KiB/s wr, 77 op/s 2024-03-24T09:24:55.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:55 smithi117 ceph-mon[35788]: pgmap v87: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 328 KiB/s wr, 77 op/s 2024-03-24T09:24:55.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:55 smithi116 ceph-mon[32207]: pgmap v87: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 328 KiB/s wr, 77 op/s 2024-03-24T09:24:55.617 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:24:55.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:24:55.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:24:55.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 -- 172.21.15.116:0/1043437685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0101680 msgr2=0x7f5bd0101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/1043437685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0101680 0x7f5bd0101b00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009930 tx=0x7f5bb802bc30 comp rx=0 tx=0).stop 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 -- 172.21.15.116:0/1043437685 shutdown_connections 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/1043437685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bd0102040 0x7f5bd010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/1043437685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0101680 0x7f5bd0101b00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/1043437685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 0x7f5bd0107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 -- 172.21.15.116:0/1043437685 >> 172.21.15.116:0/1043437685 conn(0x7f5bd00fd170 msgr2=0x7f5bd00ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 -- 172.21.15.116:0/1043437685 shutdown_connections 2024-03-24T09:24:55.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.736+0000 7f5bd4da5640 1 -- 172.21.15.116:0/1043437685 wait complete. 2024-03-24T09:24:55.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 Processor -- start 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 -- start start 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bd0101680 0x7f5bd019da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 0x7f5bd019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 0x7f5bd0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01982a0 con 0x7f5bd0102040 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd0198410 con 0x7f5bd0107690 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bd4da5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd01985d0 con 0x7f5bd0101680 2024-03-24T09:24:55.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 0x7f5bd019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 0x7f5bd019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37604/0 (socket says 172.21.15.116:37604) 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 -- 172.21.15.116:0/3931580341 learned_addr learned my addr 172.21.15.116:0/3931580341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bced9d640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 0x7f5bd0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bce59c640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bd0101680 0x7f5bd019da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 -- 172.21.15.116:0/3931580341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bd0101680 msgr2=0x7f5bd019da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bd0101680 0x7f5bd019da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 -- 172.21.15.116:0/3931580341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 msgr2=0x7f5bd0197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 0x7f5bd0197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bc7fff640 1 -- 172.21.15.116:0/3931580341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb8009560 con 0x7f5bd0102040 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.737+0000 7f5bced9d640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 0x7f5bd0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.738+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 0x7f5bd019df90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009230 tx=0x7f5bb8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.738+0000 7f5bc77fe640 1 -- 172.21.15.116:0/3931580341 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8004350 con 0x7f5bd0102040 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.738+0000 7f5bc77fe640 1 -- 172.21.15.116:0/3931580341 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bb802de90 con 0x7f5bd0102040 2024-03-24T09:24:55.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.738+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd0198850 con 0x7f5bd0102040 2024-03-24T09:24:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.738+0000 7f5bc77fe640 1 -- 172.21.15.116:0/3931580341 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8005220 con 0x7f5bd0102040 2024-03-24T09:24:55.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.738+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd01a49d0 con 0x7f5bd0102040 2024-03-24T09:24:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.740+0000 7f5bc77fe640 1 -- 172.21.15.116:0/3931580341 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f5bb8003c30 con 0x7f5bd0102040 2024-03-24T09:24:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.740+0000 7f5bd7983740 1 Processor -- start 2024-03-24T09:24:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.740+0000 7f5bd7983740 1 -- start start 2024-03-24T09:24:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fe410 0x56532c1fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc77fe640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5bbc0a2230 0x7f5bbc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 0x56532c20bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c20c070 0x56532c20e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bd7983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56532c16c530 con 0x56532c1fed30 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bd7983740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56532c169a60 con 0x56532c20c070 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bd7983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56532c165240 con 0x56532c1fe410 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 0x56532c20bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bced9d640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5bbc0a2230 0x7f5bbc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 0x56532c20bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37612/0 (socket says 172.21.15.116:37612) 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc7fff640 1 -- 172.21.15.116:0/2867114062 learned_addr learned my addr 172.21.15.116:0/2867114062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fe410 0x56532c1fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bced9d640 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c20c070 0x56532c20e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc77fe640 1 -- 172.21.15.116:0/3931580341 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5bb80f0280 con 0x7f5bd0102040 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 -- 172.21.15.116:0/2867114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c20c070 msgr2=0x56532c20e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c20c070 0x56532c20e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 -- 172.21.15.116:0/2867114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 msgr2=0x56532c20bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 0x56532c20bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 -- 172.21.15.116:0/2867114062 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56532c1638c0 con 0x56532c1fe410 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 0x56532c20bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bced9d640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5bbc0a2230 0x7f5bbc0a46f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80099e0 tx=0x7f5bc8013660 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bce59c640 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fe410 0x56532c1fe7f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009c20 tx=0x7f5bc0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=562c5c8e9c310bae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.741+0000 7f5bc4ff9640 1 -- 172.21.15.116:0/2867114062 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0004050 con 0x56532c1fe410 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bc4ff9640 1 -- 172.21.15.116:0/2867114062 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc00035f0 con 0x56532c1fe410 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bc4ff9640 1 -- 172.21.15.116:0/2867114062 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0019620 con 0x56532c1fe410 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 -- 172.21.15.116:0/2867114062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fe410 msgr2=0x56532c1fe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fe410 0x56532c1fe7f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0009c20 tx=0x7f5bc0011810 comp rx=0 tx=0).stop 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 -- 172.21.15.116:0/2867114062 shutdown_connections 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c20c070 0x56532c20e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fed30 0x56532c20bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 --2- 172.21.15.116:0/2867114062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fe410 0x56532c1fe7f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 -- 172.21.15.116:0/2867114062 >> 172.21.15.116:0/2867114062 conn(0x56532c1f8e90 msgr2=0x56532c1f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 -- 172.21.15.116:0/2867114062 shutdown_connections 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.742+0000 7f5bd7983740 1 -- 172.21.15.116:0/2867114062 wait complete. 2024-03-24T09:24:55.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 -- start start 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 Processor -- start 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c208f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c2094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c2099e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56532c109f80 con 0x56532c20c070 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56532be384b0 con 0x56532c1fe410 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56532c0eaad0 con 0x56532c1fed30 2024-03-24T09:24:55.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bd7983740 10 client.0 _ll_register_callbacks cb 0x56532c11db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c2099e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bc7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c2094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c2099e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37618/0 (socket says 172.21.15.116:37618) 2024-03-24T09:24:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 -- 172.21.15.116:0/640343584 learned_addr learned my addr 172.21.15.116:0/640343584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bce59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c208f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 -- 172.21.15.116:0/640343584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 msgr2=0x56532c2094a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c2094a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 -- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 msgr2=0x56532c208f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c208f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bced9d640 1 -- 172.21.15.116:0/640343584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc0009850 con 0x56532c20c070 2024-03-24T09:24:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.743+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c2094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bced9d640 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c2099e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8013bc0 tx=0x7f5bc801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a0719b72841fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:55.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/640343584 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc8013170 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/640343584 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5bc8025030 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56532c1897a0 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/640343584 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bc80132d0 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 10 client.16533 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56532c11b6d0 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/640343584 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56532c1897a0 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/640343584 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f5bc801f540 con 0x56532c20c070 2024-03-24T09:24:55.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 10 client.16533 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 20 client.16533 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c21c250 0x56532c21e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 4 client.16533 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:55.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.744+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56532c15b1f0 con 0x56532c21c250 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.745+0000 7f5bce59c640 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c21c250 0x56532c21e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.745+0000 7f5bce59c640 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c21c250 0x56532c21e670 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25bcaf80f65a684c server_cookie=2545665ac9708f30 in_seq=0 out_seq=0 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.745+0000 7f5bb5ffb640 10 client.16533 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/640343584 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56532c15b1f0 con 0x56532c21c250 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bb5ffb640 10 client.16533 handle_command_reply: tid=0 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 client.16533 shutdown 2024-03-24T09:24:55.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 msgr2=0x56532c2099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c2099e0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8013bc0 tx=0x7f5bc801d000 comp rx=0 tx=0).stop 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 shutdown_connections 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c2099e0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c2094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c21c250 0x56532c21e670 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 --2- 172.21.15.116:0/640343584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c208f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bb5ffb640 0 client.16533 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 >> 172.21.15.116:0/640343584 conn(0x56532c1f8e90 msgr2=0x56532c1fabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 shutdown_connections 2024-03-24T09:24:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 1 -- 172.21.15.116:0/640343584 wait complete. 2024-03-24T09:24:55.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.746+0000 7f5bd7983740 20 client.16533 trim_cache size 0 max 16384 2024-03-24T09:24:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.763+0000 7f5bd7983740 1 Processor -- start 2024-03-24T09:24:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.763+0000 7f5bd7983740 1 -- start start 2024-03-24T09:24:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.763+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c20b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.763+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c209800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c223f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bd7983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56532c1f4620 con 0x56532c20c070 2024-03-24T09:24:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bd7983740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56532c20be90 con 0x56532c1fe410 2024-03-24T09:24:55.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bd7983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56532c1fd850 con 0x56532c1fed30 2024-03-24T09:24:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c20b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c223f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c20b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:39342/0 (socket says 172.21.15.116:39342) 2024-03-24T09:24:55.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 -- 172.21.15.116:0/1272666855 learned_addr learned my addr 172.21.15.116:0/1272666855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bced9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c223f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37632/0 (socket says 172.21.15.116:37632) 2024-03-24T09:24:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c209800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 -- 172.21.15.116:0/1272666855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 msgr2=0x56532c209800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c209800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 -- 172.21.15.116:0/1272666855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 msgr2=0x56532c223f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c223f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 -- 172.21.15.116:0/1272666855 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56532c0ffc60 con 0x56532c1fe410 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c209800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bced9d640 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c223f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bce59c640 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c20b690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000a010 tx=0x7f5bc0005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d0c984637923275 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bb7fff640 1 -- 172.21.15.116:0/1272666855 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0003a60 con 0x56532c1fe410 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bb7fff640 1 -- 172.21.15.116:0/1272666855 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc00054f0 con 0x56532c1fe410 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.764+0000 7f5bb7fff640 1 -- 172.21.15.116:0/1272666855 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0019a70 con 0x56532c1fe410 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- 172.21.15.116:0/1272666855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 msgr2=0x56532c20b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c20b690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000a010 tx=0x7f5bc0005f70 comp rx=0 tx=0).stop 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- 172.21.15.116:0/1272666855 shutdown_connections 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c20c070 0x56532c223f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c1fed30 0x56532c209800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- 172.21.15.116:0/1272666855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fe410 0x56532c20b690 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- 172.21.15.116:0/1272666855 >> 172.21.15.116:0/1272666855 conn(0x56532c1f8e90 msgr2=0x56532c1f9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- 172.21.15.116:0/1272666855 shutdown_connections 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- 172.21.15.116:0/1272666855 wait complete. 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- start start 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 Processor -- start 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 0x56532c210250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fed30 0x56532c210790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c20c070 0x56532c27b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56532c27bb30 con 0x56532c1fe410 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56532c27bc70 con 0x56532c1fed30 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56532c27be30 con 0x56532c20c070 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.765+0000 7f5bd7983740 10 client.0 _ll_register_callbacks cb 0x7f5bc8003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 0x56532c210250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 0x56532c210250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37648/0 (socket says 172.21.15.116:37648) 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 -- 172.21.15.116:0/3618799438 learned_addr learned my addr 172.21.15.116:0/3618799438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fed30 0x56532c210790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bced9d640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c20c070 0x56532c27b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 -- 172.21.15.116:0/3618799438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c20c070 msgr2=0x56532c27b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c20c070 0x56532c27b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 -- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fed30 msgr2=0x56532c210790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fed30 0x56532c210790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 -- 172.21.15.116:0/3618799438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56532c11b6d0 con 0x56532c1fe410 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bce59c640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 0x56532c210250 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0006010 tx=0x7f5bc0009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6620df6657d3767c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:55.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/3618799438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00194e0 con 0x56532c1fe410 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/3618799438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5bc0003860 con 0x56532c1fe410 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bc00035f0 con 0x56532c1fe410 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/3618799438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5bc0003a20 con 0x56532c1fe410 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bd7983740 10 client.16539 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56532c1638c0 con 0x56532c1fe410 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.766+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/3618799438 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5bc00035f0 con 0x56532c1fe410 2024-03-24T09:24:55.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/3618799438 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f5bc003e070 con 0x56532c1fe410 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bd7983740 10 client.16539 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bd7983740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bd7983740 20 client.16539 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bd7983740 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c2834b0 0x56532c2858d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bd7983740 4 client.16539 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56532be64860 con 0x56532c2834b0 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c2834b0 0x56532c2858d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bc7fff640 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c2834b0 0x56532c2858d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32e92527056869cb server_cookie=902b45dc514466aa in_seq=0 out_seq=0 2024-03-24T09:24:55.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.767+0000 7f5bb5ffb640 10 client.16539 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bb5ffb640 1 -- 172.21.15.116:0/3618799438 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56532be64860 con 0x56532c2834b0 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bb5ffb640 10 client.16539 handle_command_reply: tid=0 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 client.16539 shutdown 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 msgr2=0x56532c210250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 0x56532c210250 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f5bc0006010 tx=0x7f5bc0009ee0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 shutdown_connections 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56532c20c070 0x56532c27b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56532c2834b0 0x56532c2858d0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56532c1fed30 0x56532c210790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 --2- 172.21.15.116:0/3618799438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56532c1fe410 0x56532c210250 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bb5ffb640 0 client.16539 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 >> 172.21.15.116:0/3618799438 conn(0x56532c1f8e90 msgr2=0x56532c1fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 shutdown_connections 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.768+0000 7f5bd7983740 1 -- 172.21.15.116:0/3618799438 wait complete. 2024-03-24T09:24:55.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd7983740 20 client.16539 trim_cache size 0 max 16384 2024-03-24T09:24:55.782 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:24:55.782 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:24:55.782 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "d2a13b53-12e4-4085-b06c-1e7efeefb0b7", 2024-03-24T09:24:55.782 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:24:55.782 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:24:55.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5bbc0a2230 msgr2=0x7f5bbc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5bbc0a2230 0x7f5bbc0a46f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80099e0 tx=0x7f5bc8013660 comp rx=0 tx=0).stop 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 msgr2=0x7f5bd019df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 0x7f5bd019df90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8009230 tx=0x7f5bb8005dd0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 shutdown_connections 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bd0107690 0x7f5bd0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd0102040 0x7f5bd019df90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5bbc0a2230 0x7f5bbc0a46f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 --2- 172.21.15.116:0/3931580341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5bd0101680 0x7f5bd019da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 >> 172.21.15.116:0/3931580341 conn(0x7f5bd00fd170 msgr2=0x7f5bd00fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 shutdown_connections 2024-03-24T09:24:55.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.769+0000 7f5bd4da5640 1 -- 172.21.15.116:0/3931580341 wait complete. 2024-03-24T09:24:55.785 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:24:55.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:24:55.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 -- 172.21.15.116:0/550470809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 msgr2=0x7f6188068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 --2- 172.21.15.116:0/550470809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 0x7f6188068d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6178009860 tx=0x7f617802bb70 comp rx=0 tx=0).stop 2024-03-24T09:24:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 -- 172.21.15.116:0/550470809 shutdown_connections 2024-03-24T09:24:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 --2- 172.21.15.116:0/550470809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61880692b0 0x7f618810e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 --2- 172.21.15.116:0/550470809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 0x7f6188068d70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 --2- 172.21.15.116:0/550470809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f6188107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 -- 172.21.15.116:0/550470809 >> 172.21.15.116:0/550470809 conn(0x7f6188077710 msgr2=0x7f6188077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 -- 172.21.15.116:0/550470809 shutdown_connections 2024-03-24T09:24:55.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.906+0000 7f618caae640 1 -- 172.21.15.116:0/550470809 wait complete. 2024-03-24T09:24:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 Processor -- start 2024-03-24T09:24:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 -- start start 2024-03-24T09:24:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 0x7f6188199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61880692b0 0x7f618819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f618819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f618819ecf0 con 0x7f6188107690 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f618819ee60 con 0x7f61880688f0 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618caae640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f618819f020 con 0x7f61880692b0 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f6186d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f618819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f6186d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f618819e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37666/0 (socket says 172.21.15.116:37666) 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f6186d9d640 1 -- 172.21.15.116:0/414373508 learned_addr learned my addr 172.21.15.116:0/414373508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.907+0000 7f618659c640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 0x7f6188199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6185d9b640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61880692b0 0x7f618819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6186d9d640 1 -- 172.21.15.116:0/414373508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61880692b0 msgr2=0x7f618819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6186d9d640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61880692b0 0x7f618819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6186d9d640 1 -- 172.21.15.116:0/414373508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 msgr2=0x7f6188199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6186d9d640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 0x7f6188199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6186d9d640 1 -- 172.21.15.116:0/414373508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c009560 con 0x7f6188107690 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f6186d9d640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f618819e6f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f618000eaa0 tx=0x7f618000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f61777fe640 1 -- 172.21.15.116:0/414373508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618000cdb0 con 0x7f6188107690 2024-03-24T09:24:55.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f61777fe640 1 -- 172.21.15.116:0/414373508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6180005020 con 0x7f6188107690 2024-03-24T09:24:55.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.908+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6178009490 con 0x7f6188107690 2024-03-24T09:24:55.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.909+0000 7f61777fe640 1 -- 172.21.15.116:0/414373508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61800116b0 con 0x7f6188107690 2024-03-24T09:24:55.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.909+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61881a69a0 con 0x7f6188107690 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.911+0000 7f61777fe640 1 -- 172.21.15.116:0/414373508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6180011810 con 0x7f6188107690 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.911+0000 7f618f76d740 1 Processor -- start 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 -- start start 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f61777fe640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61700a22b0 0x7f61700a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b2365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b23f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b23f6d0 0x55a43b241ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a43b15af30 con 0x55a43b236af0 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a43b1a4810 con 0x55a43b2361d0 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f618f76d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a43b1a1d40 con 0x55a43b23f6d0 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f6185d9b640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61700a22b0 0x7f61700a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f61777fe640 1 -- 172.21.15.116:0/414373508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f61800ce4f0 con 0x7f6188107690 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b23f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b23f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37680/0 (socket says 172.21.15.116:37680) 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.912+0000 7f6185d9b640 1 -- 172.21.15.116:0/2731261805 learned_addr learned my addr 172.21.15.116:0/2731261805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f618659c640 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b2365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6186d9d640 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b23f6d0 0x55a43b241ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 -- 172.21.15.116:0/2731261805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b23f6d0 msgr2=0x55a43b241ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b23f6d0 0x55a43b241ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 -- 172.21.15.116:0/2731261805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 msgr2=0x55a43b2365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b2365b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 -- 172.21.15.116:0/2731261805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a43b15eea0 con 0x55a43b236af0 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6186d9d640 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b23f6d0 0x55a43b241ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61700a22b0 0x7f61700a4770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6178035fd0 tx=0x7f6178035c80 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6185d9b640 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b23f190 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f61780057c0 tx=0x7f6178047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5adeabb2b63b9e8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6174ff9640 1 -- 172.21.15.116:0/2731261805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178047070 con 0x55a43b236af0 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6174ff9640 1 -- 172.21.15.116:0/2731261805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f617802ce90 con 0x55a43b236af0 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.913+0000 7f6174ff9640 1 -- 172.21.15.116:0/2731261805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617802c770 con 0x55a43b236af0 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 -- 172.21.15.116:0/2731261805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 msgr2=0x55a43b23f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b23f190 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f61780057c0 tx=0x7f6178047000 comp rx=0 tx=0).stop 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 -- 172.21.15.116:0/2731261805 shutdown_connections 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b23f6d0 0x55a43b241ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b23f190 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 --2- 172.21.15.116:0/2731261805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b2365b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 -- 172.21.15.116:0/2731261805 >> 172.21.15.116:0/2731261805 conn(0x55a43b230900 msgr2=0x55a43b230ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 -- 172.21.15.116:0/2731261805 shutdown_connections 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.914+0000 7f618f76d740 1 -- 172.21.15.116:0/2731261805 wait complete. 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 -- start start 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 Processor -- start 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b23c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 0x55a43b23d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a43b19d520 con 0x55a43b23f6d0 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a43ae70430 con 0x55a43b2361d0 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a43ae2ce00 con 0x55a43b236af0 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f6186d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 0x55a43b23d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f6186d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 0x55a43b23d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37696/0 (socket says 172.21.15.116:37696) 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f6186d9d640 1 -- 172.21.15.116:0/3409212990 learned_addr learned my addr 172.21.15.116:0/3409212990 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.915+0000 7f618f76d740 10 client.0 _ll_register_callbacks cb 0x55a43b13bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:55.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f618659c640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b23c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6185d9b640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6186d9d640 1 -- 172.21.15.116:0/3409212990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 msgr2=0x55a43b23cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6186d9d640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6186d9d640 1 -- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 msgr2=0x55a43b23c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6186d9d640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b23c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6186d9d640 1 -- 172.21.15.116:0/3409212990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618810ee50 con 0x55a43b23f6d0 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f6186d9d640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 0x55a43b23d1a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f618000efd0 tx=0x7f618009a810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4356c53dde04c7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f616dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:55.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f616dffb640 1 -- 172.21.15.116:0/3409212990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618001b850 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f616dffb640 1 -- 172.21.15.116:0/3409212990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f618009abe0 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a43ae9c7e0 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.916+0000 7f616dffb640 1 -- 172.21.15.116:0/3409212990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f618009ec80 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 10 client.16551 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a43b19bba0 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f616dffb640 1 -- 172.21.15.116:0/3409212990 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f61800b7020 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f616dffb640 1 -- 172.21.15.116:0/3409212990 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f6180018030 con 0x55a43b23f6d0 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 10 client.16551 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:55.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 20 client.16551 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b24fb60 0x55a43b251f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 4 client.16551 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:24:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a43aed9720 con 0x55a43b24fb60 2024-03-24T09:24:55.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.917+0000 7f618659c640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b24fb60 0x55a43b251f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.918+0000 7f616dffb640 10 client.16551 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.918+0000 7f618659c640 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b24fb60 0x55a43b251f80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5891921317d553f0 server_cookie=27b00fd410febb4a in_seq=0 out_seq=0 2024-03-24T09:24:55.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f616dffb640 1 -- 172.21.15.116:0/3409212990 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a43aed9720 con 0x55a43b24fb60 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f616dffb640 10 client.16551 handle_command_reply: tid=0 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 client.16551 shutdown 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 msgr2=0x55a43b23d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 0x55a43b23d1a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f618000efd0 tx=0x7f618009a810 comp rx=0 tx=0).stop 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 shutdown_connections 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b23f6d0 0x55a43b23d1a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b24fb60 0x55a43b251f80 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f616dffb640 0 client.16551 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 --2- 172.21.15.116:0/3409212990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b2361d0 0x55a43b23c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 >> 172.21.15.116:0/3409212990 conn(0x55a43b230900 msgr2=0x55a43b232650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.919+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 shutdown_connections 2024-03-24T09:24:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.920+0000 7f618f76d740 1 -- 172.21.15.116:0/3409212990 wait complete. 2024-03-24T09:24:55.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.920+0000 7f618f76d740 20 client.16551 trim_cache size 0 max 16384 2024-03-24T09:24:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 Processor -- start 2024-03-24T09:24:55.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 -- start start 2024-03-24T09:24:55.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b2361d0 0x55a43b24af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b232440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a43b122430 con 0x55a43b2361d0 2024-03-24T09:24:55.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a43b237570 con 0x55a43b23f6d0 2024-03-24T09:24:55.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618f76d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a43b2224d0 con 0x55a43b236af0 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f618659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b2361d0 0x55a43b24af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:46820/0 (socket says 172.21.15.116:46820) 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.943+0000 7f6185d9b640 1 -- 172.21.15.116:0/945907989 learned_addr learned my addr 172.21.15.116:0/945907989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f618659c640 1 -- 172.21.15.116:0/945907989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 msgr2=0x55a43b23cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f6186d9d640 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b232440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f618659c640 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f618659c640 1 -- 172.21.15.116:0/945907989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 msgr2=0x55a43b232440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f618659c640 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b232440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f618659c640 1 -- 172.21.15.116:0/945907989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a43b137b20 con 0x55a43b2361d0 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f6186d9d640 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b232440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f618659c640 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b2361d0 0x55a43b24af50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x55a43b24ff40 tx=0x7f617c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340298905cd2c3eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f616ffff640 1 -- 172.21.15.116:0/945907989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c017040 con 0x55a43b2361d0 2024-03-24T09:24:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f616ffff640 1 -- 172.21.15.116:0/945907989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f617c018040 con 0x55a43b2361d0 2024-03-24T09:24:55.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.944+0000 7f616ffff640 1 -- 172.21.15.116:0/945907989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c0035f0 con 0x55a43b2361d0 2024-03-24T09:24:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 -- 172.21.15.116:0/945907989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b2361d0 msgr2=0x55a43b24af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b2361d0 0x55a43b24af50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x55a43b24ff40 tx=0x7f617c009260 comp rx=0 tx=0).stop 2024-03-24T09:24:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 -- 172.21.15.116:0/945907989 shutdown_connections 2024-03-24T09:24:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b232440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b236af0 0x55a43b23cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 --2- 172.21.15.116:0/945907989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b2361d0 0x55a43b24af50 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 -- 172.21.15.116:0/945907989 >> 172.21.15.116:0/945907989 conn(0x55a43b230900 msgr2=0x55a43b231d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 -- 172.21.15.116:0/945907989 shutdown_connections 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.945+0000 7f618f76d740 1 -- 172.21.15.116:0/945907989 wait complete. 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 -- start start 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 Processor -- start 2024-03-24T09:24:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b2361d0 0x55a43b239260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b2397a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b239ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a43b2b4ec0 con 0x55a43b236af0 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a43b2b5000 con 0x55a43b23f6d0 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a43b2b5140 con 0x55a43b2361d0 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618f76d740 10 client.0 _ll_register_callbacks cb 0x7f618009aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:24:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b2397a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618659c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b2361d0 0x55a43b239260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f6185d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b2397a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37712/0 (socket says 172.21.15.116:37712) 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f6185d9b640 1 -- 172.21.15.116:0/2184365265 learned_addr learned my addr 172.21.15.116:0/2184365265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f618659c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b2361d0 0x55a43b239260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:46836/0 (socket says 172.21.15.116:46836) 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.946+0000 7f6186d9d640 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b239ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f6185d9b640 1 -- 172.21.15.116:0/2184365265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b2361d0 msgr2=0x55a43b239260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f6185d9b640 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b2361d0 0x55a43b239260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f6185d9b640 1 -- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 msgr2=0x55a43b239ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f6185d9b640 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b239ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f6185d9b640 1 -- 172.21.15.116:0/2184365265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c009d50 con 0x55a43b236af0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f6185d9b640 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b2397a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6178037040 tx=0x7f6178004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3390b3cbf3d61529 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f616dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f616dffb640 1 -- 172.21.15.116:0/2184365265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178034850 con 0x55a43b236af0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f617802ce90 con 0x55a43b236af0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f616dffb640 1 -- 172.21.15.116:0/2184365265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6178053070 con 0x55a43b236af0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.947+0000 7f616dffb640 1 -- 172.21.15.116:0/2184365265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f617802c450 con 0x55a43b236af0 2024-03-24T09:24:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 10 client.16557 fetch_fsmap learned FSMap version 32 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a43b15d700 con 0x55a43b236af0 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f616dffb640 1 -- 172.21.15.116:0/2184365265 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f617802ce90 con 0x55a43b236af0 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f616dffb640 1 -- 172.21.15.116:0/2184365265 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f6178003780 con 0x55a43b236af0 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 10 client.16557 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 20 client.16557 populate_metadata read hostname 'smithi116' 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b2b6730 0x55a43b2b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 4 client.16557 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:24:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a43af28890 con 0x55a43b2b6730 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.948+0000 7f618659c640 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b2b6730 0x55a43b2b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618659c640 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b2b6730 0x55a43b2b8b50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff7d38a66e336bd5 server_cookie=7e2086b79a58e2cf in_seq=0 out_seq=0 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f616dffb640 10 client.16557 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f616dffb640 1 -- 172.21.15.116:0/2184365265 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a43af28890 con 0x55a43b2b6730 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f616dffb640 10 client.16557 handle_command_reply: tid=0 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 client.16557 shutdown 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 msgr2=0x55a43b2397a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b2397a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6178037040 tx=0x7f6178004280 comp rx=0 tx=0).stop 2024-03-24T09:24:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 shutdown_connections 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a43b2b6730 0x55a43b2b8b50 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a43b23f6d0 0x55a43b239ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a43b236af0 0x55a43b2397a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 --2- 172.21.15.116:0/2184365265 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a43b2361d0 0x55a43b239260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 >> 172.21.15.116:0/2184365265 conn(0x55a43b230900 msgr2=0x55a43b22fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f616dffb640 0 client.16557 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.949+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 shutdown_connections 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618f76d740 1 -- 172.21.15.116:0/2184365265 wait complete. 2024-03-24T09:24:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618f76d740 20 client.16557 trim_cache size 0 max 16384 2024-03-24T09:24:55.959 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:24:55.959 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:24:55.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:24:55.959 INFO:teuthology.orchestra.run.smithi116.stdout: "d2a13b53-12e4-4085-b06c-1e7efeefb0b7": { 2024-03-24T09:24:55.959 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:24:55.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "d2a13b53-12e4-4085-b06c-1e7efeefb0b7", 2024-03-24T09:24:55.960 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:24:55.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:24:55.960 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:24:55.960 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61700a22b0 msgr2=0x7f61700a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61700a22b0 0x7f61700a4770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6178035fd0 tx=0x7f6178035c80 comp rx=0 tx=0).stop 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 msgr2=0x7f618819e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f618819e6f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f618000eaa0 tx=0x7f618000ef60 comp rx=0 tx=0).stop 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 shutdown_connections 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6188107690 0x7f618819e6f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f61700a22b0 0x7f61700a4770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f61880692b0 0x7f618819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 --2- 172.21.15.116:0/414373508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f61880688f0 0x7f6188199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:24:55.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 >> 172.21.15.116:0/414373508 conn(0x7f6188077710 msgr2=0x7f618810ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:24:55.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 shutdown_connections 2024-03-24T09:24:55.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:24:55.950+0000 7f618caae640 1 -- 172.21.15.116:0/414373508 wait complete. 2024-03-24T09:24:55.964 INFO:tasks.cephfs.filesystem:scrub status for tag:d2a13b53-12e4-4085-b06c-1e7efeefb0b7 - {'path': '/', 'tag': 'd2a13b53-12e4-4085-b06c-1e7efeefb0b7', 'options': 'recursive,force'} 2024-03-24T09:24:56.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:24:56.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:56.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:56.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:56.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:56 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:24:56.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:56 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:24:56.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:56.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:56.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:56.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:56.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:56 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: pgmap v88: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 304 KiB/s wr, 62 op/s 2024-03-24T09:24:57.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:57.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:57.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:24:57.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:57 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: pgmap v88: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 304 KiB/s wr, 62 op/s 2024-03-24T09:24:57.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:57.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:57.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:24:57.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:57 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: pgmap v88: 129 pgs: 129 active+clean; 150 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 304 KiB/s wr, 62 op/s 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:57 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:24:59.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:24:59 smithi143 ceph-mon[35965]: pgmap v89: 129 pgs: 129 active+clean; 151 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 389 KiB/s wr, 95 op/s 2024-03-24T09:24:59.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:24:59 smithi117 ceph-mon[35788]: pgmap v89: 129 pgs: 129 active+clean; 151 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 389 KiB/s wr, 95 op/s 2024-03-24T09:24:59.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:24:59 smithi116 ceph-mon[32207]: pgmap v89: 129 pgs: 129 active+clean; 151 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 389 KiB/s wr, 95 op/s 2024-03-24T09:25:01.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:01 smithi116 ceph-mon[32207]: pgmap v90: 129 pgs: 129 active+clean; 151 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 150 KiB/s wr, 49 op/s 2024-03-24T09:25:01.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:01 smithi143 ceph-mon[35965]: pgmap v90: 129 pgs: 129 active+clean; 151 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 150 KiB/s wr, 49 op/s 2024-03-24T09:25:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:01 smithi117 ceph-mon[35788]: pgmap v90: 129 pgs: 129 active+clean; 151 MiB data, 898 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 150 KiB/s wr, 49 op/s 2024-03-24T09:25:03.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:03 smithi143 ceph-mon[35965]: pgmap v91: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 177 KiB/s wr, 71 op/s 2024-03-24T09:25:03.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:03 smithi117 ceph-mon[35788]: pgmap v91: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 177 KiB/s wr, 71 op/s 2024-03-24T09:25:03.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:03 smithi116 ceph-mon[32207]: pgmap v91: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 177 KiB/s wr, 71 op/s 2024-03-24T09:25:05.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:05 smithi143 ceph-mon[35965]: pgmap v92: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 126 KiB/s wr, 55 op/s 2024-03-24T09:25:05.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:05 smithi117 ceph-mon[35788]: pgmap v92: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 126 KiB/s wr, 55 op/s 2024-03-24T09:25:05.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:05 smithi116 ceph-mon[32207]: pgmap v92: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 126 KiB/s wr, 55 op/s 2024-03-24T09:25:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:07 smithi143 ceph-mon[35965]: pgmap v93: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 112 KiB/s wr, 53 op/s 2024-03-24T09:25:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:07 smithi117 ceph-mon[35788]: pgmap v93: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 112 KiB/s wr, 53 op/s 2024-03-24T09:25:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:07 smithi116 ceph-mon[32207]: pgmap v93: 129 pgs: 129 active+clean; 151 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 112 KiB/s wr, 53 op/s 2024-03-24T09:25:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:08 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:08 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:08 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:08 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:08.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:08 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:08.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:08 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:09 smithi143 ceph-mon[35965]: pgmap v94: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 117 KiB/s wr, 59 op/s 2024-03-24T09:25:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:09 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:09 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:09 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:09 smithi117 ceph-mon[35788]: pgmap v94: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 117 KiB/s wr, 59 op/s 2024-03-24T09:25:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:09 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:09 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:09 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:09 smithi116 ceph-mon[32207]: pgmap v94: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 117 KiB/s wr, 59 op/s 2024-03-24T09:25:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:09 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:09 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:09 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:11.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:11 smithi143 ceph-mon[35965]: pgmap v95: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-03-24T09:25:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:11 smithi117 ceph-mon[35788]: pgmap v95: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-03-24T09:25:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:11 smithi116 ceph-mon[32207]: pgmap v95: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 32 KiB/s wr, 26 op/s 2024-03-24T09:25:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:13 smithi143 ceph-mon[35965]: pgmap v96: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 KiB/s wr, 31 op/s 2024-03-24T09:25:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:13 smithi117 ceph-mon[35788]: pgmap v96: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 KiB/s wr, 31 op/s 2024-03-24T09:25:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:13 smithi116 ceph-mon[32207]: pgmap v96: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 33 KiB/s wr, 31 op/s 2024-03-24T09:25:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:14 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:25:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:14 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:25:14.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:14 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:25:14.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:14 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:25:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:14 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:25:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:14 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:25:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:14 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:25:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:14 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:25:14.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:14 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:25:14.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:14 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:25:14.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:14 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:25:14.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:14 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:25:15.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:15 smithi143 ceph-mon[35965]: pgmap v97: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-03-24T09:25:15.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:15 smithi117 ceph-mon[35788]: pgmap v97: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-03-24T09:25:15.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:15 smithi116 ceph-mon[32207]: pgmap v97: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-03-24T09:25:17.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:17 smithi143 ceph-mon[35965]: pgmap v98: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-03-24T09:25:17.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:17 smithi117 ceph-mon[35788]: pgmap v98: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-03-24T09:25:17.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:17 smithi116 ceph-mon[32207]: pgmap v98: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-03-24T09:25:19.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:19 smithi117 ceph-mon[35788]: pgmap v99: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 14 op/s 2024-03-24T09:25:19.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:19 smithi143 ceph-mon[35965]: pgmap v99: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 14 op/s 2024-03-24T09:25:19.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:19 smithi116 ceph-mon[32207]: pgmap v99: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.5 KiB/s wr, 14 op/s 2024-03-24T09:25:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:21 smithi143 ceph-mon[35965]: pgmap v100: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-24T09:25:21.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:21 smithi117 ceph-mon[35788]: pgmap v100: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-24T09:25:21.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:21 smithi116 ceph-mon[32207]: pgmap v100: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.0 KiB/s wr, 9 op/s 2024-03-24T09:25:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:23 smithi143 ceph-mon[35965]: pgmap v101: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-03-24T09:25:23.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:23 smithi117 ceph-mon[35788]: pgmap v101: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-03-24T09:25:23.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:23 smithi116 ceph-mon[32207]: pgmap v101: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.0 KiB/s wr, 13 op/s 2024-03-24T09:25:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:25 smithi143 ceph-mon[35965]: pgmap v102: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-24T09:25:25.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:25 smithi117 ceph-mon[35788]: pgmap v102: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-24T09:25:25.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:25 smithi116 ceph-mon[32207]: pgmap v102: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-24T09:25:25.965 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:25:25.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:25:26.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 -- 172.21.15.116:0/751353525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f010bce0 msgr2=0x7ff5f010e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 --2- 172.21.15.116:0/751353525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f010bce0 0x7ff5f010e150 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8009900 tx=0x7ff5e802bcd0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 -- 172.21.15.116:0/751353525 shutdown_connections 2024-03-24T09:25:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 --2- 172.21.15.116:0/751353525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f010bce0 0x7ff5f010e150 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 --2- 172.21.15.116:0/751353525 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f01032c0 0x7ff5f010b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 --2- 172.21.15.116:0/751353525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5f01029a0 0x7ff5f0102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 -- 172.21.15.116:0/751353525 >> 172.21.15.116:0/751353525 conn(0x7ff5f0077770 msgr2=0x7ff5f0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 -- 172.21.15.116:0/751353525 shutdown_connections 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.080+0000 7ff5f5943640 1 -- 172.21.15.116:0/751353525 wait complete. 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 Processor -- start 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 -- start start 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 0x7ff5f0199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5f01032c0 0x7ff5f019a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f010bce0 0x7ff5f019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f019ed40 con 0x7ff5f01029a0 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f019eeb0 con 0x7ff5f01032c0 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5f5943640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f019f070 con 0x7ff5f010bce0 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 0x7ff5f0199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 0x7ff5f0199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55922/0 (socket says 172.21.15.116:55922) 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1779809342 learned_addr learned my addr 172.21.15.116:0/1779809342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5f01032c0 0x7ff5f019a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f010bce0 0x7ff5f019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1779809342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f010bce0 msgr2=0x7ff5f019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f010bce0 0x7ff5f019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1779809342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5f01032c0 msgr2=0x7ff5f019a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5f01032c0 0x7ff5f019a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.081+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1779809342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e0009840 con 0x7ff5f01029a0 2024-03-24T09:25:26.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.082+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 0x7ff5f0199e80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009d50 tx=0x7ff5e000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.082+0000 7ff5f4941640 1 -- 172.21.15.116:0/1779809342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0003b70 con 0x7ff5f01029a0 2024-03-24T09:25:26.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.082+0000 7ff5f4941640 1 -- 172.21.15.116:0/1779809342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5e0003cd0 con 0x7ff5f01029a0 2024-03-24T09:25:26.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.082+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e8009560 con 0x7ff5f01029a0 2024-03-24T09:25:26.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.082+0000 7ff5f4941640 1 -- 172.21.15.116:0/1779809342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0010810 con 0x7ff5f01029a0 2024-03-24T09:25:26.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.082+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f01a69f0 con 0x7ff5f01029a0 2024-03-24T09:25:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.084+0000 7ff5f4941640 1 -- 172.21.15.116:0/1779809342 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7ff5e0004000 con 0x7ff5f01029a0 2024-03-24T09:25:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.084+0000 7ff5f85f3740 1 Processor -- start 2024-03-24T09:25:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.084+0000 7ff5f85f3740 1 -- start start 2024-03-24T09:25:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.084+0000 7ff5f4941640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5dc0a2230 0x7ff5dc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea2595b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea259af0 0x556eea262190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea264ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eea17df30 con 0x556eea2626d0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556eea1c7810 con 0x556eea2591d0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556eea1c4d40 con 0x556eea259af0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5dc0a2230 0x7ff5dc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5f4941640 1 -- 172.21.15.116:0/1779809342 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff5e0014070 con 0x7ff5f01029a0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea2595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5ee7fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea259af0 0x556eea262190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea2595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58972/0 (socket says 172.21.15.116:58972) 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1274804118 learned_addr learned my addr 172.21.15.116:0/1274804118 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea264ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1274804118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea259af0 msgr2=0x556eea262190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea259af0 0x556eea262190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1274804118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 msgr2=0x556eea264ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea264ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 -- 172.21.15.116:0/1274804118 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eea181ea0 con 0x556eea2591d0 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea259af0 0x556eea262190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5dc0a2230 0x7ff5dc0a46f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8005ab0 tx=0x7ff5e8004bc0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea2595b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00a3540 tx=0x7ff5e009ae10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d3dae21e3f21840 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5e5ffb640 1 -- 172.21.15.116:0/1274804118 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0087110 con 0x556eea2591d0 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5e5ffb640 1 -- 172.21.15.116:0/1274804118 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5e009b2d0 con 0x556eea2591d0 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.085+0000 7ff5e5ffb640 1 -- 172.21.15.116:0/1274804118 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0004560 con 0x556eea2591d0 2024-03-24T09:25:26.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 -- 172.21.15.116:0/1274804118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 msgr2=0x556eea2595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea2595b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5e00a3540 tx=0x7ff5e009ae10 comp rx=0 tx=0).stop 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 -- 172.21.15.116:0/1274804118 shutdown_connections 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea264ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea259af0 0x556eea262190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/1274804118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea2595b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 -- 172.21.15.116:0/1274804118 >> 172.21.15.116:0/1274804118 conn(0x556eea253900 msgr2=0x556eea253ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 -- 172.21.15.116:0/1274804118 shutdown_connections 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 -- 172.21.15.116:0/1274804118 wait complete. 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.086+0000 7ff5f85f3740 1 -- start start 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 Processor -- start 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea25f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea25fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2626d0 0x556eea260180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eea1c0520 con 0x556eea259af0 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ee9e93430 con 0x556eea2591d0 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ee9e4fe00 con 0x556eea2626d0 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5f85f3740 10 client.0 _ll_register_callbacks cb 0x556eea15ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5ee7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea25fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea25f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5ee7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea25fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55928/0 (socket says 172.21.15.116:55928) 2024-03-24T09:25:26.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5ee7fc640 1 -- 172.21.15.116:0/3555695512 learned_addr learned my addr 172.21.15.116:0/3555695512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2626d0 0x556eea260180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5eeffd640 1 -- 172.21.15.116:0/3555695512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2626d0 msgr2=0x556eea260180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2626d0 0x556eea260180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5eeffd640 1 -- 172.21.15.116:0/3555695512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 msgr2=0x556eea25fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea25fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5eeffd640 1 -- 172.21.15.116:0/3555695512 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f01a69f0 con 0x556eea2591d0 2024-03-24T09:25:26.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.087+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea25fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea25f700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff5e009ada0 tx=0x7ff5e0099640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4a0aab30e05af0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:25:26.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ceffd640 1 -- 172.21.15.116:0/3555695512 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0087110 con 0x556eea2591d0 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ceffd640 1 -- 172.21.15.116:0/3555695512 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff5e0095070 con 0x556eea2591d0 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ee9ebf7e0 con 0x556eea2591d0 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ceffd640 1 -- 172.21.15.116:0/3555695512 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff5e009f420 con 0x556eea2591d0 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 10 client.26396 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556eea1beba0 con 0x556eea2591d0 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ceffd640 1 -- 172.21.15.116:0/3555695512 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556ee9ebf7e0 con 0x556eea2591d0 2024-03-24T09:25:26.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ceffd640 1 -- 172.21.15.116:0/3555695512 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff5e0090ba0 con 0x556eea2591d0 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 10 client.26396 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 10 client.26396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 20 client.26396 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea272850 0x556eea274cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 4 client.26396 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ee9efc720 con 0x556eea272850 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.088+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea272850 0x556eea274cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.089+0000 7ff5ceffd640 10 client.26396 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:26.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.089+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea272850 0x556eea274cb0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecebc6d422e9ba70 server_cookie=bcc0a0ad21ea0975 in_seq=0 out_seq=0 2024-03-24T09:25:26.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.089+0000 7ff5ceffd640 1 -- 172.21.15.116:0/3555695512 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ee9efc720 con 0x556eea272850 2024-03-24T09:25:26.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.089+0000 7ff5ceffd640 10 client.26396 handle_command_reply: tid=0 2024-03-24T09:25:26.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.089+0000 7ff5f85f3740 1 client.26396 shutdown 2024-03-24T09:25:26.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 msgr2=0x556eea25f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea25f700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff5e009ada0 tx=0x7ff5e0099640 comp rx=0 tx=0).stop 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 shutdown_connections 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2626d0 0x556eea260180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea25fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea272850 0x556eea274cb0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/3555695512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2591d0 0x556eea25f700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5ceffd640 0 client.26396 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:26.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 >> 172.21.15.116:0/3555695512 conn(0x556eea253900 msgr2=0x556eea252b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 shutdown_connections 2024-03-24T09:25:26.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 1 -- 172.21.15.116:0/3555695512 wait complete. 2024-03-24T09:25:26.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.090+0000 7ff5f85f3740 20 client.26396 trim_cache size 0 max 16384 2024-03-24T09:25:26.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 Processor -- start 2024-03-24T09:25:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 -- start start 2024-03-24T09:25:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea261e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea259af0 0x556eea26df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea27a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eea145430 con 0x556eea2626d0 2024-03-24T09:25:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556eea26f5e0 con 0x556eea259af0 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556eea254900 con 0x556eea2591d0 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea27a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea261e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea27a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55930/0 (socket says 172.21.15.116:55930) 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/4187104235 learned_addr learned my addr 172.21.15.116:0/4187104235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea259af0 0x556eea26df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/4187104235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 msgr2=0x556eea261e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea261e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/4187104235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea259af0 msgr2=0x556eea26df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea259af0 0x556eea26df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ef7fe640 1 -- 172.21.15.116:0/4187104235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eea15ab20 con 0x556eea2626d0 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea261e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.107+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea259af0 0x556eea26df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea27a580 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8036040 tx=0x7ff5e8046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfbb25ced09f4ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5e57fa640 1 -- 172.21.15.116:0/4187104235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e80046c0 con 0x556eea2626d0 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5e57fa640 1 -- 172.21.15.116:0/4187104235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5e802ca30 con 0x556eea2626d0 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5e57fa640 1 -- 172.21.15.116:0/4187104235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8005020 con 0x556eea2626d0 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 -- 172.21.15.116:0/4187104235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 msgr2=0x556eea27a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea27a580 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8036040 tx=0x7ff5e8046000 comp rx=0 tx=0).stop 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 -- 172.21.15.116:0/4187104235 shutdown_connections 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea2626d0 0x556eea27a580 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea259af0 0x556eea26df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/4187104235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea261e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 -- 172.21.15.116:0/4187104235 >> 172.21.15.116:0/4187104235 conn(0x556eea253900 msgr2=0x556eea252b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.108+0000 7ff5f85f3740 1 -- 172.21.15.116:0/4187104235 shutdown_connections 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 -- 172.21.15.116:0/4187104235 wait complete. 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 -- start start 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 Processor -- start 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea26a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea26a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2626d0 0x556eea266aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556eea267170 con 0x556eea259af0 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556eea2672e0 con 0x556eea2626d0 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556eea2674a0 con 0x556eea2591d0 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5f85f3740 10 client.0 _ll_register_callbacks cb 0x7ff5d80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5ee7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea26a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5eeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea26a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5ee7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea26a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55936/0 (socket says 172.21.15.116:55936) 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5ee7fc640 1 -- 172.21.15.116:0/2710394669 learned_addr learned my addr 172.21.15.116:0/2710394669 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.109+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2626d0 0x556eea266aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ee7fc640 1 -- 172.21.15.116:0/2710394669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 msgr2=0x556eea26a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea26a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ee7fc640 1 -- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2626d0 msgr2=0x556eea266aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2626d0 0x556eea266aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ee7fc640 1 -- 172.21.15.116:0/2710394669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eea1beba0 con 0x556eea259af0 2024-03-24T09:25:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea26a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ef7fe640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2626d0 0x556eea266aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ee7fc640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea26a9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8004700 tx=0x7ff5d8012ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9c0216242df448 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ceffd640 1 -- 172.21.15.116:0/2710394669 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8017030 con 0x556eea259af0 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ceffd640 1 -- 172.21.15.116:0/2710394669 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff5d80205d0 con 0x556eea259af0 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5e009b2d0 con 0x556eea259af0 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ceffd640 1 -- 172.21.15.116:0/2710394669 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff5d8020790 con 0x556eea259af0 2024-03-24T09:25:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5f85f3740 10 client.16575 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7ff5d8012040 con 0x556eea259af0 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.110+0000 7ff5ceffd640 1 -- 172.21.15.116:0/2710394669 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5e009b2d0 con 0x556eea259af0 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5ceffd640 1 -- 172.21.15.116:0/2710394669 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff5d8026030 con 0x556eea259af0 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 10 client.16575 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 20 client.16575 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea2da140 0x556eea2dc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 4 client.16575 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ee9f4b890 con 0x556eea2da140 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea2da140 0x556eea2dc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5ceffd640 10 client.16575 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5eeffd640 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea2da140 0x556eea2dc560 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=635eb0762db0ec server_cookie=a8c14b3f3163b780 in_seq=0 out_seq=0 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5ceffd640 1 -- 172.21.15.116:0/2710394669 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ee9f4b890 con 0x556eea2da140 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5ceffd640 10 client.16575 handle_command_reply: tid=0 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 client.16575 shutdown 2024-03-24T09:25:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 msgr2=0x556eea26a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea26a9c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8004700 tx=0x7ff5d8012ae0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 shutdown_connections 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556eea2da140 0x556eea2dc560 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556eea2626d0 0x556eea266aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5ceffd640 0 client.16575 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556eea259af0 0x556eea26a9c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 --2- 172.21.15.116:0/2710394669 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556eea2591d0 0x556eea26a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 >> 172.21.15.116:0/2710394669 conn(0x556eea253900 msgr2=0x556eea254ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 shutdown_connections 2024-03-24T09:25:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.111+0000 7ff5f85f3740 1 -- 172.21.15.116:0/2710394669 wait complete. 2024-03-24T09:25:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f85f3740 20 client.16575 trim_cache size 0 max 16384 2024-03-24T09:25:26.128 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:25:26.128 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:25:26.128 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:25:26.129 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:25:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5dc0a2230 msgr2=0x7ff5dc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5dc0a2230 0x7ff5dc0a46f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5e8005ab0 tx=0x7ff5e8004bc0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 msgr2=0x7ff5f0199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 0x7ff5f0199e80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009d50 tx=0x7ff5e000c6a0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 shutdown_connections 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5dc0a2230 0x7ff5dc0a46f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5f010bce0 0x7ff5f019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5f01032c0 0x7ff5f019a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 --2- 172.21.15.116:0/1779809342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5f01029a0 0x7ff5f0199e80 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 >> 172.21.15.116:0/1779809342 conn(0x7ff5f0077770 msgr2=0x7ff5f0107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 shutdown_connections 2024-03-24T09:25:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.112+0000 7ff5f5943640 1 -- 172.21.15.116:0/1779809342 wait complete. 2024-03-24T09:25:26.132 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:25:26.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:25:26.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.232+0000 7f67d19e2640 1 -- 172.21.15.116:0/1915431743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 msgr2=0x7f67cc10e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.232+0000 7f67d19e2640 1 --2- 172.21.15.116:0/1915431743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc10e1f0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009900 tx=0x7f67c402bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.232+0000 7f67d19e2640 1 -- 172.21.15.116:0/1915431743 shutdown_connections 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.232+0000 7f67d19e2640 1 --2- 172.21.15.116:0/1915431743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc10e1f0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.232+0000 7f67d19e2640 1 --2- 172.21.15.116:0/1915431743 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc1032a0 0x7f67cc10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.232+0000 7f67d19e2640 1 --2- 172.21.15.116:0/1915431743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc102980 0x7f67cc102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- 172.21.15.116:0/1915431743 >> 172.21.15.116:0/1915431743 conn(0x7f67cc077710 msgr2=0x7f67cc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- 172.21.15.116:0/1915431743 shutdown_connections 2024-03-24T09:25:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- 172.21.15.116:0/1915431743 wait complete. 2024-03-24T09:25:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 Processor -- start 2024-03-24T09:25:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- start start 2024-03-24T09:25:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc102980 0x7f67cc19d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 0x7f67cc19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc1980b0 con 0x7f67cc10bd50 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc198220 con 0x7f67cc1032a0 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.233+0000 7f67d19e2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc1983e0 con 0x7f67cc102980 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67ca7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 0x7f67cc19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67caffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc102980 0x7f67cc19d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67caffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc102980 0x7f67cc19d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:47834/0 (socket says 172.21.15.116:47834) 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67ca7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 0x7f67cc19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59012/0 (socket says 172.21.15.116:59012) 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67caffd640 1 -- 172.21.15.116:0/4247785466 learned_addr learned my addr 172.21.15.116:0/4247785466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 -- 172.21.15.116:0/4247785466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc102980 msgr2=0x7f67cc19d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc102980 0x7f67cc19d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 -- 172.21.15.116:0/4247785466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 msgr2=0x7f67cc19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 0x7f67cc19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 -- 172.21.15.116:0/4247785466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc009840 con 0x7f67cc10bd50 2024-03-24T09:25:26.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67ca7fc640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 0x7f67cc19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67cb7fe640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc197a60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009230 tx=0x7f67c4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4004350 con 0x7f67cc10bd50 2024-03-24T09:25:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c0009490 con 0x7f67cc10bd50 2024-03-24T09:25:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f67c402de90 con 0x7f67cc10bd50 2024-03-24T09:25:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c4005220 con 0x7f67cc10bd50 2024-03-24T09:25:26.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.234+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c4009560 con 0x7f67cc10bd50 2024-03-24T09:25:26.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.235+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67cc068500 con 0x7f67cc10bd50 2024-03-24T09:25:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.236+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f67c4003c30 con 0x7f67cc10bd50 2024-03-24T09:25:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.237+0000 7f67d09e0640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67b40a2230 0x7f67b40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.237+0000 7f67caffd640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67b40a2230 0x7f67b40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.237+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f67c40f0210 con 0x7f67cc10bd50 2024-03-24T09:25:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.237+0000 7f67caffd640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67b40a2230 0x7f67b40a46f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67cc198ba0 tx=0x7f67bc009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.242+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f67c40b8c00 con 0x7f67cc10bd50 2024-03-24T09:25:26.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.454+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67cc106e40 con 0x7f67cc10bd50 2024-03-24T09:25:26.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.455+0000 7f67d09e0640 1 -- 172.21.15.116:0/4247785466 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7f67c40bc2c0 con 0x7f67cc10bd50 2024-03-24T09:25:26.458 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:25:26.458 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:25:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:25:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67b40a2230 msgr2=0x7f67b40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67b40a2230 0x7f67b40a46f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67cc198ba0 tx=0x7f67bc009500 comp rx=0 tx=0).stop 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 msgr2=0x7f67cc197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc197a60 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f67c4009230 tx=0x7f67c4005dd0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 shutdown_connections 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc10bd50 0x7f67cc197a60 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67cc1032a0 0x7f67cc19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f67b40a2230 0x7f67b40a46f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 --2- 172.21.15.116:0/4247785466 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67cc102980 0x7f67cc19d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 >> 172.21.15.116:0/4247785466 conn(0x7f67cc077710 msgr2=0x7f67cc0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 shutdown_connections 2024-03-24T09:25:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.457+0000 7f67d19e2640 1 -- 172.21.15.116:0/4247785466 wait complete. 2024-03-24T09:25:26.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:25:26.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.572+0000 7f16d388e640 1 -- 172.21.15.116:0/2031146038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1016b0 msgr2=0x7f16cc101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.572+0000 7f16d388e640 1 --2- 172.21.15.116:0/2031146038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1016b0 0x7f16cc101b30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009930 tx=0x7f16bc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:26.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- 172.21.15.116:0/2031146038 shutdown_connections 2024-03-24T09:25:26.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 --2- 172.21.15.116:0/2031146038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16cc102070 0x7f16cc10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 --2- 172.21.15.116:0/2031146038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1016b0 0x7f16cc101b30 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 --2- 172.21.15.116:0/2031146038 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16cc1076c0 0x7f16cc107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- 172.21.15.116:0/2031146038 >> 172.21.15.116:0/2031146038 conn(0x7f16cc0fd140 msgr2=0x7f16cc0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- 172.21.15.116:0/2031146038 shutdown_connections 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- 172.21.15.116:0/2031146038 wait complete. 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 Processor -- start 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- start start 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16cc1016b0 0x7f16cc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16cc102070 0x7f16cc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 0x7f16cc197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc198130 con 0x7f16cc1076c0 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1982a0 con 0x7f16cc102070 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.573+0000 7f16d388e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc198460 con 0x7f16cc1016b0 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 0x7f16cc197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d162a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16cc1016b0 0x7f16cc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 0x7f16cc197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43712/0 (socket says 172.21.15.116:43712) 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 -- 172.21.15.116:0/4106207747 learned_addr learned my addr 172.21.15.116:0/4106207747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d0e29640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16cc102070 0x7f16cc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 -- 172.21.15.116:0/4106207747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16cc1016b0 msgr2=0x7f16cc19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16cc1016b0 0x7f16cc19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 -- 172.21.15.116:0/4106207747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16cc102070 msgr2=0x7f16cc19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16cc102070 0x7f16cc19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 -- 172.21.15.116:0/4106207747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c8009490 con 0x7f16cc1076c0 2024-03-24T09:25:26.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 0x7f16cc197ae0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f16c400eb10 tx=0x7f16c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16c27fc640 1 -- 172.21.15.116:0/4106207747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4004b20 con 0x7f16cc1076c0 2024-03-24T09:25:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16c27fc640 1 -- 172.21.15.116:0/4106207747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c4004c80 con 0x7f16cc1076c0 2024-03-24T09:25:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16bc009560 con 0x7f16cc1076c0 2024-03-24T09:25:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16c27fc640 1 -- 172.21.15.116:0/4106207747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4010690 con 0x7f16cc1076c0 2024-03-24T09:25:26.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.574+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc1a47f0 con 0x7f16cc1076c0 2024-03-24T09:25:26.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.576+0000 7f16c27fc640 1 -- 172.21.15.116:0/4106207747 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f16c4005020 con 0x7f16cc1076c0 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 Processor -- start 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 -- start start 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16c27fc640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f16b80a2230 0x7f16b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11145500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145a40 0x55ea111527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 0x55ea111550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea110b3840 con 0x55ea11152d00 2024-03-24T09:25:26.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ea110b0d70 con 0x55ea11145a40 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d645f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea110ac550 con 0x55ea11145120 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d1e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 0x55ea111550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d0e29640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f16b80a2230 0x7f16b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d162a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d1e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 0x55ea111550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43728/0 (socket says 172.21.15.116:43728) 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d1e2b640 1 -- 172.21.15.116:0/2157966223 learned_addr learned my addr 172.21.15.116:0/2157966223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16c27fc640 1 -- 172.21.15.116:0/4106207747 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f16c4014070 con 0x7f16cc1076c0 2024-03-24T09:25:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d0e29640 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145a40 0x55ea111527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.577+0000 7f16d1e2b640 1 -- 172.21.15.116:0/2157966223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 msgr2=0x55ea11145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d1e2b640 1 -- 172.21.15.116:0/2157966223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145a40 msgr2=0x55ea111527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145a40 0x55ea111527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d1e2b640 1 -- 172.21.15.116:0/2157966223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea11041de0 con 0x55ea11152d00 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d162a640 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 0x55ea111550f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f16c40049d0 tx=0x7f16c401bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4176e0674132dc85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16d0e29640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f16b80a2230 0x7f16b80a46f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f16bc005ab0 tx=0x7f16bc004750 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16b77fe640 1 -- 172.21.15.116:0/2157966223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c409ab90 con 0x55ea11152d00 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16b77fe640 1 -- 172.21.15.116:0/2157966223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c409a9b0 con 0x55ea11152d00 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.578+0000 7f16b77fe640 1 -- 172.21.15.116:0/2157966223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c40bc0b0 con 0x55ea11152d00 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- 172.21.15.116:0/2157966223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 msgr2=0x55ea111550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 0x55ea111550f0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f16c40049d0 tx=0x7f16c401bb30 comp rx=0 tx=0).stop 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- 172.21.15.116:0/2157966223 shutdown_connections 2024-03-24T09:25:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11152d00 0x55ea111550f0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145a40 0x55ea111527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- 172.21.15.116:0/2157966223 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- 172.21.15.116:0/2157966223 >> 172.21.15.116:0/2157966223 conn(0x55ea1113fb20 msgr2=0x55ea1113ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- 172.21.15.116:0/2157966223 shutdown_connections 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- 172.21.15.116:0/2157966223 wait complete. 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- start start 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 Processor -- start 2024-03-24T09:25:26.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145120 0x55ea1114fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea111502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11152d00 0x55ea111507e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea10d7f430 con 0x55ea11145a40 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ea10d3be00 con 0x55ea11145120 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea11031590 con 0x55ea11152d00 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.579+0000 7f16d645f740 10 client.0 _ll_register_callbacks cb 0x55ea110a0fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea111502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea111502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43730/0 (socket says 172.21.15.116:43730) 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 -- 172.21.15.116:0/4089357292 learned_addr learned my addr 172.21.15.116:0/4089357292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11152d00 0x55ea111507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d162a640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145120 0x55ea1114fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 -- 172.21.15.116:0/4089357292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11152d00 msgr2=0x55ea111507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11152d00 0x55ea111507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 -- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145120 msgr2=0x55ea1114fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145120 0x55ea1114fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 -- 172.21.15.116:0/4089357292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc1a47f0 con 0x55ea11145a40 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d0e29640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea111502a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f16bc035000 tx=0x7f16bc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccec8d5a48bef10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16b4ff9640 1 -- 172.21.15.116:0/4089357292 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc048070 con 0x55ea11145a40 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ea10dab7e0 con 0x55ea11145a40 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16b4ff9640 1 -- 172.21.15.116:0/4089357292 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f16bc02cb80 con 0x55ea11145a40 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16b4ff9640 1 -- 172.21.15.116:0/4089357292 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16bc003800 con 0x55ea11145a40 2024-03-24T09:25:26.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d645f740 10 client.16593 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.580+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ea110aabd0 con 0x55ea11145a40 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16b4ff9640 1 -- 172.21.15.116:0/4089357292 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ea10dab7e0 con 0x55ea11145a40 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16b4ff9640 1 -- 172.21.15.116:0/4089357292 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f16bc005450 con 0x55ea11145a40 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d645f740 10 client.16593 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d645f740 10 client.16593 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d645f740 20 client.16593 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d645f740 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea11162ec0 0x55ea111652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d645f740 4 client.16593 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:26.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea10de8720 con 0x55ea11162ec0 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d162a640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea11162ec0 0x55ea111652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16d162a640 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea11162ec0 0x55ea111652e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=aad9e1f961a0bb10 server_cookie=333fabdde55b000a in_seq=0 out_seq=0 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.581+0000 7f16b4ff9640 10 client.16593 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16b4ff9640 1 -- 172.21.15.116:0/4089357292 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ea10de8720 con 0x55ea11162ec0 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16b4ff9640 10 client.16593 handle_command_reply: tid=0 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 client.16593 shutdown 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 msgr2=0x55ea111502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea111502a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f16bc035000 tx=0x7f16bc005fd0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 shutdown_connections 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11152d00 0x55ea111507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea11162ec0 0x55ea111652e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea111502a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 --2- 172.21.15.116:0/4089357292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11145120 0x55ea1114fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16b4ff9640 0 client.16593 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 >> 172.21.15.116:0/4089357292 conn(0x55ea1113fb20 msgr2=0x55ea11141870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 shutdown_connections 2024-03-24T09:25:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 1 -- 172.21.15.116:0/4089357292 wait complete. 2024-03-24T09:25:26.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.582+0000 7f16d645f740 20 client.16593 trim_cache size 0 max 16384 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.595+0000 7f16d645f740 1 Processor -- start 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.595+0000 7f16d645f740 1 -- start start 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d645f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 0x55ea11152490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d645f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145a40 0x55ea1113ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d645f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea11150600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d645f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea11144b60 con 0x55ea11145120 2024-03-24T09:25:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d645f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ea1113f200 con 0x55ea11152d00 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d645f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea11165960 con 0x55ea11145a40 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 0x55ea11152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 0x55ea11152490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43736/0 (socket says 172.21.15.116:43736) 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 -- 172.21.15.116:0/1939244996 learned_addr learned my addr 172.21.15.116:0/1939244996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea11150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d0e29640 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145a40 0x55ea1113ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 -- 172.21.15.116:0/1939244996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145a40 msgr2=0x55ea1113ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145a40 0x55ea1113ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 -- 172.21.15.116:0/1939244996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 msgr2=0x55ea11150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea11150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 -- 172.21.15.116:0/1939244996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea11046c80 con 0x55ea11145120 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea11150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16d162a640 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 0x55ea11152490 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f16c8000c00 tx=0x7f16c8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dfbdfb68e760d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16b6ffd640 1 -- 172.21.15.116:0/1939244996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8017040 con 0x55ea11145120 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16b6ffd640 1 -- 172.21.15.116:0/1939244996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f16c8018040 con 0x55ea11145120 2024-03-24T09:25:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.596+0000 7f16b6ffd640 1 -- 172.21.15.116:0/1939244996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c80223d0 con 0x55ea11145120 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 -- 172.21.15.116:0/1939244996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 msgr2=0x55ea11152490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 0x55ea11152490 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f16c8000c00 tx=0x7f16c8005f90 comp rx=0 tx=0).stop 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 -- 172.21.15.116:0/1939244996 shutdown_connections 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea11150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145a40 0x55ea1113ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 --2- 172.21.15.116:0/1939244996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145120 0x55ea11152490 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 -- 172.21.15.116:0/1939244996 >> 172.21.15.116:0/1939244996 conn(0x55ea1113fb20 msgr2=0x55ea11140c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 -- 172.21.15.116:0/1939244996 shutdown_connections 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 -- 172.21.15.116:0/1939244996 wait complete. 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 -- start start 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.597+0000 7f16d645f740 1 Processor -- start 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea11157300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea111c2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ea111c27a0 con 0x55ea11145a40 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ea111c2910 con 0x55ea11152d00 2024-03-24T09:25:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ea111c2ad0 con 0x55ea11145120 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 10 client.0 _ll_register_callbacks cb 0x7f16bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea11157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea11157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43746/0 (socket says 172.21.15.116:43746) 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 -- 172.21.15.116:0/976938611 learned_addr learned my addr 172.21.15.116:0/976938611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d162a640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d1e2b640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea111c2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 -- 172.21.15.116:0/976938611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 msgr2=0x55ea11156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 -- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 msgr2=0x55ea111c2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea111c2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 -- 172.21.15.116:0/976938611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea11041de0 con 0x55ea11145a40 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d0e29640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea11157300 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009930 tx=0x7f16bc050890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872b1bc3a1ad6734 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16b4ff9640 1 -- 172.21.15.116:0/976938611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc004410 con 0x55ea11145a40 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16b4ff9640 1 -- 172.21.15.116:0/976938611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f16bc004960 con 0x55ea11145a40 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.598+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ea1113f3c0 con 0x55ea11145a40 2024-03-24T09:25:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16b4ff9640 1 -- 172.21.15.116:0/976938611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16bc02cb80 con 0x55ea11145a40 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 10 client.16599 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f16c8012530 con 0x55ea11145a40 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16b4ff9640 1 -- 172.21.15.116:0/976938611 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ea1113f3c0 con 0x55ea11145a40 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16b4ff9640 1 -- 172.21.15.116:0/976938611 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f16bc004dc0 con 0x55ea11145a40 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 10 client.16599 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 10 client.16599 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 20 client.16599 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea111ca030 0x55ea111cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 4 client.16599 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f16c409a9b0 con 0x55ea111ca030 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d162a640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea111ca030 0x55ea111cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16b4ff9640 10 client.16599 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.599+0000 7f16d162a640 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea111ca030 0x55ea111cc450 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8b877aaba8f192bd server_cookie=5b94cf279f06be1a in_seq=0 out_seq=0 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16b4ff9640 1 -- 172.21.15.116:0/976938611 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f16c409a9b0 con 0x55ea111ca030 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16b4ff9640 10 client.16599 handle_command_reply: tid=0 2024-03-24T09:25:26.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 client.16599 shutdown 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 msgr2=0x55ea11157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea11157300 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009930 tx=0x7f16bc050890 comp rx=0 tx=0).stop 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 shutdown_connections 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ea11152d00 0x55ea111c2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55ea111ca030 0x55ea111cc450 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ea11145a40 0x55ea11157300 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 --2- 172.21.15.116:0/976938611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ea11145120 0x55ea11156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16b4ff9640 0 client.16599 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:26.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 >> 172.21.15.116:0/976938611 conn(0x55ea1113fb20 msgr2=0x55ea11141520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 shutdown_connections 2024-03-24T09:25:26.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 1 -- 172.21.15.116:0/976938611 wait complete. 2024-03-24T09:25:26.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d645f740 20 client.16599 trim_cache size 0 max 16384 2024-03-24T09:25:26.614 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:26.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f16b80a2230 msgr2=0x7f16b80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f16b80a2230 0x7f16b80a46f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f16bc005ab0 tx=0x7f16bc004750 comp rx=0 tx=0).stop 2024-03-24T09:25:26.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 msgr2=0x7f16cc197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 0x7f16cc197ae0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f16c400eb10 tx=0x7f16c400c2d0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 shutdown_connections 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16cc1076c0 0x7f16cc197ae0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16cc102070 0x7f16cc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f16b80a2230 0x7f16b80a46f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 --2- 172.21.15.116:0/4106207747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f16cc1016b0 0x7f16cc19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 >> 172.21.15.116:0/4106207747 conn(0x7f16cc0fd140 msgr2=0x7f16cc0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.600+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 shutdown_connections 2024-03-24T09:25:26.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.601+0000 7f16d388e640 1 -- 172.21.15.116:0/4106207747 wait complete. 2024-03-24T09:25:26.618 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 -- 172.21.15.116:0/2580781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 msgr2=0x7fc4e4101b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2580781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e4101b40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009930 tx=0x7fc4d802bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 -- 172.21.15.116:0/2580781068 shutdown_connections 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2580781068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4e4102080 0x7fc4e410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2580781068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e4101b40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2580781068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e41076d0 0x7fc4e4107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 -- 172.21.15.116:0/2580781068 >> 172.21.15.116:0/2580781068 conn(0x7fc4e40fd150 msgr2=0x7fc4e40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 -- 172.21.15.116:0/2580781068 shutdown_connections 2024-03-24T09:25:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 -- 172.21.15.116:0/2580781068 wait complete. 2024-03-24T09:25:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.720+0000 7fc4ea047640 1 Processor -- start 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 -- start start 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4e4102080 0x7fc4e419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e41076d0 0x7fc4e4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4198100 con 0x7fc4e41016c0 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4198270 con 0x7fc4e4102080 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4ea047640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4198430 con 0x7fc4e41076d0 2024-03-24T09:25:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e419d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43756/0 (socket says 172.21.15.116:43756) 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 -- 172.21.15.116:0/2339304228 learned_addr learned my addr 172.21.15.116:0/2339304228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e41076d0 0x7fc4e4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4e4102080 0x7fc4e419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 -- 172.21.15.116:0/2339304228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e41076d0 msgr2=0x7fc4e4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e41076d0 0x7fc4e4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 -- 172.21.15.116:0/2339304228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4e4102080 msgr2=0x7fc4e419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4e4102080 0x7fc4e419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 -- 172.21.15.116:0/2339304228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d4009490 con 0x7fc4e41016c0 2024-03-24T09:25:26.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.721+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e419d900 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400ea60 tx=0x7fc4d400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.722+0000 7fc4e0ff9640 1 -- 172.21.15.116:0/2339304228 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400cde0 con 0x7fc4e41016c0 2024-03-24T09:25:26.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.722+0000 7fc4e0ff9640 1 -- 172.21.15.116:0/2339304228 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4d40044d0 con 0x7fc4e41016c0 2024-03-24T09:25:26.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.722+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d8009560 con 0x7fc4e41016c0 2024-03-24T09:25:26.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.722+0000 7fc4e0ff9640 1 -- 172.21.15.116:0/2339304228 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4010690 con 0x7fc4e41016c0 2024-03-24T09:25:26.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.722+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e41a47c0 con 0x7fc4e41016c0 2024-03-24T09:25:26.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e0ff9640 1 -- 172.21.15.116:0/2339304228 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc4d4003e40 con 0x7fc4e41016c0 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 Processor -- start 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 -- start start 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbd87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbe13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbe18f0 0x55632cbe3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55632cb46840 con 0x55632cbd8d10 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55632cb43d70 con 0x55632cbd83f0 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55632cb3f550 con 0x55632cbe18f0 2024-03-24T09:25:26.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e0ff9640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4cc0a2230 0x7fc4cc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbd87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbe13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbd87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58182/0 (socket says 172.21.15.116:58182) 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e3fff640 1 -- 172.21.15.116:0/2242249421 learned_addr learned my addr 172.21.15.116:0/2242249421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbe18f0 0x55632cbe3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4cc0a2230 0x7fc4cc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.724+0000 7fc4e0ff9640 1 -- 172.21.15.116:0/2339304228 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4d4014070 con 0x7fc4e41016c0 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 -- 172.21.15.116:0/2242249421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbe18f0 msgr2=0x55632cbe3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbe18f0 0x55632cbe3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 -- 172.21.15.116:0/2242249421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 msgr2=0x55632cbe13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbe13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 -- 172.21.15.116:0/2242249421 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55632cad4de0 con 0x55632cbd83f0 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbe18f0 0x55632cbe3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4cc0a2230 0x7fc4cc0a46f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00fc90 tx=0x7fc4dc018000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbd87d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc011e00 tx=0x7fc4dc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=700b264797d16257 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4c9ffb640 1 -- 172.21.15.116:0/2242249421 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc011910 con 0x55632cbd83f0 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4c9ffb640 1 -- 172.21.15.116:0/2242249421 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4dc00ed10 con 0x55632cbd83f0 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.725+0000 7fc4c9ffb640 1 -- 172.21.15.116:0/2242249421 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc004290 con 0x55632cbd83f0 2024-03-24T09:25:26.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2242249421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 msgr2=0x55632cbd87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbd87d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc011e00 tx=0x7fc4dc00f790 comp rx=0 tx=0).stop 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2242249421 shutdown_connections 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbe18f0 0x55632cbe3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbe13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2242249421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbd87d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2242249421 >> 172.21.15.116:0/2242249421 conn(0x55632cbd2b20 msgr2=0x55632cbd2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2242249421 shutdown_connections 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2242249421 wait complete. 2024-03-24T09:25:26.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 -- start start 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 Processor -- start 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 0x55632cbde9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.726+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbdef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbdf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55632c812430 con 0x55632cbd83f0 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55632c7cee00 con 0x55632cbe18f0 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55632cac4590 con 0x55632cbd8d10 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 10 client.0 _ll_register_callbacks cb 0x55632cb33fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 0x55632cbde9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 0x55632cbde9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43784/0 (socket says 172.21.15.116:43784) 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbdf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 -- 172.21.15.116:0/1423136928 learned_addr learned my addr 172.21.15.116:0/1423136928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e2ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbdef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 -- 172.21.15.116:0/1423136928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 msgr2=0x55632cbdef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbdef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 -- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 msgr2=0x55632cbdf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbdf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 -- 172.21.15.116:0/1423136928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4dc019040 con 0x55632cbd83f0 2024-03-24T09:25:26.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbdef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 0x55632cbde9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40972f0 tx=0x7fc4d406a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72baed8ed19a9644 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/1423136928 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d409efe0 con 0x55632cbd83f0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/1423136928 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4d40a3630 con 0x55632cbd83f0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55632c83e7e0 con 0x55632cbd83f0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/1423136928 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc4d40b71a0 con 0x55632cbd83f0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 10 client.16611 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55632cb3dbd0 con 0x55632cbd83f0 2024-03-24T09:25:26.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.727+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/1423136928 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55632c83e7e0 con 0x55632cbd83f0 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/1423136928 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc4d40b75e0 con 0x55632cbd83f0 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4ecc29740 10 client.16611 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4ecc29740 10 client.16611 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4ecc29740 20 client.16611 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cbf3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4ecc29740 4 client.16611 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55632c87b720 con 0x55632cbf19a0 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cbf3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4c2ffd640 10 client.16611 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.728+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cbf3d90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9ee7a665f91eba06 server_cookie=d5ec433a3cb5ca55 in_seq=0 out_seq=0 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/1423136928 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55632c87b720 con 0x55632cbf19a0 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4c2ffd640 10 client.16611 handle_command_reply: tid=0 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 client.16611 shutdown 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 msgr2=0x55632cbde9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 0x55632cbde9c0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40972f0 tx=0x7fc4d406a030 comp rx=0 tx=0).stop 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 shutdown_connections 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cbf3d90 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbdf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4c2ffd640 0 client.16611 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbdef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/1423136928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd83f0 0x55632cbde9c0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.729+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 >> 172.21.15.116:0/1423136928 conn(0x55632cbd2b20 msgr2=0x55632cbd4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.730+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 shutdown_connections 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.730+0000 7fc4ecc29740 1 -- 172.21.15.116:0/1423136928 wait complete. 2024-03-24T09:25:26.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.730+0000 7fc4ecc29740 20 client.16611 trim_cache size 0 max 16384 2024-03-24T09:25:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 Processor -- start 2024-03-24T09:25:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 -- start start 2024-03-24T09:25:26.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd83f0 0x55632cbed120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbcdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbf9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55632cbce090 con 0x55632cbd8d10 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55632cbd7b90 con 0x55632cbe18f0 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55632cbe3ce0 con 0x55632cbd83f0 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbcdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbcdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43788/0 (socket says 172.21.15.116:43788) 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 -- 172.21.15.116:0/2710400969 learned_addr learned my addr 172.21.15.116:0/2710400969 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbf9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd83f0 0x55632cbed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 -- 172.21.15.116:0/2710400969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd83f0 msgr2=0x55632cbed120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd83f0 0x55632cbed120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 -- 172.21.15.116:0/2710400969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 msgr2=0x55632cbf9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbf9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 -- 172.21.15.116:0/2710400969 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55632cad9c80 con 0x55632cbd8d10 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.743+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbcdb50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009930 tx=0x7fc4d8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f04cd1c2c6b972 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4c97fa640 1 -- 172.21.15.116:0/2710400969 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d804abd0 con 0x55632cbd8d10 2024-03-24T09:25:26.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4c97fa640 1 -- 172.21.15.116:0/2710400969 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4d802de90 con 0x55632cbd8d10 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4c97fa640 1 -- 172.21.15.116:0/2710400969 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d804a590 con 0x55632cbd8d10 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2710400969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 msgr2=0x55632cbcdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbcdb50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009930 tx=0x7fc4d8009260 comp rx=0 tx=0).stop 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2710400969 shutdown_connections 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbe18f0 0x55632cbf9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbd8d10 0x55632cbcdb50 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/2710400969 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd83f0 0x55632cbed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2710400969 >> 172.21.15.116:0/2710400969 conn(0x55632cbd2b20 msgr2=0x55632cbd1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2710400969 shutdown_connections 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.744+0000 7fc4ecc29740 1 -- 172.21.15.116:0/2710400969 wait complete. 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 -- start start 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 Processor -- start 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbe5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbe6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 0x55632cc4fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55632cc50220 con 0x55632cbe18f0 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55632cc50390 con 0x55632cbd83f0 2024-03-24T09:25:26.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55632cc50550 con 0x55632cbd8d10 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4ecc29740 10 client.0 _ll_register_callbacks cb 0x7fc4d40971d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 0x55632cc4fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbe5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 0x55632cc4fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43794/0 (socket says 172.21.15.116:43794) 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4e3fff640 1 -- 172.21.15.116:0/4086493982 learned_addr learned my addr 172.21.15.116:0/4086493982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4e37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbe5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58204/0 (socket says 172.21.15.116:58204) 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.745+0000 7fc4e2ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbe6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e3fff640 1 -- 172.21.15.116:0/4086493982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 msgr2=0x55632cbe6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbe6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e3fff640 1 -- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 msgr2=0x55632cbe5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbe5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e3fff640 1 -- 172.21.15.116:0/4086493982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d8030070 con 0x55632cbe18f0 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e2ffd640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbe6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbe5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4e3fff640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 0x55632cc4fc60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc011e00 tx=0x7fc4dc011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72b23cc85ccc7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/4086493982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc02b040 con 0x55632cbe18f0 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/4086493982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4dc00ee80 con 0x55632cbe18f0 2024-03-24T09:25:26.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4dc00ed10 con 0x55632cbe18f0 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/4086493982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc4dc003970 con 0x55632cbe18f0 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4ecc29740 10 client.16617 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55632caff920 con 0x55632cbe18f0 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/4086493982 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4dc00ed10 con 0x55632cbe18f0 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/4086493982 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc4dc005890 con 0x55632cbe18f0 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4ecc29740 10 client.16617 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4ecc29740 10 client.16617 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.746+0000 7fc4ecc29740 20 client.16617 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cc5af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 4 client.16617 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55632c8ca890 con 0x55632cbf19a0 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cc5af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4c2ffd640 10 client.16617 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4e37fe640 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cc5af60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc0c4236c170690b server_cookie=23f3d3afa9cea768 in_seq=0 out_seq=0 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4c2ffd640 1 -- 172.21.15.116:0/4086493982 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55632c8ca890 con 0x55632cbf19a0 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4c2ffd640 10 client.16617 handle_command_reply: tid=0 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 client.16617 shutdown 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 msgr2=0x55632cc4fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 0x55632cc4fc60 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc011e00 tx=0x7fc4dc011f50 comp rx=0 tx=0).stop 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 shutdown_connections 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55632cbf19a0 0x55632cc5af60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55632cbe18f0 0x55632cc4fc60 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55632cbd8d10 0x55632cbe6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 --2- 172.21.15.116:0/4086493982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55632cbd83f0 0x55632cbe5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4c2ffd640 0 client.16617 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 >> 172.21.15.116:0/4086493982 conn(0x55632cbd2b20 msgr2=0x55632cbd3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.747+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 shutdown_connections 2024-03-24T09:25:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ecc29740 1 -- 172.21.15.116:0/4086493982 wait complete. 2024-03-24T09:25:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ecc29740 20 client.16617 trim_cache size 0 max 16384 2024-03-24T09:25:26.759 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4cc0a2230 msgr2=0x7fc4cc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4cc0a2230 0x7fc4cc0a46f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00fc90 tx=0x7fc4dc018000 comp rx=0 tx=0).stop 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 msgr2=0x7fc4e419d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e419d900 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400ea60 tx=0x7fc4d400ef20 comp rx=0 tx=0).stop 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 shutdown_connections 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4cc0a2230 0x7fc4cc0a46f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc4e41076d0 0x7fc4e4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4e4102080 0x7fc4e419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 --2- 172.21.15.116:0/2339304228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc4e41016c0 0x7fc4e419d900 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 >> 172.21.15.116:0/2339304228 conn(0x7fc4e40fd150 msgr2=0x7fc4e40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 shutdown_connections 2024-03-24T09:25:26.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.748+0000 7fc4ea047640 1 -- 172.21.15.116:0/2339304228 wait complete. 2024-03-24T09:25:26.762 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:25:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 -- 172.21.15.116:0/493426521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73741055d0 msgr2=0x7f73741059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 --2- 172.21.15.116:0/493426521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73741055d0 0x7f73741059b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7364009ca0 tx=0x7f736402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 -- 172.21.15.116:0/493426521 shutdown_connections 2024-03-24T09:25:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 --2- 172.21.15.116:0/493426521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7374100f70 0x7f737410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 --2- 172.21.15.116:0/493426521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73741005b0 0x7f7374100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 --2- 172.21.15.116:0/493426521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73741055d0 0x7f73741059b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 -- 172.21.15.116:0/493426521 >> 172.21.15.116:0/493426521 conn(0x7f73740776a0 msgr2=0x7f7374077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 -- 172.21.15.116:0/493426521 shutdown_connections 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.863+0000 7f737955c640 1 -- 172.21.15.116:0/493426521 wait complete. 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 Processor -- start 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 -- start start 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73741005b0 0x7f7374198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7374100f70 0x7f7374198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 0x7f73741a3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73741997d0 con 0x7f73741055d0 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7374199940 con 0x7f7374100f70 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f737955c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f73741a40b0 con 0x7f73741005b0 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 0x7f73741a3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f7372ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73741005b0 0x7f7374198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 0x7f73741a3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43804/0 (socket says 172.21.15.116:43804) 2024-03-24T09:25:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 -- 172.21.15.116:0/3868272644 learned_addr learned my addr 172.21.15.116:0/3868272644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73727fc640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7374100f70 0x7f7374198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 -- 172.21.15.116:0/3868272644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73741005b0 msgr2=0x7f7374198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73741005b0 0x7f7374198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 -- 172.21.15.116:0/3868272644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7374100f70 msgr2=0x7f7374198fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.864+0000 7f73737fe640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7374100f70 0x7f7374198fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f73737fe640 1 -- 172.21.15.116:0/3868272644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7364009900 con 0x7f73741055d0 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f73737fe640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 0x7f73741a3b20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f736c00e970 tx=0x7f736c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f735bfff640 1 -- 172.21.15.116:0/3868272644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c009720 con 0x7f73741055d0 2024-03-24T09:25:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73741a4390 con 0x7f73741055d0 2024-03-24T09:25:26.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f735bfff640 1 -- 172.21.15.116:0/3868272644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f736c0054d0 con 0x7f73741055d0 2024-03-24T09:25:26.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f735bfff640 1 -- 172.21.15.116:0/3868272644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c010430 con 0x7f73741055d0 2024-03-24T09:25:26.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.865+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73741a48a0 con 0x7f73741055d0 2024-03-24T09:25:26.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.867+0000 7f735bfff640 1 -- 172.21.15.116:0/3868272644 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f736c004f30 con 0x7f73741055d0 2024-03-24T09:25:26.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.867+0000 7f737c20f740 1 Processor -- start 2024-03-24T09:25:26.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 -- start start 2024-03-24T09:25:26.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f7809fbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f78095e840 con 0x55f7809f98f0 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f78095bd70 con 0x55f7809f03f0 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f737c20f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f780957550 con 0x55f7809f0d10 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f7809fbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f7372ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f7809fbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43818/0 (socket says 172.21.15.116:43818) 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f735bfff640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f735c0a2230 0x7f735c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 -- 172.21.15.116:0/680934472 learned_addr learned my addr 172.21.15.116:0/680934472 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f7372ffd640 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40932/0 (socket says 172.21.15.116:40932) 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73737fe640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f735c0a2230 0x7f735c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 -- 172.21.15.116:0/680934472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 msgr2=0x55f7809f93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 -- 172.21.15.116:0/680934472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 msgr2=0x55f7809f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 -- 172.21.15.116:0/680934472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7808ecde0 con 0x55f7809f98f0 2024-03-24T09:25:26.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f7372ffd640 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f735bfff640 1 -- 172.21.15.116:0/3868272644 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f736c014070 con 0x7f73741055d0 2024-03-24T09:25:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.868+0000 7f73727fc640 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f7809fbce0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7368011810 tx=0x7f73680097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb45e78fc7b72172 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f73597fa640 1 -- 172.21.15.116:0/680934472 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7368004030 con 0x55f7809f98f0 2024-03-24T09:25:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f73737fe640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f735c0a2230 0x7f735c0a46f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f736c09ad20 tx=0x7f736c066330 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f73597fa640 1 -- 172.21.15.116:0/680934472 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f736800f7c0 con 0x55f7809f98f0 2024-03-24T09:25:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f73597fa640 1 -- 172.21.15.116:0/680934472 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7368003970 con 0x55f7809f98f0 2024-03-24T09:25:26.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 -- 172.21.15.116:0/680934472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 msgr2=0x55f7809fbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f7809fbce0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f7368011810 tx=0x7f73680097d0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 -- 172.21.15.116:0/680934472 shutdown_connections 2024-03-24T09:25:26.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f7809fbce0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 --2- 172.21.15.116:0/680934472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 -- 172.21.15.116:0/680934472 >> 172.21.15.116:0/680934472 conn(0x55f7809eab20 msgr2=0x55f7809eaf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 -- 172.21.15.116:0/680934472 shutdown_connections 2024-03-24T09:25:26.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.869+0000 7f737c20f740 1 -- 172.21.15.116:0/680934472 wait complete. 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 -- start start 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 Processor -- start 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f7809f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f7809f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f78062a430 con 0x55f7809f0d10 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f7805e6e00 con 0x55f7809f98f0 2024-03-24T09:25:26.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f7808dc590 con 0x55f7809f03f0 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f737c20f740 10 client.0 _ll_register_callbacks cb 0x55f78094bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f7809f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f7809f6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43822/0 (socket says 172.21.15.116:43822) 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 -- 172.21.15.116:0/2530770237 learned_addr learned my addr 172.21.15.116:0/2530770237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f73727fc640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f7809f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f73727fc640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 -- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 msgr2=0x55f7809f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f7809f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 -- 172.21.15.116:0/2530770237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 msgr2=0x55f7809f7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.870+0000 7f7372ffd640 1 -- 172.21.15.116:0/2530770237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7368019040 con 0x55f7809f0d10 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7372ffd640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f7809f6ee0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7364004af0 tx=0x7f7364034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366697b6394b3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7356ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7356ff5640 1 -- 172.21.15.116:0/2530770237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364032820 con 0x55f7809f0d10 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7356ff5640 1 -- 172.21.15.116:0/2530770237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7364046070 con 0x55f7809f0d10 2024-03-24T09:25:26.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f7806567e0 con 0x55f7809f0d10 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7356ff5640 1 -- 172.21.15.116:0/2530770237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7364032460 con 0x55f7809f0d10 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 10 client.16629 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f780955bd0 con 0x55f7809f0d10 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7356ff5640 1 -- 172.21.15.116:0/2530770237 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f7806567e0 con 0x55f7809f0d10 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f7356ff5640 1 -- 172.21.15.116:0/2530770237 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f73640043c0 con 0x55f7809f0d10 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 10 client.16629 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 10 client.16629 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:25:26.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 20 client.16629 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a09ad0 0x55f780a0bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 4 client.16629 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f780693720 con 0x55f780a09ad0 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.871+0000 7f73727fc640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a09ad0 0x55f780a0bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f7356ff5640 10 client.16629 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f73727fc640 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a09ad0 0x55f780a0bef0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=790a439a3d915e90 server_cookie=38e2509340a782c9 in_seq=0 out_seq=0 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f7356ff5640 1 -- 172.21.15.116:0/2530770237 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f780693720 con 0x55f780a09ad0 2024-03-24T09:25:26.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f7356ff5640 10 client.16629 handle_command_reply: tid=0 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f737c20f740 1 client.16629 shutdown 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 msgr2=0x55f7809f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.872+0000 7f737c20f740 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f7809f6ee0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7364004af0 tx=0x7f7364034670 comp rx=0 tx=0).stop 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 shutdown_connections 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a09ad0 0x55f780a0bef0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f7809f6ee0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f7356ff5640 0 client.16629 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:26.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 --2- 172.21.15.116:0/2530770237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f7809f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 >> 172.21.15.116:0/2530770237 conn(0x55f7809eab20 msgr2=0x55f7809ec870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 shutdown_connections 2024-03-24T09:25:26.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 1 -- 172.21.15.116:0/2530770237 wait complete. 2024-03-24T09:25:26.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.873+0000 7f737c20f740 20 client.16629 trim_cache size 0 max 16384 2024-03-24T09:25:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.885+0000 7f737c20f740 1 Processor -- start 2024-03-24T09:25:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 -- start start 2024-03-24T09:25:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f780a05180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f780a11780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f780a064f0 con 0x55f7809f0d10 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f780a0c550 con 0x55f7809f98f0 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f737c20f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f7809efdc0 con 0x55f7809f03f0 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f780a11780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f780a11780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43838/0 (socket says 172.21.15.116:43838) 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 -- 172.21.15.116:0/3761341434 learned_addr learned my addr 172.21.15.116:0/3761341434 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f73727fc640 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f780a05180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f73727fc640 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 -- 172.21.15.116:0/3761341434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 msgr2=0x55f780a05180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f780a05180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 -- 172.21.15.116:0/3761341434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 msgr2=0x55f7809f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 -- 172.21.15.116:0/3761341434 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7808f1c80 con 0x55f7809f0d10 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f73727fc640 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7372ffd640 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f780a11780 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7364002230 tx=0x7f7364005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9792924dd174342 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7358ff9640 1 -- 172.21.15.116:0/3761341434 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736402c7f0 con 0x55f7809f0d10 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7358ff9640 1 -- 172.21.15.116:0/3761341434 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7364044070 con 0x55f7809f0d10 2024-03-24T09:25:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.886+0000 7f7358ff9640 1 -- 172.21.15.116:0/3761341434 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736404abe0 con 0x55f7809f0d10 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- 172.21.15.116:0/3761341434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 msgr2=0x55f780a11780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f780a11780 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f7364002230 tx=0x7f7364005d80 comp rx=0 tx=0).stop 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- 172.21.15.116:0/3761341434 shutdown_connections 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f98f0 0x55f7809f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f0d10 0x55f780a11780 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- 172.21.15.116:0/3761341434 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f03f0 0x55f780a05180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- 172.21.15.116:0/3761341434 >> 172.21.15.116:0/3761341434 conn(0x55f7809eab20 msgr2=0x55f7809e9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- 172.21.15.116:0/3761341434 shutdown_connections 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- 172.21.15.116:0/3761341434 wait complete. 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- start start 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 Processor -- start 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f780a6edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7809f45a0 con 0x55f7809f98f0 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f7809f4710 con 0x55f7809f03f0 2024-03-24T09:25:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.887+0000 7f737c20f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f780a6f2f0 con 0x55f7809f0d10 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f737c20f740 10 client.0 _ll_register_callbacks cb 0x7f7368005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f780a6edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f780a6edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43840/0 (socket says 172.21.15.116:43840) 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 -- 172.21.15.116:0/3890037275 learned_addr learned my addr 172.21.15.116:0/3890037275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f7372ffd640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 -- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 msgr2=0x55f7809f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 -- 172.21.15.116:0/3890037275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 msgr2=0x55f7809f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 -- 172.21.15.116:0/3890037275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f780955bd0 con 0x55f7809f98f0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f73727fc640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f780a6edb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7368010010 tx=0x7f7368009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112c04fc7d46f9c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f7356ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f7356ff5640 1 -- 172.21.15.116:0/3890037275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7368011110 con 0x55f7809f98f0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f7356ff5640 1 -- 172.21.15.116:0/3890037275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7368011270 con 0x55f7809f98f0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f736800f7c0 con 0x55f7809f98f0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f7356ff5640 1 -- 172.21.15.116:0/3890037275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f73680050d0 con 0x55f7809f98f0 2024-03-24T09:25:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f737c20f740 10 client.16635 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f780917920 con 0x55f7809f98f0 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.888+0000 7f7356ff5640 1 -- 172.21.15.116:0/3890037275 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7368038020 con 0x55f7809f98f0 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f7356ff5640 1 -- 172.21.15.116:0/3890037275 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f73680035f0 con 0x55f7809f98f0 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 10 client.16635 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 10 client.16635 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 20 client.16635 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a70a80 0x55f780a72ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:26.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 4 client.16635 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7806e2890 con 0x55f780a70a80 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f7372ffd640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a70a80 0x55f780a72ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f7356ff5640 10 client.16635 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f7372ffd640 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a70a80 0x55f780a72ea0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=12501334c4fa00c2 server_cookie=e741e5e111b25563 in_seq=0 out_seq=0 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f7356ff5640 1 -- 172.21.15.116:0/3890037275 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7806e2890 con 0x55f780a70a80 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f7356ff5640 10 client.16635 handle_command_reply: tid=0 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 client.16635 shutdown 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 msgr2=0x55f780a6edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f780a6edb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7368010010 tx=0x7f7368009b70 comp rx=0 tx=0).stop 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 shutdown_connections 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7809f98f0 0x55f780a6edb0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7809f0d10 0x55f7809f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f780a70a80 0x55f780a72ea0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.889+0000 7f737c20f740 1 --2- 172.21.15.116:0/3890037275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7809f03f0 0x55f7809f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f7356ff5640 0 client.16635 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 >> 172.21.15.116:0/3890037275 conn(0x55f7809eab20 msgr2=0x55f7809f8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 shutdown_connections 2024-03-24T09:25:26.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737c20f740 1 -- 172.21.15.116:0/3890037275 wait complete. 2024-03-24T09:25:26.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737c20f740 20 client.16635 trim_cache size 0 max 16384 2024-03-24T09:25:26.905 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:26.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f735c0a2230 msgr2=0x7f735c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f735c0a2230 0x7f735c0a46f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f736c09ad20 tx=0x7f736c066330 comp rx=0 tx=0).stop 2024-03-24T09:25:26.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 msgr2=0x7f73741a3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 0x7f73741a3b20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f736c00e970 tx=0x7f736c00ed70 comp rx=0 tx=0).stop 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 shutdown_connections 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73741055d0 0x7f73741a3b20 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7374100f70 0x7f7374198fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f735c0a2230 0x7f735c0a46f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 --2- 172.21.15.116:0/3868272644 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73741005b0 0x7f7374198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 >> 172.21.15.116:0/3868272644 conn(0x7f73740776a0 msgr2=0x7f73740fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 shutdown_connections 2024-03-24T09:25:26.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:26.890+0000 7f737955c640 1 -- 172.21.15.116:0/3868272644 wait complete. 2024-03-24T09:25:26.908 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:25:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.010+0000 7f9266c0e640 1 -- 172.21.15.116:0/1347223454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260106660 msgr2=0x7f9260106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.010+0000 7f9266c0e640 1 --2- 172.21.15.116:0/1347223454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260106660 0x7f9260106a40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9250009890 tx=0x7f925002bb70 comp rx=0 tx=0).stop 2024-03-24T09:25:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.010+0000 7f9266c0e640 1 -- 172.21.15.116:0/1347223454 shutdown_connections 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.010+0000 7f9266c0e640 1 --2- 172.21.15.116:0/1347223454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9260100f60 0x7f926010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.010+0000 7f9266c0e640 1 --2- 172.21.15.116:0/1347223454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92601005a0 0x7f9260100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 --2- 172.21.15.116:0/1347223454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260106660 0x7f9260106a40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- 172.21.15.116:0/1347223454 >> 172.21.15.116:0/1347223454 conn(0x7f9260077710 msgr2=0x7f9260077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- 172.21.15.116:0/1347223454 shutdown_connections 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- 172.21.15.116:0/1347223454 wait complete. 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 Processor -- start 2024-03-24T09:25:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- start start 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 0x7f9260198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260100f60 0x7f9260198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9260106660 0x7f92601a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9260199850 con 0x7f9260106660 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f92601a4050 con 0x7f9260100f60 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f9266c0e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f92601a4190 con 0x7f92601005a0 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f92651ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9260106660 0x7f92601a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f92649aa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 0x7f9260198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f925ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260100f60 0x7f9260198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f92649aa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 0x7f9260198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:40980/0 (socket says 172.21.15.116:40980) 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.011+0000 7f92649aa640 1 -- 172.21.15.116:0/3545144897 learned_addr learned my addr 172.21.15.116:0/3545144897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92651ab640 1 -- 172.21.15.116:0/3545144897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 msgr2=0x7f9260198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92651ab640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 0x7f9260198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92651ab640 1 -- 172.21.15.116:0/3545144897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260100f60 msgr2=0x7f9260198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92651ab640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260100f60 0x7f9260198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92651ab640 1 -- 172.21.15.116:0/3545144897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9250009490 con 0x7f9260106660 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92649aa640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 0x7f9260198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f92651ab640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9260106660 0x7f92601a3b10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f925800ea80 tx=0x7f925800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f925dffb640 1 -- 172.21.15.116:0/3545144897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925800cd80 con 0x7f9260106660 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92601a43f0 con 0x7f9260106660 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f925dffb640 1 -- 172.21.15.116:0/3545144897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9258005020 con 0x7f9260106660 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f925dffb640 1 -- 172.21.15.116:0/3545144897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925800cee0 con 0x7f9260106660 2024-03-24T09:25:27.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.012+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92601a4950 con 0x7f9260106660 2024-03-24T09:25:27.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.014+0000 7f925dffb640 1 -- 172.21.15.116:0/3545144897 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9258005190 con 0x7f9260106660 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f925dffb640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92480a2230 0x7f92480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 Processor -- start 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 -- start start 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92649aa640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92480a2230 0x7f92480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f925dffb640 1 -- 172.21.15.116:0/3545144897 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9258014070 con 0x7f9260106660 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x56513155e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x5651315673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131569ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5651314cc840 con 0x56513155e3f0 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5651314c9d70 con 0x5651315678f0 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92697ef740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5651314c5550 con 0x56513155ed10 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92649aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x56513155e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92649aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x56513155e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43880/0 (socket says 172.21.15.116:43880) 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f925ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x5651315673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92649aa640 1 -- 172.21.15.116:0/356386978 learned_addr learned my addr 172.21.15.116:0/356386978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92651ab640 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131569ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92651ab640 1 -- 172.21.15.116:0/356386978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 msgr2=0x5651315673b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92651ab640 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x5651315673b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92649aa640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92480a2230 0x7f92480a46f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9250009860 tx=0x7f9250004290 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92651ab640 1 -- 172.21.15.116:0/356386978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 msgr2=0x56513155e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92651ab640 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x56513155e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f92651ab640 1 -- 172.21.15.116:0/356386978 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56513145ade0 con 0x5651315678f0 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.015+0000 7f925ffff640 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x5651315673b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92651ab640 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131569ce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f92580cbde0 tx=0x7f925809aaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523a74892e89258b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f9246ffd640 1 -- 172.21.15.116:0/356386978 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925809b780 con 0x5651315678f0 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f9246ffd640 1 -- 172.21.15.116:0/356386978 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f925809b080 con 0x5651315678f0 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f9246ffd640 1 -- 172.21.15.116:0/356386978 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580976f0 con 0x5651315678f0 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 -- 172.21.15.116:0/356386978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 msgr2=0x565131569ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131569ce0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f92580cbde0 tx=0x7f925809aaa0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 -- 172.21.15.116:0/356386978 shutdown_connections 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131569ce0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x5651315673b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 --2- 172.21.15.116:0/356386978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x56513155e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 -- 172.21.15.116:0/356386978 >> 172.21.15.116:0/356386978 conn(0x565131558b20 msgr2=0x565131558f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 -- 172.21.15.116:0/356386978 shutdown_connections 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.016+0000 7f92697ef740 1 -- 172.21.15.116:0/356386978 wait complete. 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 -- start start 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 Processor -- start 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x565131564980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131564ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131565400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565131198430 con 0x56513155e3f0 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x565131154e00 con 0x5651315678f0 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56513144a590 con 0x56513155ed10 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92697ef740 10 client.0 _ll_register_callbacks cb 0x5651314b9fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92649aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x565131564980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92649aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x565131564980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43884/0 (socket says 172.21.15.116:43884) 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92649aa640 1 -- 172.21.15.116:0/3253298214 learned_addr learned my addr 172.21.15.116:0/3253298214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f92651ab640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131565400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.017+0000 7f925ffff640 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131564ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92649aa640 1 -- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 msgr2=0x565131564ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92649aa640 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131564ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92649aa640 1 -- 172.21.15.116:0/3253298214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 msgr2=0x565131565400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92649aa640 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131565400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92649aa640 1 -- 172.21.15.116:0/3253298214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92601a4950 con 0x56513155e3f0 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92649aa640 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x565131564980 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9250037040 tx=0x7f9250045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c613f2bf77232ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f923ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f923ffff640 1 -- 172.21.15.116:0/3253298214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925002c920 con 0x56513155e3f0 2024-03-24T09:25:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f923ffff640 1 -- 172.21.15.116:0/3253298214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f92500348b0 con 0x56513155e3f0 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5651311c47e0 con 0x56513155e3f0 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f923ffff640 1 -- 172.21.15.116:0/3253298214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f925002ca80 con 0x56513155e3f0 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 10 client.16647 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5651314c3bd0 con 0x56513155e3f0 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f923ffff640 1 -- 172.21.15.116:0/3253298214 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5651311c47e0 con 0x56513155e3f0 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f923ffff640 1 -- 172.21.15.116:0/3253298214 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f9250053070 con 0x56513155e3f0 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 10 client.16647 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 10 client.16647 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:25:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 20 client.16647 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x565131577b00 0x565131579f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 4 client.16647 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.018+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565131201720 con 0x565131577b00 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.019+0000 7f925ffff640 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x565131577b00 0x565131579f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.019+0000 7f925ffff640 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x565131577b00 0x565131579f20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=47790aa899729e1e server_cookie=c82b483e827e3b61 in_seq=0 out_seq=0 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.019+0000 7f923ffff640 10 client.16647 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f923ffff640 1 -- 172.21.15.116:0/3253298214 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565131201720 con 0x565131577b00 2024-03-24T09:25:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f923ffff640 10 client.16647 handle_command_reply: tid=0 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 client.16647 shutdown 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 msgr2=0x565131564980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x565131564980 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f9250037040 tx=0x7f9250045040 comp rx=0 tx=0).stop 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 shutdown_connections 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x565131577b00 0x565131579f20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5651315678f0 0x565131565400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131564ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 --2- 172.21.15.116:0/3253298214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155e3f0 0x565131564980 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f923ffff640 0 client.16647 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 >> 172.21.15.116:0/3253298214 conn(0x565131558b20 msgr2=0x56513155a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 shutdown_connections 2024-03-24T09:25:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 1 -- 172.21.15.116:0/3253298214 wait complete. 2024-03-24T09:25:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.020+0000 7f92697ef740 20 client.16647 trim_cache size 0 max 16384 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 Processor -- start 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 -- start start 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x565131573180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131565220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 0x56513157f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565131549ea0 con 0x5651315678f0 2024-03-24T09:25:27.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56513155d740 con 0x56513155e3f0 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92697ef740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x565131562ed0 con 0x56513155ed10 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 0x56513157f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 0x56513157f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43886/0 (socket says 172.21.15.116:43886) 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 -- 172.21.15.116:0/1877625741 learned_addr learned my addr 172.21.15.116:0/1877625741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92649aa640 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x565131573180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f925ffff640 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131565220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 -- 172.21.15.116:0/1877625741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 msgr2=0x565131565220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131565220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 -- 172.21.15.116:0/1877625741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 msgr2=0x565131573180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x565131573180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 -- 172.21.15.116:0/1877625741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56513145fc80 con 0x5651315678f0 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.033+0000 7f92651ab640 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 0x56513157f7d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f92580974e0 tx=0x7f925809f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7e850246a3cb96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92467fc640 1 -- 172.21.15.116:0/1877625741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925809b780 con 0x5651315678f0 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92467fc640 1 -- 172.21.15.116:0/1877625741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f92580a3180 con 0x5651315678f0 2024-03-24T09:25:27.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92467fc640 1 -- 172.21.15.116:0/1877625741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580cc060 con 0x5651315678f0 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 -- 172.21.15.116:0/1877625741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 msgr2=0x56513157f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 0x56513157f7d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f92580974e0 tx=0x7f925809f040 comp rx=0 tx=0).stop 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 -- 172.21.15.116:0/1877625741 shutdown_connections 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5651315678f0 0x56513157f7d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56513155ed10 0x565131565220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 --2- 172.21.15.116:0/1877625741 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x565131573180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 -- 172.21.15.116:0/1877625741 >> 172.21.15.116:0/1877625741 conn(0x565131558b20 msgr2=0x56513155a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 -- 172.21.15.116:0/1877625741 shutdown_connections 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 -- 172.21.15.116:0/1877625741 wait complete. 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.034+0000 7f92697ef740 1 -- start start 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 Processor -- start 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x56513156f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155ed10 0x56513156bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5651315678f0 0x56513156c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56513156c560 con 0x56513155ed10 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56513156c6d0 con 0x56513155e3f0 2024-03-24T09:25:27.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56513156c890 con 0x5651315678f0 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92697ef740 10 client.0 _ll_register_callbacks cb 0x7f92580093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x56513156f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f925ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155ed10 0x56513156bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x56513156f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58302/0 (socket says 172.21.15.116:58302) 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 -- 172.21.15.116:0/3031212896 learned_addr learned my addr 172.21.15.116:0/3031212896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92651ab640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5651315678f0 0x56513156c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 -- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5651315678f0 msgr2=0x56513156c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5651315678f0 0x56513156c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 -- 172.21.15.116:0/3031212896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155ed10 msgr2=0x56513156bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155ed10 0x56513156bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f92649aa640 1 -- 172.21.15.116:0/3031212896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56513145ade0 con 0x56513155e3f0 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.035+0000 7f925ffff640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155ed10 0x56513156bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92649aa640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x56513156f760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9250037040 tx=0x7f92500096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8cf8d81e16bdc76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f923ffff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f923ffff640 1 -- 172.21.15.116:0/3031212896 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925002c270 con 0x56513155e3f0 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f923ffff640 1 -- 172.21.15.116:0/3031212896 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f925002c720 con 0x56513155e3f0 2024-03-24T09:25:27.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f925809b080 con 0x56513155e3f0 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f923ffff640 1 -- 172.21.15.116:0/3031212896 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f925002c8e0 con 0x56513155e3f0 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 10 client.26480 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x565131485920 con 0x56513155e3f0 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f923ffff640 1 -- 172.21.15.116:0/3031212896 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f925809b080 con 0x56513155e3f0 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f923ffff640 1 -- 172.21.15.116:0/3031212896 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f9250053040 con 0x56513155e3f0 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 10 client.26480 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 10 client.26480 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:25:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 20 client.26480 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5651315defd0 0x5651315e13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 4 client.26480 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x565131250890 con 0x5651315defd0 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.036+0000 7f925ffff640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5651315defd0 0x5651315e13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f923ffff640 10 client.26480 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f925ffff640 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5651315defd0 0x5651315e13f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c507b5726617238b server_cookie=843330b268b97c99 in_seq=0 out_seq=0 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f923ffff640 1 -- 172.21.15.116:0/3031212896 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x565131250890 con 0x5651315defd0 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f923ffff640 10 client.26480 handle_command_reply: tid=0 2024-03-24T09:25:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 client.26480 shutdown 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 msgr2=0x56513156f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x56513156f760 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9250037040 tx=0x7f92500096b0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 shutdown_connections 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5651315defd0 0x5651315e13f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5651315678f0 0x56513156c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f923ffff640 0 client.26480 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56513155ed10 0x56513156bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 --2- 172.21.15.116:0/3031212896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56513155e3f0 0x56513156f760 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 >> 172.21.15.116:0/3031212896 conn(0x565131558b20 msgr2=0x565131557430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 shutdown_connections 2024-03-24T09:25:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 1 -- 172.21.15.116:0/3031212896 wait complete. 2024-03-24T09:25:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.037+0000 7f92697ef740 20 client.26480 trim_cache size 0 max 16384 2024-03-24T09:25:27.051 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92480a2230 msgr2=0x7f92480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92480a2230 0x7f92480a46f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9250009860 tx=0x7f9250004290 comp rx=0 tx=0).stop 2024-03-24T09:25:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9260106660 msgr2=0x7f92601a3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9260106660 0x7f92601a3b10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f925800ea80 tx=0x7f925800ee80 comp rx=0 tx=0).stop 2024-03-24T09:25:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 shutdown_connections 2024-03-24T09:25:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9260106660 0x7f92601a3b10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9260100f60 0x7f9260198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f92480a2230 0x7f92480a46f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 --2- 172.21.15.116:0/3545144897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f92601005a0 0x7f9260198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 >> 172.21.15.116:0/3545144897 conn(0x7f9260077710 msgr2=0x7f92600ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 shutdown_connections 2024-03-24T09:25:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.038+0000 7f9266c0e640 1 -- 172.21.15.116:0/3545144897 wait complete. 2024-03-24T09:25:27.054 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:25:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 -- 172.21.15.116:0/1203123860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8024102060 msgr2=0x7f802410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 --2- 172.21.15.116:0/1203123860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8024102060 0x7f802410e570 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8014009cd0 tx=0x7f801402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 -- 172.21.15.116:0/1203123860 shutdown_connections 2024-03-24T09:25:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 --2- 172.21.15.116:0/1203123860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8024102060 0x7f802410e570 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 --2- 172.21.15.116:0/1203123860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f80241016a0 0x7f8024101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 --2- 172.21.15.116:0/1203123860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241076b0 0x7f8024107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 -- 172.21.15.116:0/1203123860 >> 172.21.15.116:0/1203123860 conn(0x7f8024078b10 msgr2=0x7f8024078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 -- 172.21.15.116:0/1203123860 shutdown_connections 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.157+0000 7f802913e640 1 -- 172.21.15.116:0/1203123860 wait complete. 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 Processor -- start 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 -- start start 2024-03-24T09:25:27.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241016a0 0x7f802419d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8024102060 0x7f802419dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 0x7f80241979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8024198040 con 0x7f80241076b0 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f80241981b0 con 0x7f8024102060 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.158+0000 7f802913e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8024198370 con 0x7f80241016a0 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 0x7f80241979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 0x7f80241979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43914/0 (socket says 172.21.15.116:43914) 2024-03-24T09:25:27.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f8022d9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241016a0 0x7f802419d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 -- 172.21.15.116:0/783018505 learned_addr learned my addr 172.21.15.116:0/783018505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802259c640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8024102060 0x7f802419dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 -- 172.21.15.116:0/783018505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241016a0 msgr2=0x7f802419d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241016a0 0x7f802419d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 -- 172.21.15.116:0/783018505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8024102060 msgr2=0x7f802419dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8024102060 0x7f802419dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 -- 172.21.15.116:0/783018505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8014009900 con 0x7f80241076b0 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f8022d9d640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241016a0 0x7f802419d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802259c640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8024102060 0x7f802419dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802359e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 0x7f80241979f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8014004af0 tx=0x7f8014032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f800bfff640 1 -- 172.21.15.116:0/783018505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80140325d0 con 0x7f80241076b0 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80241985f0 con 0x7f80241076b0 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f800bfff640 1 -- 172.21.15.116:0/783018505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8014003970 con 0x7f80241076b0 2024-03-24T09:25:27.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f800bfff640 1 -- 172.21.15.116:0/783018505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80140046b0 con 0x7f80241076b0 2024-03-24T09:25:27.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.159+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80241a48e0 con 0x7f80241076b0 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.161+0000 7f800bfff640 1 -- 172.21.15.116:0/783018505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f801403b040 con 0x7f80241076b0 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 Processor -- start 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 -- start start 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ec7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56376b05a840 con 0x56376b0ec3f0 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56376b057d70 con 0x56376b0f58f0 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802bdfb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56376b053550 con 0x56376b0ecd10 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f800bfff640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f800c0a2230 0x7f800c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ec7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ec7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43924/0 (socket says 172.21.15.116:43924) 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 -- 172.21.15.116:0/1680773821 learned_addr learned my addr 172.21.15.116:0/1680773821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f8022d9d640 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802359e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f800c0a2230 0x7f800c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f800bfff640 1 -- 172.21.15.116:0/783018505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f80140f0c60 con 0x7f80241076b0 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 -- 172.21.15.116:0/1680773821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 msgr2=0x56376b0f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 -- 172.21.15.116:0/1680773821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 msgr2=0x56376b0f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 -- 172.21.15.116:0/1680773821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56376afe8de0 con 0x56376b0ec3f0 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.162+0000 7f802259c640 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ec7d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f80180139e0 tx=0x7f8018009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b4fc817b8747c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802359e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f800c0a2230 0x7f800c0a46f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f80140ee1c0 tx=0x7f801408d2a0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f80097fa640 1 -- 172.21.15.116:0/1680773821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018004030 con 0x56376b0ec3f0 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f80097fa640 1 -- 172.21.15.116:0/1680773821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f801800f980 con 0x56376b0ec3f0 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f80097fa640 1 -- 172.21.15.116:0/1680773821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018003970 con 0x56376b0ec3f0 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 -- 172.21.15.116:0/1680773821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 msgr2=0x56376b0ec7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ec7d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f80180139e0 tx=0x7f8018009c00 comp rx=0 tx=0).stop 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 -- 172.21.15.116:0/1680773821 shutdown_connections 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1680773821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ec7d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 -- 172.21.15.116:0/1680773821 >> 172.21.15.116:0/1680773821 conn(0x56376b0e6b20 msgr2=0x56376b0e6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 -- 172.21.15.116:0/1680773821 shutdown_connections 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.163+0000 7f802bdfb740 1 -- 172.21.15.116:0/1680773821 wait complete. 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 -- start start 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 Processor -- start 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ec3f0 0x56376b0f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 0x56376b0f3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56376ad26430 con 0x56376b0f58f0 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56376ace2e00 con 0x56376b0ec3f0 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56376afd8590 con 0x56376b0ecd10 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802bdfb740 10 client.0 _ll_register_callbacks cb 0x56376b047fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 0x56376b0f3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 0x56376b0f3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43932/0 (socket says 172.21.15.116:43932) 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 -- 172.21.15.116:0/2190541069 learned_addr learned my addr 172.21.15.116:0/2190541069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ec3f0 0x56376b0f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f8022d9d640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 -- 172.21.15.116:0/2190541069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 msgr2=0x56376b0f2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 -- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ec3f0 msgr2=0x56376b0f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ec3f0 0x56376b0f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.164+0000 7f802259c640 1 -- 172.21.15.116:0/2190541069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8018013560 con 0x56376b0f58f0 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802259c640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 0x56376b0f3320 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8018022740 tx=0x7f80180096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86cdbdf23fe0989 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8006ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:27.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8006ff5640 1 -- 172.21.15.116:0/2190541069 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018022080 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8006ff5640 1 -- 172.21.15.116:0/2190541069 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f80180221e0 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56376ad527e0 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8006ff5640 1 -- 172.21.15.116:0/2190541069 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8018005260 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 10 client.16665 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56376b051bd0 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8006ff5640 1 -- 172.21.15.116:0/2190541069 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8018032020 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8006ff5640 1 -- 172.21.15.116:0/2190541069 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f80180046c0 con 0x56376b0f58f0 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 10 client.16665 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:27.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 20 client.16665 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b105a20 0x56376b107e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 4 client.16665 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56376ad8f720 con 0x56376b105a20 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.165+0000 7f8022d9d640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b105a20 0x56376b107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f8006ff5640 10 client.16665 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f8022d9d640 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b105a20 0x56376b107e40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=963707a114d1fd2a server_cookie=56d1f07706bdc70d in_seq=0 out_seq=0 2024-03-24T09:25:27.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f8006ff5640 1 -- 172.21.15.116:0/2190541069 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56376ad8f720 con 0x56376b105a20 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f8006ff5640 10 client.16665 handle_command_reply: tid=0 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f802bdfb740 1 client.16665 shutdown 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 msgr2=0x56376b0f3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.166+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 0x56376b0f3320 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8018022740 tx=0x7f80180096c0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 shutdown_connections 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0f58f0 0x56376b0f3320 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ecd10 0x56376b0f2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b105a20 0x56376b107e40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2190541069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ec3f0 0x56376b0f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f8006ff5640 0 client.16665 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 >> 172.21.15.116:0/2190541069 conn(0x56376b0e6b20 msgr2=0x56376b0e8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 shutdown_connections 2024-03-24T09:25:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 1 -- 172.21.15.116:0/2190541069 wait complete. 2024-03-24T09:25:27.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.167+0000 7f802bdfb740 20 client.16665 trim_cache size 0 max 16384 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 Processor -- start 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 -- start start 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ec3f0 0x56376b0f4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 0x56376b1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56376b1016f0 con 0x56376b0ecd10 2024-03-24T09:25:27.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56376b104140 con 0x56376b0f58f0 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802bdfb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56376b0ebb90 con 0x56376b0ec3f0 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 0x56376b1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 0x56376b1011b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43934/0 (socket says 172.21.15.116:43934) 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 -- 172.21.15.116:0/2232651348 learned_addr learned my addr 172.21.15.116:0/2232651348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ec3f0 0x56376b0f4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f802259c640 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 -- 172.21.15.116:0/2232651348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ec3f0 msgr2=0x56376b0f4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ec3f0 0x56376b0f4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 -- 172.21.15.116:0/2232651348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 msgr2=0x56376b0e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 -- 172.21.15.116:0/2232651348 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56376afedc80 con 0x56376b0ecd10 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8022d9d640 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 0x56376b1011b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f801c012620 tx=0x7f801c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443f228288ea0dd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8008ff9640 1 -- 172.21.15.116:0/2232651348 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c017030 con 0x56376b0ecd10 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8008ff9640 1 -- 172.21.15.116:0/2232651348 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f801c018040 con 0x56376b0ecd10 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.180+0000 7f8008ff9640 1 -- 172.21.15.116:0/2232651348 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c004a00 con 0x56376b0ecd10 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 -- 172.21.15.116:0/2232651348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 msgr2=0x56376b1011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 0x56376b1011b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f801c012620 tx=0x7f801c005f90 comp rx=0 tx=0).stop 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 -- 172.21.15.116:0/2232651348 shutdown_connections 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0f58f0 0x56376b0e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ecd10 0x56376b1011b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 --2- 172.21.15.116:0/2232651348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0ec3f0 0x56376b0f4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 -- 172.21.15.116:0/2232651348 >> 172.21.15.116:0/2232651348 conn(0x56376b0e6b20 msgr2=0x56376b0e5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 -- 172.21.15.116:0/2232651348 shutdown_connections 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 -- 172.21.15.116:0/2232651348 wait complete. 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 -- start start 2024-03-24T09:25:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 Processor -- start 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.181+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ef540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ecd10 0x56376b0efa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0f58f0 0x56376b0effc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56376b16ad20 con 0x56376b0ec3f0 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56376b16ae60 con 0x56376b0ecd10 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56376b16b020 con 0x56376b0f58f0 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 10 client.0 _ll_register_callbacks cb 0x7f801c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ef540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43946/0 (socket says 172.21.15.116:43946) 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 -- 172.21.15.116:0/1810450507 learned_addr learned my addr 172.21.15.116:0/1810450507 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f8022d9d640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ecd10 0x56376b0efa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0f58f0 0x56376b0effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 -- 172.21.15.116:0/1810450507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0f58f0 msgr2=0x56376b0effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0f58f0 0x56376b0effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 -- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ecd10 msgr2=0x56376b0efa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ecd10 0x56376b0efa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 -- 172.21.15.116:0/1810450507 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f801c0120f0 con 0x56376b0ec3f0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f8022d9d640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ecd10 0x56376b0efa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802259c640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ef540 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f801800f890 tx=0x7f80180096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee5d9f32cfc206d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f8006ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f8006ff5640 1 -- 172.21.15.116:0/1810450507 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8018005470 con 0x56376b0ec3f0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f8006ff5640 1 -- 172.21.15.116:0/1810450507 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8018000e40 con 0x56376b0ec3f0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f801800f980 con 0x56376b0ec3f0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f8006ff5640 1 -- 172.21.15.116:0/1810450507 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f80180055d0 con 0x56376b0ec3f0 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 10 client.16671 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:27.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.182+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56376b013920 con 0x56376b0ec3f0 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8006ff5640 1 -- 172.21.15.116:0/1810450507 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f801800f980 con 0x56376b0ec3f0 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8006ff5640 1 -- 172.21.15.116:0/1810450507 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f8018035040 con 0x56376b0ec3f0 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 10 client.16671 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 20 client.16671 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b16c550 0x56376b16e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 4 client.16671 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56376adde890 con 0x56376b16c550 2024-03-24T09:25:27.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8022d9d640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b16c550 0x56376b16e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8006ff5640 10 client.16671 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8022d9d640 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b16c550 0x56376b16e970 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7b3991a01465fbb server_cookie=ed9f2c4823af6ab0 in_seq=0 out_seq=0 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8006ff5640 1 -- 172.21.15.116:0/1810450507 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56376adde890 con 0x56376b16c550 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8006ff5640 10 client.16671 handle_command_reply: tid=0 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 client.16671 shutdown 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 msgr2=0x56376b0ef540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ef540 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f801800f890 tx=0x7f80180096c0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 shutdown_connections 2024-03-24T09:25:27.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56376b0f58f0 0x56376b0effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56376b16c550 0x56376b16e970 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56376b0ecd10 0x56376b0efa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f8006ff5640 0 client.16671 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 --2- 172.21.15.116:0/1810450507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56376b0ec3f0 0x56376b0ef540 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.183+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 >> 172.21.15.116:0/1810450507 conn(0x56376b0e6b20 msgr2=0x56376b0e87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 shutdown_connections 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802bdfb740 1 -- 172.21.15.116:0/1810450507 wait complete. 2024-03-24T09:25:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802bdfb740 20 client.16671 trim_cache size 0 max 16384 2024-03-24T09:25:27.199 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f800c0a2230 msgr2=0x7f800c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f800c0a2230 0x7f800c0a46f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f80140ee1c0 tx=0x7f801408d2a0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 msgr2=0x7f80241979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 0x7f80241979f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f8014004af0 tx=0x7f8014032f30 comp rx=0 tx=0).stop 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 shutdown_connections 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80241076b0 0x7f80241979f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8024102060 0x7f802419dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f800c0a2230 0x7f800c0a46f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 --2- 172.21.15.116:0/783018505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f80241016a0 0x7f802419d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 >> 172.21.15.116:0/783018505 conn(0x7f8024078b10 msgr2=0x7f8024104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 shutdown_connections 2024-03-24T09:25:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:27.184+0000 7f802913e640 1 -- 172.21.15.116:0/783018505 wait complete. 2024-03-24T09:25:27.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:27 smithi116 ceph-mon[32207]: pgmap v103: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-24T09:25:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:27 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4247785466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:25:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:27 smithi143 ceph-mon[35965]: pgmap v103: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-24T09:25:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:27 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4247785466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:25:27.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:27 smithi117 ceph-mon[35788]: pgmap v103: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-24T09:25:27.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:27 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4247785466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:25:28.202 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:25:28.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:25:28.203 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:25:28.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 -- 172.21.15.116:0/471313647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 msgr2=0x7f7154107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 --2- 172.21.15.116:0/471313647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154107aa0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7144009860 tx=0x7f714402bb70 comp rx=0 tx=0).stop 2024-03-24T09:25:28.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 -- 172.21.15.116:0/471313647 shutdown_connections 2024-03-24T09:25:28.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 --2- 172.21.15.116:0/471313647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 --2- 172.21.15.116:0/471313647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71541016b0 0x7f7154101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 --2- 172.21.15.116:0/471313647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154107aa0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 -- 172.21.15.116:0/471313647 >> 172.21.15.116:0/471313647 conn(0x7f71540fd140 msgr2=0x7f71540ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 -- 172.21.15.116:0/471313647 shutdown_connections 2024-03-24T09:25:28.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.319+0000 7f7158b3e640 1 -- 172.21.15.116:0/471313647 wait complete. 2024-03-24T09:25:28.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 Processor -- start 2024-03-24T09:25:28.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 -- start start 2024-03-24T09:25:28.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71541016b0 0x7f715419d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715419de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71541980f0 con 0x7f7154102070 2024-03-24T09:25:28.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7154198260 con 0x7f71541076c0 2024-03-24T09:25:28.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.320+0000 7f7158b3e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7154198420 con 0x7f71541016b0 2024-03-24T09:25:28.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715419de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715419de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43964/0 (socket says 172.21.15.116:43964) 2024-03-24T09:25:28.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 -- 172.21.15.116:0/2782931056 learned_addr learned my addr 172.21.15.116:0/2782931056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7152d9d640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f715259c640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71541016b0 0x7f715419d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 -- 172.21.15.116:0/2782931056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71541016b0 msgr2=0x7f715419d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71541016b0 0x7f715419d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 -- 172.21.15.116:0/2782931056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 msgr2=0x7f7154197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 -- 172.21.15.116:0/2782931056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714c009900 con 0x7f7154102070 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7152d9d640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7151d9b640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715419de30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f714800e9f0 tx=0x7f714800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f71437fe640 1 -- 172.21.15.116:0/2782931056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148009720 con 0x7f7154102070 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7144009490 con 0x7f7154102070 2024-03-24T09:25:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f71437fe640 1 -- 172.21.15.116:0/2782931056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f714800cd80 con 0x7f7154102070 2024-03-24T09:25:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f71437fe640 1 -- 172.21.15.116:0/2782931056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148010430 con 0x7f7154102070 2024-03-24T09:25:28.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.321+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71541a47b0 con 0x7f7154102070 2024-03-24T09:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.323+0000 7f71437fe640 1 -- 172.21.15.116:0/2782931056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7148005020 con 0x7f7154102070 2024-03-24T09:25:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.324+0000 7f715b7fc740 1 Processor -- start 2024-03-24T09:25:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.324+0000 7f715b7fc740 1 -- start start 2024-03-24T09:25:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.324+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 0x5558e2cdbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.324+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2ce4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.324+0000 7f71437fe640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f713c0a2230 0x7f713c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.324+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f715b7fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5558e2c49530 con 0x5558e2cdc080 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f715b7fc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5558e2c46a60 con 0x5558e2cdb760 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f715b7fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5558e2c42240 con 0x5558e2ce4c60 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 0x5558e2cdbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2ce4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 0x5558e2cdbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58384/0 (socket says 172.21.15.116:58384) 2024-03-24T09:25:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 -- 172.21.15.116:0/1621058146 learned_addr learned my addr 172.21.15.116:0/1621058146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f715259c640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f713c0a2230 0x7f713c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f71437fe640 1 -- 172.21.15.116:0/2782931056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7148014070 con 0x7f7154102070 2024-03-24T09:25:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 -- 172.21.15.116:0/1621058146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 msgr2=0x5558e2ce7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 -- 172.21.15.116:0/1621058146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 msgr2=0x5558e2ce4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2ce4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 -- 172.21.15.116:0/1621058146 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e2c408c0 con 0x5558e2cdb760 2024-03-24T09:25:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7152d9d640 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 0x5558e2cdbb40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f714c00efd0 tx=0x7f714c010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc29cdc58864fde9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7140ff9640 1 -- 172.21.15.116:0/1621058146 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c013070 con 0x5558e2cdb760 2024-03-24T09:25:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7140ff9640 1 -- 172.21.15.116:0/1621058146 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f714c014040 con 0x5558e2cdb760 2024-03-24T09:25:28.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f7140ff9640 1 -- 172.21.15.116:0/1621058146 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c004730 con 0x5558e2cdb760 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.325+0000 7f715259c640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f713c0a2230 0x7f713c0a46f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7144009830 tx=0x7f7144032360 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 -- 172.21.15.116:0/1621058146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 msgr2=0x5558e2cdbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 0x5558e2cdbb40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f714c00efd0 tx=0x7f714c010040 comp rx=0 tx=0).stop 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 -- 172.21.15.116:0/1621058146 shutdown_connections 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2ce4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 --2- 172.21.15.116:0/1621058146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdb760 0x5558e2cdbb40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 -- 172.21.15.116:0/1621058146 >> 172.21.15.116:0/1621058146 conn(0x5558e2cd5e90 msgr2=0x5558e2cd6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 -- 172.21.15.116:0/1621058146 shutdown_connections 2024-03-24T09:25:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.326+0000 7f715b7fc740 1 -- 172.21.15.116:0/1621058146 wait complete. 2024-03-24T09:25:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 -- start start 2024-03-24T09:25:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 Processor -- start 2024-03-24T09:25:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 0x5558e2ce2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5558e2be6f80 con 0x5558e2ce4c60 2024-03-24T09:25:28.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5558e29154b0 con 0x5558e2cdc080 2024-03-24T09:25:28.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5558e2bc7ad0 con 0x5558e2cdb760 2024-03-24T09:25:28.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715b7fc740 10 client.0 _ll_register_callbacks cb 0x5558e2bfab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f7152d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 0x5558e2ce2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f7152d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 0x5558e2ce2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43978/0 (socket says 172.21.15.116:43978) 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f7152d9d640 1 -- 172.21.15.116:0/368468003 learned_addr learned my addr 172.21.15.116:0/368468003 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.327+0000 7f715259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:41112/0 (socket says 172.21.15.116:41112) 2024-03-24T09:25:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7152d9d640 1 -- 172.21.15.116:0/368468003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 msgr2=0x5558e2ce1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7152d9d640 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7152d9d640 1 -- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 msgr2=0x5558e2ce2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7152d9d640 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7152d9d640 1 -- 172.21.15.116:0/368468003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f714c00ee40 con 0x5558e2ce4c60 2024-03-24T09:25:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f715259c640 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7152d9d640 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 0x5558e2ce2790 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f714c004d20 tx=0x7f714c004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42db267af80b5bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7139ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7139ffb640 1 -- 172.21.15.116:0/368468003 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c013030 con 0x5558e2ce4c60 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7139ffb640 1 -- 172.21.15.116:0/368468003 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f714c00ce50 con 0x5558e2ce4c60 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5558e2c667a0 con 0x5558e2ce4c60 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7139ffb640 1 -- 172.21.15.116:0/368468003 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f714c005bd0 con 0x5558e2ce4c60 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f715b7fc740 10 client.16683 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558e2bf86d0 con 0x5558e2ce4c60 2024-03-24T09:25:28.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7139ffb640 1 -- 172.21.15.116:0/368468003 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5558e2c667a0 con 0x5558e2ce4c60 2024-03-24T09:25:28.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f7139ffb640 1 -- 172.21.15.116:0/368468003 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f714c025040 con 0x5558e2ce4c60 2024-03-24T09:25:28.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f715b7fc740 10 client.16683 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:28.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.328+0000 7f715b7fc740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:28.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f715b7fc740 20 client.16683 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:28.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f715b7fc740 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2cf4cd0 0x5558e2cf70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f715b7fc740 4 client.16683 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558e2c381f0 con 0x5558e2cf4cd0 2024-03-24T09:25:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f715259c640 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2cf4cd0 0x5558e2cf70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f7139ffb640 10 client.16683 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.329+0000 7f715259c640 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2cf4cd0 0x5558e2cf70f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4e1f50949b82c83 server_cookie=ca37b5e4f63536f in_seq=0 out_seq=0 2024-03-24T09:25:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f7139ffb640 1 -- 172.21.15.116:0/368468003 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558e2c381f0 con 0x5558e2cf4cd0 2024-03-24T09:25:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f7139ffb640 10 client.16683 handle_command_reply: tid=0 2024-03-24T09:25:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 client.16683 shutdown 2024-03-24T09:25:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 msgr2=0x5558e2ce2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 0x5558e2ce2790 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f714c004d20 tx=0x7f714c004d50 comp rx=0 tx=0).stop 2024-03-24T09:25:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 shutdown_connections 2024-03-24T09:25:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2ce4c60 0x5558e2ce2790 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2cf4cd0 0x5558e2cf70f0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f7139ffb640 0 client.16683 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 --2- 172.21.15.116:0/368468003 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 >> 172.21.15.116:0/368468003 conn(0x5558e2cd5e90 msgr2=0x5558e2cd7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 shutdown_connections 2024-03-24T09:25:28.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 1 -- 172.21.15.116:0/368468003 wait complete. 2024-03-24T09:25:28.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.330+0000 7f715b7fc740 20 client.16683 trim_cache size 0 max 16384 2024-03-24T09:25:28.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 Processor -- start 2024-03-24T09:25:28.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 -- start start 2024-03-24T09:25:28.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2cf0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2ce4c60 0x5558e2cfc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5558e2cd1620 con 0x5558e2cdc080 2024-03-24T09:25:28.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5558e2cdb1a0 con 0x5558e2ce4c60 2024-03-24T09:25:28.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715b7fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5558e2ce2c70 con 0x5558e2cdb760 2024-03-24T09:25:28.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2cf0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2cf0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43982/0 (socket says 172.21.15.116:43982) 2024-03-24T09:25:28.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 -- 172.21.15.116:0/697776654 learned_addr learned my addr 172.21.15.116:0/697776654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7152d9d640 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2ce4c60 0x5558e2cfc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f715259c640 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 -- 172.21.15.116:0/697776654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 msgr2=0x5558e2ce4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 -- 172.21.15.116:0/697776654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2ce4c60 msgr2=0x5558e2cfc9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2ce4c60 0x5558e2cfc9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 -- 172.21.15.116:0/697776654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e2bdcc60 con 0x5558e2cdc080 2024-03-24T09:25:28.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.348+0000 7f7151d9b640 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2cf0490 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f71480977d0 tx=0x7f714800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c07541a9a28e1ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f713bfff640 1 -- 172.21.15.116:0/697776654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714809d040 con 0x5558e2cdc080 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f713bfff640 1 -- 172.21.15.116:0/697776654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7148084040 con 0x5558e2cdc080 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f713bfff640 1 -- 172.21.15.116:0/697776654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714809ef50 con 0x5558e2cdc080 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 -- 172.21.15.116:0/697776654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 msgr2=0x5558e2cf0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2cf0490 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f71480977d0 tx=0x7f714800cef0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 -- 172.21.15.116:0/697776654 shutdown_connections 2024-03-24T09:25:28.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2ce4c60 0x5558e2cfc9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdc080 0x5558e2cf0490 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 --2- 172.21.15.116:0/697776654 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2cdb760 0x5558e2ce4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 -- 172.21.15.116:0/697776654 >> 172.21.15.116:0/697776654 conn(0x5558e2cd5e90 msgr2=0x5558e2ce2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 -- 172.21.15.116:0/697776654 shutdown_connections 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 -- 172.21.15.116:0/697776654 wait complete. 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.349+0000 7f715b7fc740 1 -- start start 2024-03-24T09:25:28.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 Processor -- start 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 0x5558e2cecac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5558e2ced250 con 0x5558e2cdb760 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5558e2cea510 con 0x5558e2cdc080 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5558e2cea6d0 con 0x5558e2ce4c60 2024-03-24T09:25:28.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 10 client.0 _ll_register_callbacks cb 0x7f714c00c400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 0x5558e2cecac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 0x5558e2cecac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43994/0 (socket says 172.21.15.116:43994) 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 -- 172.21.15.116:0/2478590581 learned_addr learned my addr 172.21.15.116:0/2478590581 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f7152d9d640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f7151d9b640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 -- 172.21.15.116:0/2478590581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 msgr2=0x5558e2ce9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 -- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 msgr2=0x5558e2ce9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 -- 172.21.15.116:0/2478590581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71541a47b0 con 0x5558e2cdb760 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715259c640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 0x5558e2cecac0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f714402c0b0 tx=0x7f714402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd300416c1fcc97e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f7139ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f7139ffb640 1 -- 172.21.15.116:0/2478590581 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144053040 con 0x5558e2cdb760 2024-03-24T09:25:28.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f7139ffb640 1 -- 172.21.15.116:0/2478590581 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7144046050 con 0x5558e2cdb760 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.350+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5558e2cdd6e0 con 0x5558e2cdb760 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f7139ffb640 1 -- 172.21.15.116:0/2478590581 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f714402c7d0 con 0x5558e2cdb760 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 10 client.16689 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5558e2c02970 con 0x5558e2cdb760 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f7139ffb640 1 -- 172.21.15.116:0/2478590581 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5558e2cdd6e0 con 0x5558e2cdb760 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f7139ffb640 1 -- 172.21.15.116:0/2478590581 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f714403f480 con 0x5558e2cdb760 2024-03-24T09:25:28.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 10 client.16689 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 20 client.16689 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2d5c4d0 0x5558e2d5e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 4 client.16689 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f714c014040 con 0x5558e2d5c4d0 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f7151d9b640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2d5c4d0 0x5558e2d5e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f7139ffb640 10 client.16689 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.351+0000 7f7151d9b640 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2d5c4d0 0x5558e2d5e8f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9265f32a7d168da server_cookie=27ed26d8656eb056 in_seq=0 out_seq=0 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7139ffb640 1 -- 172.21.15.116:0/2478590581 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f714c014040 con 0x5558e2d5c4d0 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7139ffb640 10 client.16689 handle_command_reply: tid=0 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 client.16689 shutdown 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 msgr2=0x5558e2cecac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 0x5558e2cecac0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f714402c0b0 tx=0x7f714402e050 comp rx=0 tx=0).stop 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 shutdown_connections 2024-03-24T09:25:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5558e2ce4c60 0x5558e2ce9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5558e2d5c4d0 0x5558e2d5e8f0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2cdc080 0x5558e2ce9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 --2- 172.21.15.116:0/2478590581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5558e2cdb760 0x5558e2cecac0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 >> 172.21.15.116:0/2478590581 conn(0x5558e2cd5e90 msgr2=0x5558e2cd4660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7139ffb640 0 client.16689 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 shutdown_connections 2024-03-24T09:25:28.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 1 -- 172.21.15.116:0/2478590581 wait complete. 2024-03-24T09:25:28.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f715b7fc740 20 client.16689 trim_cache size 0 max 16384 2024-03-24T09:25:28.371 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:25:28.371 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:25:28.371 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "69651ac8-a01c-4d41-abfc-c8b0284e10c6", 2024-03-24T09:25:28.371 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:25:28.372 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:25:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f713c0a2230 msgr2=0x7f713c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f713c0a2230 0x7f713c0a46f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7144009830 tx=0x7f7144032360 comp rx=0 tx=0).stop 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 msgr2=0x7f715419de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715419de30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f714800e9f0 tx=0x7f714800edf0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 shutdown_connections 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71541076c0 0x7f7154197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7154102070 0x7f715419de30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f713c0a2230 0x7f713c0a46f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 --2- 172.21.15.116:0/2782931056 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71541016b0 0x7f715419d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.353+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 >> 172.21.15.116:0/2782931056 conn(0x7f71540fd140 msgr2=0x7f71540fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.354+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 shutdown_connections 2024-03-24T09:25:28.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.354+0000 7f7158b3e640 1 -- 172.21.15.116:0/2782931056 wait complete. 2024-03-24T09:25:28.375 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:25:28.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:25:28.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2191272342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 msgr2=0x7ff5d0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2191272342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0106a40 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4009900 tx=0x7ff5c402bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:28.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2191272342 shutdown_connections 2024-03-24T09:25:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2191272342 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5d0100f60 0x7ff5d010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2191272342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 0x7ff5d0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2191272342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0106a40 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.495+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2191272342 >> 172.21.15.116:0/2191272342 conn(0x7ff5d0077710 msgr2=0x7ff5d0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.496+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2191272342 shutdown_connections 2024-03-24T09:25:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.496+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2191272342 wait complete. 2024-03-24T09:25:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.496+0000 7ff5d6b6c640 1 Processor -- start 2024-03-24T09:25:28.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 -- start start 2024-03-24T09:25:28.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 0x7ff5d019d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5d0100f60 0x7ff5d019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01980a0 con 0x7ff5d0106660 2024-03-24T09:25:28.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d0198210 con 0x7ff5d01005a0 2024-03-24T09:25:28.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d6b6c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01983d0 con 0x7ff5d0100f60 2024-03-24T09:25:28.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44020/0 (socket says 172.21.15.116:44020) 2024-03-24T09:25:28.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 -- 172.21.15.116:0/2868315327 learned_addr learned my addr 172.21.15.116:0/2868315327 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5cffff640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5d0100f60 0x7ff5d019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d4908640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 0x7ff5d019d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 -- 172.21.15.116:0/2868315327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5d0100f60 msgr2=0x7ff5d019dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5d0100f60 0x7ff5d019dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 -- 172.21.15.116:0/2868315327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 msgr2=0x7ff5d019d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 0x7ff5d019d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.497+0000 7ff5d5109640 1 -- 172.21.15.116:0/2868315327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5c4009560 con 0x7ff5d0106660 2024-03-24T09:25:28.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5d4908640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 0x7ff5d019d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:25:28.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5d5109640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0197a50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800cce0 tx=0x7ff5c800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5cdffb640 1 -- 172.21.15.116:0/2868315327 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c800f450 con 0x7ff5d0106660 2024-03-24T09:25:28.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5cdffb640 1 -- 172.21.15.116:0/2868315327 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5c800fa60 con 0x7ff5d0106660 2024-03-24T09:25:28.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5cdffb640 1 -- 172.21.15.116:0/2868315327 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8018b10 con 0x7ff5d0106660 2024-03-24T09:25:28.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d01986b0 con 0x7ff5d0106660 2024-03-24T09:25:28.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.498+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d01a48a0 con 0x7ff5d0106660 2024-03-24T09:25:28.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.501+0000 7ff5cdffb640 1 -- 172.21.15.116:0/2868315327 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7ff5c800fbd0 con 0x7ff5d0106660 2024-03-24T09:25:28.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.501+0000 7ff5d9826740 1 Processor -- start 2024-03-24T09:25:28.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 -- start start 2024-03-24T09:25:28.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f1560 0x555e770fe5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770feae0 0x555e77100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555e77015f30 con 0x555e770feae0 2024-03-24T09:25:28.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555e7705f810 con 0x555e770f0bb0 2024-03-24T09:25:28.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d9826740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555e7705cd40 con 0x555e770f1560 2024-03-24T09:25:28.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5cdffb640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5c00a22b0 0x7ff5c00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770feae0 0x555e77100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770feae0 0x555e77100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44034/0 (socket says 172.21.15.116:44034) 2024-03-24T09:25:28.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5cffff640 1 -- 172.21.15.116:0/3090517725 learned_addr learned my addr 172.21.15.116:0/3090517725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5cffff640 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d5109640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5c00a22b0 0x7ff5c00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5cdffb640 1 -- 172.21.15.116:0/2868315327 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff5c80ce820 con 0x7ff5d0106660 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.502+0000 7ff5d4908640 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f1560 0x555e770fe5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5cffff640 1 -- 172.21.15.116:0/3090517725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f1560 msgr2=0x555e770fe5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5cffff640 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f1560 0x555e770fe5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5cffff640 1 -- 172.21.15.116:0/3090517725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770feae0 msgr2=0x555e77100ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5cffff640 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770feae0 0x555e77100ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5cffff640 1 -- 172.21.15.116:0/3090517725 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e77019ea0 con 0x555e770f0bb0 2024-03-24T09:25:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5d5109640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5c00a22b0 0x7ff5c00a4770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80cbc70 tx=0x7ff5c8097200 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5cffff640 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770f0f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0058c0 tx=0x7ff5bc005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b11b99715182322b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5baffd640 1 -- 172.21.15.116:0/3090517725 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc011d70 con 0x555e770f0bb0 2024-03-24T09:25:28.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5baffd640 1 -- 172.21.15.116:0/3090517725 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5bc0044d0 con 0x555e770f0bb0 2024-03-24T09:25:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.503+0000 7ff5baffd640 1 -- 172.21.15.116:0/3090517725 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc004a00 con 0x555e770f0bb0 2024-03-24T09:25:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.504+0000 7ff5d9826740 1 -- 172.21.15.116:0/3090517725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 msgr2=0x555e770f0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.504+0000 7ff5d9826740 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770f0f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0058c0 tx=0x7ff5bc005f90 comp rx=0 tx=0).stop 2024-03-24T09:25:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 -- 172.21.15.116:0/3090517725 shutdown_connections 2024-03-24T09:25:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770feae0 0x555e77100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f1560 0x555e770fe5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 --2- 172.21.15.116:0/3090517725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770f0f90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 -- 172.21.15.116:0/3090517725 >> 172.21.15.116:0/3090517725 conn(0x555e770eb900 msgr2=0x555e770ebce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 -- 172.21.15.116:0/3090517725 shutdown_connections 2024-03-24T09:25:28.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 -- 172.21.15.116:0/3090517725 wait complete. 2024-03-24T09:25:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.505+0000 7ff5d9826740 1 -- start start 2024-03-24T09:25:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5d9826740 1 Processor -- start 2024-03-24T09:25:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770fbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e770fc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e770fc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5d9826740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555e77058520 con 0x555e770f1560 2024-03-24T09:25:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770fbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770fbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58462/0 (socket says 172.21.15.116:58462) 2024-03-24T09:25:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5cffff640 1 -- 172.21.15.116:0/174526280 learned_addr learned my addr 172.21.15.116:0/174526280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.506+0000 7ff5d4908640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e770fc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555e76d2b430 con 0x555e770f0bb0 2024-03-24T09:25:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555e76ce7e00 con 0x555e770feae0 2024-03-24T09:25:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e770fc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5d9826740 10 client.0 _ll_register_callbacks cb 0x555e76ff6d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 -- 172.21.15.116:0/174526280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 msgr2=0x555e770fc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e770fc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 -- 172.21.15.116:0/174526280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 msgr2=0x555e770fc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e770fc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 -- 172.21.15.116:0/174526280 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5bc019040 con 0x555e770f0bb0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5cffff640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770fbb70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009800 tx=0x7ff5bc005520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=291e58636a0124bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.507+0000 7ff5affff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5affff640 1 -- 172.21.15.116:0/174526280 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc013eb0 con 0x555e770f0bb0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5affff640 1 -- 172.21.15.116:0/174526280 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff5bc00f9a0 con 0x555e770f0bb0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555e76d577e0 con 0x555e770f0bb0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5affff640 1 -- 172.21.15.116:0/174526280 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff5bc00fb80 con 0x555e770f0bb0 2024-03-24T09:25:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 10 client.26528 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555e77056ba0 con 0x555e770f0bb0 2024-03-24T09:25:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5affff640 1 -- 172.21.15.116:0/174526280 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5bc00fce0 con 0x555e770f0bb0 2024-03-24T09:25:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5affff640 1 -- 172.21.15.116:0/174526280 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff5bc01d070 con 0x555e770f0bb0 2024-03-24T09:25:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 10 client.26528 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 10 client.26528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 20 client.26528 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77111290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 4 client.26528 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e76d577e0 con 0x555e7710edd0 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.509+0000 7ff5d4908640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77111290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.510+0000 7ff5affff640 10 client.26528 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.510+0000 7ff5d4908640 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77111290 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb89fba751ac427a server_cookie=e1c5b871f08d1f07 in_seq=0 out_seq=0 2024-03-24T09:25:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5affff640 1 -- 172.21.15.116:0/174526280 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555e76d577e0 con 0x555e7710edd0 2024-03-24T09:25:28.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5affff640 10 client.26528 handle_command_reply: tid=0 2024-03-24T09:25:28.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 client.26528 shutdown 2024-03-24T09:25:28.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 msgr2=0x555e770fbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770fbb70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009800 tx=0x7ff5bc005520 comp rx=0 tx=0).stop 2024-03-24T09:25:28.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 shutdown_connections 2024-03-24T09:25:28.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e770fc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e770fc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77111290 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5affff640 0 client.26528 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 --2- 172.21.15.116:0/174526280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f0bb0 0x555e770fbb70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 >> 172.21.15.116:0/174526280 conn(0x555e770eb900 msgr2=0x555e770ed650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 shutdown_connections 2024-03-24T09:25:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 1 -- 172.21.15.116:0/174526280 wait complete. 2024-03-24T09:25:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.512+0000 7ff5d9826740 20 client.26528 trim_cache size 0 max 16384 2024-03-24T09:25:28.530 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:28 smithi143 ceph-mon[35965]: pgmap v104: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-03-24T09:25:28.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 Processor -- start 2024-03-24T09:25:28.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 -- start start 2024-03-24T09:25:28.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 0x555e770fe230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f1560 0x555e7710a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e77116b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555e76fdd430 con 0x555e770f0bb0 2024-03-24T09:25:28.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555e770fb960 con 0x555e770f1560 2024-03-24T09:25:28.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d9826740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555e770dd4d0 con 0x555e770feae0 2024-03-24T09:25:28.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 0x555e770fe230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 0x555e770fe230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44060/0 (socket says 172.21.15.116:44060) 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 -- 172.21.15.116:0/1592709870 learned_addr learned my addr 172.21.15.116:0/1592709870 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5d4908640 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f1560 0x555e7710a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 -- 172.21.15.116:0/1592709870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 msgr2=0x555e77116b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e77116b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 -- 172.21.15.116:0/1592709870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f1560 msgr2=0x555e7710a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f1560 0x555e7710a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.534+0000 7ff5cffff640 1 -- 172.21.15.116:0/1592709870 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e76ff2b20 con 0x555e770f0bb0 2024-03-24T09:25:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.536+0000 7ff5cffff640 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 0x555e770fe230 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009940 tx=0x7ff5bc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10567a8d79b627d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.536+0000 7ff5ba7fc640 1 -- 172.21.15.116:0/1592709870 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc01f450 con 0x555e770f0bb0 2024-03-24T09:25:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.536+0000 7ff5ba7fc640 1 -- 172.21.15.116:0/1592709870 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5bc032070 con 0x555e770f0bb0 2024-03-24T09:25:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.536+0000 7ff5ba7fc640 1 -- 172.21.15.116:0/1592709870 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0099d0 con 0x555e770f0bb0 2024-03-24T09:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 -- 172.21.15.116:0/1592709870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 msgr2=0x555e770fe230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 0x555e770fe230 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009940 tx=0x7ff5bc005f90 comp rx=0 tx=0).stop 2024-03-24T09:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 -- 172.21.15.116:0/1592709870 shutdown_connections 2024-03-24T09:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770feae0 0x555e77116b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770f1560 0x555e7710a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 --2- 172.21.15.116:0/1592709870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f0bb0 0x555e770fe230 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 -- 172.21.15.116:0/1592709870 >> 172.21.15.116:0/1592709870 conn(0x555e770eb900 msgr2=0x555e770eab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 -- 172.21.15.116:0/1592709870 shutdown_connections 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 -- 172.21.15.116:0/1592709870 wait complete. 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.537+0000 7ff5d9826740 1 -- start start 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 Processor -- start 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f0bb0 0x555e77102290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e771027d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770feae0 0x555e77102d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555e77174190 con 0x555e770f1560 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555e771742d0 con 0x555e770feae0 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555e77174410 con 0x555e770f0bb0 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d9826740 10 client.0 _ll_register_callbacks cb 0x7ff5bc009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770feae0 0x555e77102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770feae0 0x555e77102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58494/0 (socket says 172.21.15.116:58494) 2024-03-24T09:25:28.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5d4908640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e771027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5cffff640 1 -- 172.21.15.116:0/4093593340 learned_addr learned my addr 172.21.15.116:0/4093593340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:28.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.538+0000 7ff5cffff640 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f0bb0 0x555e77102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d4908640 1 -- 172.21.15.116:0/4093593340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f0bb0 msgr2=0x555e77102290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d4908640 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f0bb0 0x555e77102290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d4908640 1 -- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770feae0 msgr2=0x555e77102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d4908640 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770feae0 0x555e77102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d4908640 1 -- 172.21.15.116:0/4093593340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e77056ba0 con 0x555e770f1560 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d4908640 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e771027d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0100510 tx=0x7ff5c402c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4bd29e38b41a33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5affff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5affff640 1 -- 172.21.15.116:0/4093593340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c402c820 con 0x555e770f1560 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5affff640 1 -- 172.21.15.116:0/4093593340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff5c4045040 con 0x555e770f1560 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5bc0044d0 con 0x555e770f1560 2024-03-24T09:25:28.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5affff640 1 -- 172.21.15.116:0/4093593340 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff5c402c980 con 0x555e770f1560 2024-03-24T09:25:28.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d9826740 10 client.16707 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:28.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.539+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555e77018700 con 0x555e770f1560 2024-03-24T09:25:28.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5affff640 1 -- 172.21.15.116:0/4093593340 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff5bc0044d0 con 0x555e770f1560 2024-03-24T09:25:28.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5affff640 1 -- 172.21.15.116:0/4093593340 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff5c403c040 con 0x555e770f1560 2024-03-24T09:25:28.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5d9826740 10 client.16707 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:28.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5d9826740 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5d9826740 20 client.16707 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5d9826740 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77177cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5d9826740 4 client.16707 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555e76d94720 con 0x555e7710edd0 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5cffff640 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77177cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5cffff640 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77177cf0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb0549b0c43fad31 server_cookie=603294c4b0c13938 in_seq=0 out_seq=0 2024-03-24T09:25:28.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.540+0000 7ff5affff640 10 client.16707 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5affff640 1 -- 172.21.15.116:0/4093593340 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x555e76d94720 con 0x555e7710edd0 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5affff640 10 client.16707 handle_command_reply: tid=0 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 client.16707 shutdown 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 msgr2=0x555e771027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e771027d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0100510 tx=0x7ff5c402c1f0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 shutdown_connections 2024-03-24T09:25:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x555e7710edd0 0x555e77177cf0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555e770feae0 0x555e77102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555e770f1560 0x555e771027d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5affff640 0 client.16707 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 --2- 172.21.15.116:0/4093593340 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555e770f0bb0 0x555e77102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 >> 172.21.15.116:0/4093593340 conn(0x555e770eb900 msgr2=0x555e770ecce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 shutdown_connections 2024-03-24T09:25:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 1 -- 172.21.15.116:0/4093593340 wait complete. 2024-03-24T09:25:28.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d9826740 20 client.16707 trim_cache size 0 max 16384 2024-03-24T09:25:28.554 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:25:28.554 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: "69651ac8-a01c-4d41-abfc-c8b0284e10c6": { 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "69651ac8-a01c-4d41-abfc-c8b0284e10c6", 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:25:28.555 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:25:28.556 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5c00a22b0 msgr2=0x7ff5c00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5c00a22b0 0x7ff5c00a4770 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff5c80cbc70 tx=0x7ff5c8097200 comp rx=0 tx=0).stop 2024-03-24T09:25:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 msgr2=0x7ff5d0197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0197a50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800cce0 tx=0x7ff5c800a620 comp rx=0 tx=0).stop 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 shutdown_connections 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff5d0106660 0x7ff5d0197a50 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff5c00a22b0 0x7ff5c00a4770 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff5d0100f60 0x7ff5d019dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 --2- 172.21.15.116:0/2868315327 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff5d01005a0 0x7ff5d019d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:28.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 >> 172.21.15.116:0/2868315327 conn(0x7ff5d0077710 msgr2=0x7ff5d00ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 shutdown_connections 2024-03-24T09:25:28.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:28.541+0000 7ff5d6b6c640 1 -- 172.21.15.116:0/2868315327 wait complete. 2024-03-24T09:25:28.558 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:28 smithi116 ceph-mon[32207]: pgmap v104: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-03-24T09:25:28.559 INFO:tasks.cephfs.filesystem:scrub status for tag:69651ac8-a01c-4d41-abfc-c8b0284e10c6 - {'path': '/', 'tag': '69651ac8-a01c-4d41-abfc-c8b0284e10c6', 'options': 'recursive,force'} 2024-03-24T09:25:28.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:28 smithi117 ceph-mon[35788]: pgmap v104: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-03-24T09:25:29.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:25:29.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:25:29.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:25:29.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:25:29.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:25:29.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:25:29.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:25:29.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:29 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:25:29.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:25:29.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:25:29.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:25:29.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:25:29.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:25:29.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:25:29.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:29 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:25:29.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:25:29.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:25:29.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:25:29.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:25:29.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:25:29.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:25:29.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:29.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:29 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:30 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:30 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:30 smithi143 ceph-mon[35965]: pgmap v105: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:30.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:30 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:30 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:30 smithi117 ceph-mon[35788]: pgmap v105: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:30.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:30 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:30 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:25:30.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:30 smithi116 ceph-mon[32207]: pgmap v105: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:32.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:32 smithi143 ceph-mon[35965]: pgmap v106: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-24T09:25:32.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:32 smithi117 ceph-mon[35788]: pgmap v106: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-24T09:25:32.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:32 smithi116 ceph-mon[32207]: pgmap v106: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-03-24T09:25:34.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:34 smithi143 ceph-mon[35965]: pgmap v107: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:34.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:34 smithi117 ceph-mon[35788]: pgmap v107: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:34.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:34 smithi116 ceph-mon[32207]: pgmap v107: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:36.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:36 smithi143 ceph-mon[35965]: pgmap v108: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:36.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:36 smithi117 ceph-mon[35788]: pgmap v108: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:36.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:36 smithi116 ceph-mon[32207]: pgmap v108: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-24T09:25:38.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:38 smithi143 ceph-mon[35965]: pgmap v109: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.0 KiB/s wr, 13 op/s 2024-03-24T09:25:38.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:38 smithi117 ceph-mon[35788]: pgmap v109: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.0 KiB/s wr, 13 op/s 2024-03-24T09:25:38.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:38 smithi116 ceph-mon[32207]: pgmap v109: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 3.0 KiB/s wr, 13 op/s 2024-03-24T09:25:40.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:40 smithi143 ceph-mon[35965]: pgmap v110: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:40.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:40 smithi117 ceph-mon[35788]: pgmap v110: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:40.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:40 smithi116 ceph-mon[32207]: pgmap v110: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:42.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:42 smithi143 ceph-mon[35965]: pgmap v111: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-24T09:25:42.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:42 smithi117 ceph-mon[35788]: pgmap v111: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-24T09:25:42.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:42 smithi116 ceph-mon[32207]: pgmap v111: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-24T09:25:43.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:43 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:43.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:43 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:43.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:43 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:43.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:43 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:43.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:43 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:43.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:43 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:44.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:44 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:44.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:44 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:44.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:44 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:25:44.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:44 smithi143 ceph-mon[35965]: pgmap v112: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:44.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:44 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:44.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:44 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:44.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:44 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:25:44.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:44 smithi117 ceph-mon[35788]: pgmap v112: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:44.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:44 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:44.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:44 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:44.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:44 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:25:44.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:44 smithi116 ceph-mon[32207]: pgmap v112: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:46.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:46 smithi143 ceph-mon[35965]: pgmap v113: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:46.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:46 smithi117 ceph-mon[35788]: pgmap v113: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:46.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:46 smithi116 ceph-mon[32207]: pgmap v113: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-24T09:25:48.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:48 smithi143 ceph-mon[35965]: pgmap v114: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-24T09:25:48.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:48 smithi117 ceph-mon[35788]: pgmap v114: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-24T09:25:48.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:48 smithi116 ceph-mon[32207]: pgmap v114: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-03-24T09:25:50.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:50 smithi143 ceph-mon[35965]: pgmap v115: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:50.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:50 smithi117 ceph-mon[35788]: pgmap v115: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:50.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:50 smithi116 ceph-mon[32207]: pgmap v115: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:52.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:52 smithi143 ceph-mon[35965]: pgmap v116: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T09:25:52.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:52 smithi117 ceph-mon[35788]: pgmap v116: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T09:25:52.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:52 smithi116 ceph-mon[32207]: pgmap v116: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T09:25:54.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:54 smithi143 ceph-mon[35965]: pgmap v117: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:54.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:54 smithi117 ceph-mon[35788]: pgmap v117: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:54.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:54 smithi116 ceph-mon[32207]: pgmap v117: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:56.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:56 smithi143 ceph-mon[35965]: pgmap v118: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:56.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:56 smithi117 ceph-mon[35788]: pgmap v118: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:56.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:56 smithi116 ceph-mon[32207]: pgmap v118: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T09:25:58.561 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:25:58.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:25:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 -- 172.21.15.116:0/2604933216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8106640 msgr2=0x7f6fe8106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/2604933216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8106640 0x7f6fe8106a20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009930 tx=0x7f6fd802bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:58.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 -- 172.21.15.116:0/2604933216 shutdown_connections 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/2604933216 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fe8100ff0 0x7f6fe810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/2604933216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fe8100630 0x7f6fe8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/2604933216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8106640 0x7f6fe8106a20 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 -- 172.21.15.116:0/2604933216 >> 172.21.15.116:0/2604933216 conn(0x7f6fe8077770 msgr2=0x7f6fe8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 -- 172.21.15.116:0/2604933216 shutdown_connections 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 -- 172.21.15.116:0/2604933216 wait complete. 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 Processor -- start 2024-03-24T09:25:58.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.678+0000 7f6fefdd0640 1 -- start start 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fe8100630 0x7f6fe8198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 0x7f6fe81990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fe8106640 0x7f6fe81a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8199890 con 0x7f6fe8100ff0 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe8199a00 con 0x7f6fe8106640 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe81a4170 con 0x7f6fe8100630 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 0x7f6fe81990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 0x7f6fe81990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56222/0 (socket says 172.21.15.116:56222) 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fedb6c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fe8100630 0x7f6fe8198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 -- 172.21.15.116:0/1514852411 learned_addr learned my addr 172.21.15.116:0/1514852411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fee36d640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fe8106640 0x7f6fe81a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 -- 172.21.15.116:0/1514852411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fe8100630 msgr2=0x7f6fe8198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fe8100630 0x7f6fe8198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 -- 172.21.15.116:0/1514852411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fe8106640 msgr2=0x7f6fe81a3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fe8106640 0x7f6fe81a3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 -- 172.21.15.116:0/1514852411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd8009560 con 0x7f6fe8100ff0 2024-03-24T09:25:58.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fed36b640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 0x7f6fe81990a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400e9f0 tx=0x7f6fe400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fdeffd640 1 -- 172.21.15.116:0/1514852411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4009700 con 0x7f6fe8100ff0 2024-03-24T09:25:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fdeffd640 1 -- 172.21.15.116:0/1514852411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fe400cd80 con 0x7f6fe8100ff0 2024-03-24T09:25:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fdeffd640 1 -- 172.21.15.116:0/1514852411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4018840 con 0x7f6fe8100ff0 2024-03-24T09:25:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe81a4450 con 0x7f6fe8100ff0 2024-03-24T09:25:58.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.679+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a49b0 con 0x7f6fe8100ff0 2024-03-24T09:25:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.682+0000 7f6fdeffd640 1 -- 172.21.15.116:0/1514852411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6fe40189a0 con 0x7f6fe8100ff0 2024-03-24T09:25:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.682+0000 7f6ff29a8740 1 Processor -- start 2024-03-24T09:25:58.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.682+0000 7f6ff29a8740 1 -- start start 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.682+0000 7f6fdeffd640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6fd40a22b0 0x7f6fd40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 0x55569bf195b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf22190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf226d0 0x55569bf24ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55569be3df30 con 0x55569bf19af0 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55569be87810 con 0x55569bf191d0 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55569be84d40 con 0x55569bf226d0 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fee36d640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6fd40a22b0 0x7f6fd40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fdeffd640 1 -- 172.21.15.116:0/1514852411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6fe4014070 con 0x7f6fe8100ff0 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 0x55569bf195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fed36b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf22190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 0x55569bf195b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44822/0 (socket says 172.21.15.116:44822) 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 -- 172.21.15.116:0/723836401 learned_addr learned my addr 172.21.15.116:0/723836401 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fee36d640 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf226d0 0x55569bf24ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 -- 172.21.15.116:0/723836401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf226d0 msgr2=0x55569bf24ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf226d0 0x55569bf24ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 -- 172.21.15.116:0/723836401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 msgr2=0x55569bf22190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf22190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 -- 172.21.15.116:0/723836401 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55569be41ea0 con 0x55569bf191d0 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fed36b640 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fee36d640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6fd40a22b0 0x7f6fd40a4770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0005fd0 tx=0x7f6fe0013ae0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.683+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 0x55569bf195b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8036040 tx=0x7f6fd8005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e9dd2aebc6ee50e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6fd3fff640 1 -- 172.21.15.116:0/723836401 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8004780 con 0x55569bf191d0 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6fd3fff640 1 -- 172.21.15.116:0/723836401 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fd802de90 con 0x55569bf191d0 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6fd3fff640 1 -- 172.21.15.116:0/723836401 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd80035f0 con 0x55569bf191d0 2024-03-24T09:25:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 -- 172.21.15.116:0/723836401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 msgr2=0x55569bf195b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 0x55569bf195b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8036040 tx=0x7f6fd8005de0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 -- 172.21.15.116:0/723836401 shutdown_connections 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf226d0 0x55569bf24ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf22190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/723836401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf191d0 0x55569bf195b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 -- 172.21.15.116:0/723836401 >> 172.21.15.116:0/723836401 conn(0x55569bf13900 msgr2=0x55569bf13ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 -- 172.21.15.116:0/723836401 shutdown_connections 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.684+0000 7f6ff29a8740 1 -- 172.21.15.116:0/723836401 wait complete. 2024-03-24T09:25:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 -- start start 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 Processor -- start 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 0x55569bf1f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf19af0 0x55569bf1fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf20120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55569be80520 con 0x55569bf191d0 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55569bb53430 con 0x55569bf226d0 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55569bb0fe00 con 0x55569bf19af0 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6ff29a8740 10 client.0 _ll_register_callbacks cb 0x55569be1ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6fedb6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 0x55569bf1f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6fee36d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf20120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6fedb6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 0x55569bf1f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56226/0 (socket says 172.21.15.116:56226) 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6fed36b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf19af0 0x55569bf1fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.685+0000 7f6fedb6c640 1 -- 172.21.15.116:0/940694585 learned_addr learned my addr 172.21.15.116:0/940694585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fedb6c640 1 -- 172.21.15.116:0/940694585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf19af0 msgr2=0x55569bf1fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf19af0 0x55569bf1fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fedb6c640 1 -- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 msgr2=0x55569bf20120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf20120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fedb6c640 1 -- 172.21.15.116:0/940694585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe810de00 con 0x55569bf191d0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fee36d640 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf20120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 0x55569bf1f6a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6fd802c170 tx=0x7f6fd8004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d081144452bcd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fd17fa640 1 -- 172.21.15.116:0/940694585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd8004dc0 con 0x55569bf191d0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fd17fa640 1 -- 172.21.15.116:0/940694585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6fd804d070 con 0x55569bf191d0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55569bb7f7e0 con 0x55569bf191d0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fd17fa640 1 -- 172.21.15.116:0/940694585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6fd8004f20 con 0x55569bf191d0 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6ff29a8740 10 client.16719 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55569be7eba0 con 0x55569bf191d0 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fd17fa640 1 -- 172.21.15.116:0/940694585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55569bb7f7e0 con 0x55569bf191d0 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.686+0000 7f6fd17fa640 1 -- 172.21.15.116:0/940694585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f6fd803c050 con 0x55569bf191d0 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6ff29a8740 10 client.16719 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6ff29a8740 10 client.16719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6ff29a8740 20 client.16719 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf32830 0x55569bf34c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6ff29a8740 4 client.16719 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55569bbbc720 con 0x55569bf32830 2024-03-24T09:25:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6fed36b640 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf32830 0x55569bf34c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6fed36b640 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf32830 0x55569bf34c70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90a02f1d466c16ec server_cookie=89c6ca780d14478e in_seq=0 out_seq=0 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.687+0000 7f6fd17fa640 10 client.16719 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6fd17fa640 1 -- 172.21.15.116:0/940694585 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55569bbbc720 con 0x55569bf32830 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6fd17fa640 10 client.16719 handle_command_reply: tid=0 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 client.16719 shutdown 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 msgr2=0x55569bf1f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 0x55569bf1f6a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6fd802c170 tx=0x7f6fd8004a80 comp rx=0 tx=0).stop 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 shutdown_connections 2024-03-24T09:25:58.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf32830 0x55569bf34c70 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf20120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf19af0 0x55569bf1fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/940694585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf191d0 0x55569bf1f6a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6fd17fa640 0 client.16719 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 >> 172.21.15.116:0/940694585 conn(0x55569bf13900 msgr2=0x55569bf15650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 shutdown_connections 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.688+0000 7f6ff29a8740 1 -- 172.21.15.116:0/940694585 wait complete. 2024-03-24T09:25:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.689+0000 7f6ff29a8740 20 client.16719 trim_cache size 0 max 16384 2024-03-24T09:25:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 Processor -- start 2024-03-24T09:25:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 -- start start 2024-03-24T09:25:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf21dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf2a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf2df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55569be05430 con 0x55569bf19af0 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55569bf0f240 con 0x55569bf226d0 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55569bf14900 con 0x55569bf191d0 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf2a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf2a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56228/0 (socket says 172.21.15.116:56228) 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 -- 172.21.15.116:0/446722685 learned_addr learned my addr 172.21.15.116:0/446722685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf21dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fee36d640 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf2df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 -- 172.21.15.116:0/446722685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 msgr2=0x55569bf21dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf21dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 -- 172.21.15.116:0/446722685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 msgr2=0x55569bf2df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf2df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 -- 172.21.15.116:0/446722685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55569be1ab20 con 0x55569bf19af0 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fed36b640 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf2a910 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400c6a0 tx=0x7f6fe40a3590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814a554369203e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.706+0000 7f6fd37fe640 1 -- 172.21.15.116:0/446722685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4098f40 con 0x55569bf19af0 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6fd37fe640 1 -- 172.21.15.116:0/446722685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fe4010b50 con 0x55569bf19af0 2024-03-24T09:25:58.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6fd37fe640 1 -- 172.21.15.116:0/446722685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40ce2e0 con 0x55569bf19af0 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 -- 172.21.15.116:0/446722685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 msgr2=0x55569bf2a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf2a910 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400c6a0 tx=0x7f6fe40a3590 comp rx=0 tx=0).stop 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 -- 172.21.15.116:0/446722685 shutdown_connections 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf226d0 0x55569bf2df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf19af0 0x55569bf2a910 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/446722685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf21dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 -- 172.21.15.116:0/446722685 >> 172.21.15.116:0/446722685 conn(0x55569bf13900 msgr2=0x55569bf122d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 -- 172.21.15.116:0/446722685 shutdown_connections 2024-03-24T09:25:58.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 -- 172.21.15.116:0/446722685 wait complete. 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 -- start start 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.707+0000 7f6ff29a8740 1 Processor -- start 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf90d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf19af0 0x55569bf91260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 0x55569bf9bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55569bf9c0f0 con 0x55569bf226d0 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55569bf9c230 con 0x55569bf19af0 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55569bf9c370 con 0x55569bf191d0 2024-03-24T09:25:58.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6ff29a8740 10 client.0 _ll_register_callbacks cb 0x7f6fe4097080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 0x55569bf9bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 0x55569bf9bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56242/0 (socket says 172.21.15.116:56242) 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 -- 172.21.15.116:0/3908516218 learned_addr learned my addr 172.21.15.116:0/3908516218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf90d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fed36b640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf19af0 0x55569bf91260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 -- 172.21.15.116:0/3908516218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 msgr2=0x55569bf90d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf90d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 -- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf19af0 msgr2=0x55569bf91260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:58.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf19af0 0x55569bf91260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 -- 172.21.15.116:0/3908516218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe81a4450 con 0x55569bf226d0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fee36d640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 0x55569bf9bbb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000f9f0 tx=0x7f6fe001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7ad5aa54da6181 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.708+0000 7f6fd17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6fd17fa640 1 -- 172.21.15.116:0/3908516218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0004b20 con 0x55569bf226d0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6fd17fa640 1 -- 172.21.15.116:0/3908516218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6fe0004c80 con 0x55569bf226d0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fd802de90 con 0x55569bf226d0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6fd17fa640 1 -- 172.21.15.116:0/3908516218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6fe0004e40 con 0x55569bf226d0 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 10 client.16725 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:58.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55569be40700 con 0x55569bf226d0 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6fd17fa640 1 -- 172.21.15.116:0/3908516218 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6fd802de90 con 0x55569bf226d0 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6fd17fa640 1 -- 172.21.15.116:0/3908516218 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f6fe0029470 con 0x55569bf226d0 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 10 client.16725 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 20 client.16725 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf9dd30 0x55569bfa0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 4 client.16725 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55569bc0b890 con 0x55569bf9dd30 2024-03-24T09:25:58.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.709+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf9dd30 0x55569bfa0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6fedb6c640 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf9dd30 0x55569bfa0150 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb6272438af5c21 server_cookie=fefe8ce5f7b4931 in_seq=0 out_seq=0 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6fd17fa640 10 client.16725 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6fd17fa640 1 -- 172.21.15.116:0/3908516218 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55569bc0b890 con 0x55569bf9dd30 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6fd17fa640 10 client.16725 handle_command_reply: tid=0 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 client.16725 shutdown 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 msgr2=0x55569bf9bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 0x55569bf9bbb0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6fe000f9f0 tx=0x7f6fe001b000 comp rx=0 tx=0).stop 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 shutdown_connections 2024-03-24T09:25:58.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55569bf226d0 0x55569bf9bbb0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55569bf9dd30 0x55569bfa0150 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55569bf19af0 0x55569bf91260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 --2- 172.21.15.116:0/3908516218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55569bf191d0 0x55569bf90d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6fd17fa640 0 client.16725 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 >> 172.21.15.116:0/3908516218 conn(0x55569bf13900 msgr2=0x55569bf12b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 shutdown_connections 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 1 -- 172.21.15.116:0/3908516218 wait complete. 2024-03-24T09:25:58.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.710+0000 7f6ff29a8740 20 client.16725 trim_cache size 0 max 16384 2024-03-24T09:25:58.721 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:25:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:25:58.721 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:25:58.721 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6fd40a22b0 msgr2=0x7f6fd40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6fd40a22b0 0x7f6fd40a4770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0005fd0 tx=0x7f6fe0013ae0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 msgr2=0x7f6fe81990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 0x7f6fe81990a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400e9f0 tx=0x7f6fe400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 shutdown_connections 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6fe8106640 0x7f6fe81a3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe8100ff0 0x7f6fe81990a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6fd40a22b0 0x7f6fd40a4770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 --2- 172.21.15.116:0/1514852411 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6fe8100630 0x7f6fe8198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 >> 172.21.15.116:0/1514852411 conn(0x7f6fe8077770 msgr2=0x7f6fe8103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 shutdown_connections 2024-03-24T09:25:58.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.711+0000 7f6fefdd0640 1 -- 172.21.15.116:0/1514852411 wait complete. 2024-03-24T09:25:58.723 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:58 smithi116 ceph-mon[32207]: pgmap v119: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T09:25:58.724 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:25:58.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:25:58.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:58 smithi143 ceph-mon[35965]: pgmap v119: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T09:25:58.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:58 smithi117 ceph-mon[35788]: pgmap v119: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T09:25:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 -- 172.21.15.116:0/280321825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7964100f80 msgr2=0x7f796410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 --2- 172.21.15.116:0/280321825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7964100f80 0x7f796410d490 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f795c009c10 tx=0x7f795c02bef0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 -- 172.21.15.116:0/280321825 shutdown_connections 2024-03-24T09:25:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 --2- 172.21.15.116:0/280321825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7964100f80 0x7f796410d490 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 --2- 172.21.15.116:0/280321825 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79641005c0 0x7f7964100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 --2- 172.21.15.116:0/280321825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7964106660 0x7f7964106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 -- 172.21.15.116:0/280321825 >> 172.21.15.116:0/280321825 conn(0x7f7964076990 msgr2=0x7f7964076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:58.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 -- 172.21.15.116:0/280321825 shutdown_connections 2024-03-24T09:25:58.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.826+0000 7f796b93e640 1 -- 172.21.15.116:0/280321825 wait complete. 2024-03-24T09:25:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 Processor -- start 2024-03-24T09:25:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 -- start start 2024-03-24T09:25:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 0x7f796419c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7964100f80 0x7f796419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7964106660 0x7f79641968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7964196f40 con 0x7f79641005c0 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f79641970b0 con 0x7f7964106660 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f796b93e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7964197270 con 0x7f7964100f80 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f79696da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 0x7f796419c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f79696da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 0x7f796419c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56258/0 (socket says 172.21.15.116:56258) 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f79696da640 1 -- 172.21.15.116:0/3453556284 learned_addr learned my addr 172.21.15.116:0/3453556284 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:58.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f7968ed9640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7964100f80 0x7f796419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.827+0000 7f7969edb640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7964106660 0x7f79641968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f79696da640 1 -- 172.21.15.116:0/3453556284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7964100f80 msgr2=0x7f796419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f79696da640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7964100f80 0x7f796419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f79696da640 1 -- 172.21.15.116:0/3453556284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7964106660 msgr2=0x7f79641968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f79696da640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7964106660 0x7f79641968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f79696da640 1 -- 172.21.15.116:0/3453556284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7960009560 con 0x7f79641005c0 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f79696da640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 0x7f796419c740 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f796000e9a0 tx=0x7f796000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796000cd80 con 0x7f79641005c0 2024-03-24T09:25:58.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79600108e0 con 0x7f79641005c0 2024-03-24T09:25:58.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795c009840 con 0x7f79641005c0 2024-03-24T09:25:58.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79600188e0 con 0x7f79641005c0 2024-03-24T09:25:58.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.828+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7964197740 con 0x7f79641005c0 2024-03-24T09:25:58.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.829+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7964101d00 con 0x7f79641005c0 2024-03-24T09:25:58.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.830+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7960010a50 con 0x7f79641005c0 2024-03-24T09:25:58.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.830+0000 7f795a7fc640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79500a22b0 0x7f79500a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:58.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.831+0000 7f7968ed9640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79500a22b0 0x7f79500a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:58.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.831+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7960014070 con 0x7f79641005c0 2024-03-24T09:25:58.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.831+0000 7f7968ed9640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79500a22b0 0x7f79500a4770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7964105c70 tx=0x7f7954005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:58.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:58.834+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f7960097890 con 0x7f79641005c0 2024-03-24T09:25:59.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.046+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7964068950 con 0x7f79641005c0 2024-03-24T09:25:59.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.047+0000 7f795a7fc640 1 -- 172.21.15.116:0/3453556284 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7f796009af50 con 0x7f79641005c0 2024-03-24T09:25:59.048 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:25:59.048 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:25:59.049 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:25:59.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79500a22b0 msgr2=0x7f79500a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79500a22b0 0x7f79500a4770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7964105c70 tx=0x7f7954005f90 comp rx=0 tx=0).stop 2024-03-24T09:25:59.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 msgr2=0x7f796419c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 0x7f796419c740 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f796000e9a0 tx=0x7f796000ee60 comp rx=0 tx=0).stop 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 shutdown_connections 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7964106660 0x7f79641968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79500a22b0 0x7f79500a4770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7964100f80 0x7f796419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 --2- 172.21.15.116:0/3453556284 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79641005c0 0x7f796419c740 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 >> 172.21.15.116:0/3453556284 conn(0x7f7964076990 msgr2=0x7f79640feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 shutdown_connections 2024-03-24T09:25:59.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.049+0000 7f796b93e640 1 -- 172.21.15.116:0/3453556284 wait complete. 2024-03-24T09:25:59.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:25:59.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 -- 172.21.15.116:0/1197860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100630 msgr2=0x7f3104100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 --2- 172.21.15.116:0/1197860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100630 0x7f3104100ab0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009930 tx=0x7f30f402bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:59.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 -- 172.21.15.116:0/1197860 shutdown_connections 2024-03-24T09:25:59.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 --2- 172.21.15.116:0/1197860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100ff0 0x7f310410d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 --2- 172.21.15.116:0/1197860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100630 0x7f3104100ab0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 --2- 172.21.15.116:0/1197860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3104106640 0x7f3104106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 -- 172.21.15.116:0/1197860 >> 172.21.15.116:0/1197860 conn(0x7f3104077770 msgr2=0x7f3104077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 -- 172.21.15.116:0/1197860 shutdown_connections 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 -- 172.21.15.116:0/1197860 wait complete. 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 Processor -- start 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 -- start start 2024-03-24T09:25:59.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.165+0000 7f310b29d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100630 0x7f31040fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 0x7f31040fdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3104106640 0x7f31040fe030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31041a6130 con 0x7f3104100ff0 2024-03-24T09:25:59.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f31041a62a0 con 0x7f3104100630 2024-03-24T09:25:59.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f31041a6460 con 0x7f3104106640 2024-03-24T09:25:59.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 0x7f31040fdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 0x7f31040fdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56284/0 (socket says 172.21.15.116:56284) 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 -- 172.21.15.116:0/1220576788 learned_addr learned my addr 172.21.15.116:0/1220576788 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3109039640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100630 0x7f31040fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310983a640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3104106640 0x7f31040fe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 -- 172.21.15.116:0/1220576788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3104106640 msgr2=0x7f31040fe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3104106640 0x7f31040fe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 -- 172.21.15.116:0/1220576788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100630 msgr2=0x7f31040fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100630 0x7f31040fd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 -- 172.21.15.116:0/1220576788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f4009560 con 0x7f3104100ff0 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3109039640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100630 0x7f31040fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f3108838640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 0x7f31040fdaf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009230 tx=0x7f30f4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f30fa7fc640 1 -- 172.21.15.116:0/1220576788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4004350 con 0x7f3104100ff0 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f30fa7fc640 1 -- 172.21.15.116:0/1220576788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30f402de90 con 0x7f3104100ff0 2024-03-24T09:25:59.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f30fa7fc640 1 -- 172.21.15.116:0/1220576788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30f4005220 con 0x7f3104100ff0 2024-03-24T09:25:59.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31041a66e0 con 0x7f3104100ff0 2024-03-24T09:25:59.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.166+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31041a6c40 con 0x7f3104100ff0 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.168+0000 7f30fa7fc640 1 -- 172.21.15.116:0/1220576788 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f30f4003c30 con 0x7f3104100ff0 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 Processor -- start 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 -- start start 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f30fa7fc640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f30f00a22e0 0x7f30f00a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9e712f840 con 0x55a9e71c1d10 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9e712cd70 con 0x55a9e71ca8f0 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310de83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9e7128550 con 0x55a9e71c13f0 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f3109039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310983a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f3109039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56294/0 (socket says 172.21.15.116:56294) 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f3109039640 1 -- 172.21.15.116:0/180167855 learned_addr learned my addr 172.21.15.116:0/180167855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310983a640 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f3108838640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f30f00a22e0 0x7f30f00a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f30fa7fc640 1 -- 172.21.15.116:0/1220576788 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f30f40f0200 con 0x7f3104100ff0 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310983a640 1 -- 172.21.15.116:0/180167855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 msgr2=0x55a9e71ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310983a640 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310983a640 1 -- 172.21.15.116:0/180167855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 msgr2=0x55a9e71ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.169+0000 7f310983a640 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.170+0000 7f310983a640 1 -- 172.21.15.116:0/180167855 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e70bdde0 con 0x55a9e71c13f0 2024-03-24T09:25:59.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.170+0000 7f3108838640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f30f00a22e0 0x7f30f00a47a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f30f40ed760 tx=0x7f30f4075040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.170+0000 7f310983a640 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71c17d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f30fc011a70 tx=0x7f30fc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=625d26d399cd20c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.170+0000 7f30ef7fe640 1 -- 172.21.15.116:0/180167855 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc004b20 con 0x55a9e71c13f0 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.170+0000 7f30ef7fe640 1 -- 172.21.15.116:0/180167855 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30fc004c80 con 0x55a9e71c13f0 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.170+0000 7f30ef7fe640 1 -- 172.21.15.116:0/180167855 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc0054f0 con 0x55a9e71c13f0 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- 172.21.15.116:0/180167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 msgr2=0x55a9e71c17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71c17d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f30fc011a70 tx=0x7f30fc004ab0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- 172.21.15.116:0/180167855 shutdown_connections 2024-03-24T09:25:59.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- 172.21.15.116:0/180167855 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71c17d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- 172.21.15.116:0/180167855 >> 172.21.15.116:0/180167855 conn(0x55a9e71bbb20 msgr2=0x55a9e71bbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- 172.21.15.116:0/180167855 shutdown_connections 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- 172.21.15.116:0/180167855 wait complete. 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- start start 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 Processor -- start 2024-03-24T09:25:59.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 0x55a9e71c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c1d10 0x55a9e71c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71ca8f0 0x55a9e71c8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9e6dfb430 con 0x55a9e71ca8f0 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9e6db7e00 con 0x55a9e71c13f0 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9e70ad590 con 0x55a9e71c1d10 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.171+0000 7f310de83740 10 client.0 _ll_register_callbacks cb 0x55a9e711cfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 0x55a9e71c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 0x55a9e71c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44888/0 (socket says 172.21.15.116:44888) 2024-03-24T09:25:59.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 -- 172.21.15.116:0/651962820 learned_addr learned my addr 172.21.15.116:0/651962820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f3109039640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c1d10 0x55a9e71c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71ca8f0 0x55a9e71c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 -- 172.21.15.116:0/651962820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c1d10 msgr2=0x55a9e71c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c1d10 0x55a9e71c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 -- 172.21.15.116:0/651962820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71ca8f0 msgr2=0x55a9e71c8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71ca8f0 0x55a9e71c8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 -- 172.21.15.116:0/651962820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc019040 con 0x55a9e71c13f0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f3109039640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c1d10 0x55a9e71c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310983a640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 0x55a9e71c79c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009560 tx=0x7f30fc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c75eb429179a937 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f30ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f30ecff9640 1 -- 172.21.15.116:0/651962820 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30fc011020 con 0x55a9e71c13f0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a9e6e277e0 con 0x55a9e71c13f0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f30ecff9640 1 -- 172.21.15.116:0/651962820 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f30fc011180 con 0x55a9e71c13f0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f30ecff9640 1 -- 172.21.15.116:0/651962820 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f30fc00f7c0 con 0x55a9e71c13f0 2024-03-24T09:25:59.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310de83740 10 client.26570 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a9e7126bd0 con 0x55a9e71c13f0 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.172+0000 7f30ecff9640 1 -- 172.21.15.116:0/651962820 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a9e6e277e0 con 0x55a9e71c13f0 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f30ecff9640 1 -- 172.21.15.116:0/651962820 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f30fc005570 con 0x55a9e71c13f0 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f310de83740 10 client.26570 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f310de83740 10 client.26570 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f310de83740 20 client.26570 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f310de83740 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e71daa60 0x55a9e71dce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f310de83740 4 client.26570 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e6e64720 con 0x55a9e71daa60 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f3109039640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e71daa60 0x55a9e71dce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f30ecff9640 10 client.26570 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.173+0000 7f3109039640 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e71daa60 0x55a9e71dce80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0f96569e6ffa36d server_cookie=388cd863eb727636 in_seq=0 out_seq=0 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f30ecff9640 1 -- 172.21.15.116:0/651962820 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a9e6e64720 con 0x55a9e71daa60 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f30ecff9640 10 client.26570 handle_command_reply: tid=0 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 client.26570 shutdown 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 msgr2=0x55a9e71c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 0x55a9e71c79c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009560 tx=0x7f30fc004a40 comp rx=0 tx=0).stop 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 shutdown_connections 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e71daa60 0x55a9e71dce80 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71ca8f0 0x55a9e71c8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c1d10 0x55a9e71c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f30ecff9640 0 client.26570 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 --2- 172.21.15.116:0/651962820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71c13f0 0x55a9e71c79c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 >> 172.21.15.116:0/651962820 conn(0x55a9e71bbb20 msgr2=0x55a9e71bd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 shutdown_connections 2024-03-24T09:25:59.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 1 -- 172.21.15.116:0/651962820 wait complete. 2024-03-24T09:25:59.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.174+0000 7f310de83740 20 client.26570 trim_cache size 0 max 16384 2024-03-24T09:25:59.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 Processor -- start 2024-03-24T09:25:59.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 -- start start 2024-03-24T09:25:59.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71e2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71d27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9e71bb330 con 0x55a9e71c1d10 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9e71dd4e0 con 0x55a9e71ca8f0 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310de83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9e71c1050 con 0x55a9e71c13f0 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f3109039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f3109039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71e2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56322/0 (socket says 172.21.15.116:56322) 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f3109039640 1 -- 172.21.15.116:0/2333902286 learned_addr learned my addr 172.21.15.116:0/2333902286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.187+0000 7f310983a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f310983a640 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71d27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f3109039640 1 -- 172.21.15.116:0/2333902286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 msgr2=0x55a9e71d61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f3109039640 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f3109039640 1 -- 172.21.15.116:0/2333902286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 msgr2=0x55a9e71d27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f3109039640 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71d27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f3109039640 1 -- 172.21.15.116:0/2333902286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e70c2c80 con 0x55a9e71c1d10 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f3109039640 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71e2710 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x55a9e71c0ec0 tx=0x7f3100005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dce38982497e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f30eeffd640 1 -- 172.21.15.116:0/2333902286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100017040 con 0x55a9e71c1d10 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f30eeffd640 1 -- 172.21.15.116:0/2333902286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3100026070 con 0x55a9e71c1d10 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.188+0000 7f30eeffd640 1 -- 172.21.15.116:0/2333902286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100022460 con 0x55a9e71c1d10 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- 172.21.15.116:0/2333902286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 msgr2=0x55a9e71e2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71e2710 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x55a9e71c0ec0 tx=0x7f3100005f90 comp rx=0 tx=0).stop 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- 172.21.15.116:0/2333902286 shutdown_connections 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71d27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71e2710 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- 172.21.15.116:0/2333902286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- 172.21.15.116:0/2333902286 >> 172.21.15.116:0/2333902286 conn(0x55a9e71bbb20 msgr2=0x55a9e71bcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- 172.21.15.116:0/2333902286 shutdown_connections 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- 172.21.15.116:0/2333902286 wait complete. 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- start start 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 Processor -- start 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a9e71cf9f0 con 0x55a9e71c1d10 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a9e71cfb60 con 0x55a9e71ca8f0 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a9e7240d50 con 0x55a9e71c13f0 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.189+0000 7f310de83740 10 client.0 _ll_register_callbacks cb 0x7f3100009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56336/0 (socket says 172.21.15.116:56336) 2024-03-24T09:25:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 -- 172.21.15.116:0/3259748367 learned_addr learned my addr 172.21.15.116:0/3259748367 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f310983a640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f310983a640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 -- 172.21.15.116:0/3259748367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 msgr2=0x55a9e71d2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 -- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 msgr2=0x55a9e71cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 -- 172.21.15.116:0/3259748367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3100012ae0 con 0x55a9e71c1d10 2024-03-24T09:25:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f3109039640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71cedd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3100009380 tx=0x7f310000fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252a4cffe3a1a9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f30ecff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f30ecff9640 1 -- 172.21.15.116:0/3259748367 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100017040 con 0x55a9e71c1d10 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f30ecff9640 1 -- 172.21.15.116:0/3259748367 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3100022170 con 0x55a9e71c1d10 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30fc004c80 con 0x55a9e71c1d10 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f30ecff9640 1 -- 172.21.15.116:0/3259748367 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3100022330 con 0x55a9e71c1d10 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f310de83740 10 client.16749 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a9e70e8920 con 0x55a9e71c1d10 2024-03-24T09:25:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.190+0000 7f30ecff9640 1 -- 172.21.15.116:0/3259748367 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f30fc004c80 con 0x55a9e71c1d10 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f30ecff9640 1 -- 172.21.15.116:0/3259748367 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f310002cb60 con 0x55a9e71c1d10 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 10 client.16749 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 10 client.16749 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 20 client.16749 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e7242480 0x55a9e7244870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 4 client.16749 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9e6eb3890 con 0x55a9e7242480 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310983a640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e7242480 0x55a9e7244870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f30ecff9640 10 client.16749 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310983a640 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e7242480 0x55a9e7244870 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=797d6a924bcb8ab6 server_cookie=8bb5119f0a5dcfcf in_seq=0 out_seq=0 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f30ecff9640 1 -- 172.21.15.116:0/3259748367 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9e6eb3890 con 0x55a9e7242480 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f30ecff9640 10 client.16749 handle_command_reply: tid=0 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.191+0000 7f310de83740 1 client.16749 shutdown 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 msgr2=0x55a9e71cedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71cedd0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f3100009380 tx=0x7f310000fc20 comp rx=0 tx=0).stop 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 shutdown_connections 2024-03-24T09:25:59.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a9e71ca8f0 0x55a9e71cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55a9e7242480 0x55a9e7244870 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a9e71c1d10 0x55a9e71cedd0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 --2- 172.21.15.116:0/3259748367 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a9e71c13f0 0x55a9e71d2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f30ecff9640 0 client.16749 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 >> 172.21.15.116:0/3259748367 conn(0x55a9e71bbb20 msgr2=0x55a9e71bd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 shutdown_connections 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 1 -- 172.21.15.116:0/3259748367 wait complete. 2024-03-24T09:25:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310de83740 20 client.16749 trim_cache size 0 max 16384 2024-03-24T09:25:59.208 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:59.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f30f00a22e0 msgr2=0x7f30f00a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f30f00a22e0 0x7f30f00a47a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f30f40ed760 tx=0x7f30f4075040 comp rx=0 tx=0).stop 2024-03-24T09:25:59.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 msgr2=0x7f31040fdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 0x7f31040fdaf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009230 tx=0x7f30f4005dd0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 shutdown_connections 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f30f00a22e0 0x7f30f00a47a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3104106640 0x7f31040fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3104100ff0 0x7f31040fdaf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 --2- 172.21.15.116:0/1220576788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3104100630 0x7f31040fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 >> 172.21.15.116:0/1220576788 conn(0x7f3104077770 msgr2=0x7f31040ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 shutdown_connections 2024-03-24T09:25:59.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.192+0000 7f310b29d640 1 -- 172.21.15.116:0/1220576788 wait complete. 2024-03-24T09:25:59.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:25:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.314+0000 7f057cf5c640 1 -- 172.21.15.116:0/775979181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0578100f60 msgr2=0x7f057810d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.314+0000 7f057cf5c640 1 --2- 172.21.15.116:0/775979181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0578100f60 0x7f057810d420 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0570009900 tx=0x7f057002bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 -- 172.21.15.116:0/775979181 shutdown_connections 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 --2- 172.21.15.116:0/775979181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0578100f60 0x7f057810d420 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 --2- 172.21.15.116:0/775979181 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f05781005a0 0x7f0578100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 --2- 172.21.15.116:0/775979181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0578106660 0x7f0578106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 -- 172.21.15.116:0/775979181 >> 172.21.15.116:0/775979181 conn(0x7f0578077710 msgr2=0x7f0578077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 -- 172.21.15.116:0/775979181 shutdown_connections 2024-03-24T09:25:59.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 -- 172.21.15.116:0/775979181 wait complete. 2024-03-24T09:25:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 Processor -- start 2024-03-24T09:25:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.315+0000 7f057cf5c640 1 -- start start 2024-03-24T09:25:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 0x7f05781989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0578100f60 0x7f0578198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0578106660 0x7f05781a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0578199770 con 0x7f05781005a0 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f05781a3f70 con 0x7f0578100f60 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f05781a40b0 con 0x7f0578106660 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 0x7f05781989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 0x7f05781989b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56356/0 (socket says 172.21.15.116:56356) 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 -- 172.21.15.116:0/2691385031 learned_addr learned my addr 172.21.15.116:0/2691385031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0578100f60 0x7f0578198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f0576d9d640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0578106660 0x7f05781a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 -- 172.21.15.116:0/2691385031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0578106660 msgr2=0x7f05781a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0578106660 0x7f05781a3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 -- 172.21.15.116:0/2691385031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0578100f60 msgr2=0x7f0578198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0578100f60 0x7f0578198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 -- 172.21.15.116:0/2691385031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056c009490 con 0x7f05781005a0 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057659c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 0x7f05781989b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0568009d80 tx=0x7f056800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f05677fe640 1 -- 172.21.15.116:0/2691385031 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0568003b70 con 0x7f05781005a0 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0568009840 con 0x7f05781005a0 2024-03-24T09:25:59.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f05677fe640 1 -- 172.21.15.116:0/2691385031 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0568003cd0 con 0x7f05781005a0 2024-03-24T09:25:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f05677fe640 1 -- 172.21.15.116:0/2691385031 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05680107b0 con 0x7f05781005a0 2024-03-24T09:25:59.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.316+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0570009560 con 0x7f05781005a0 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f05677fe640 1 -- 172.21.15.116:0/2691385031 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f0568003e40 con 0x7f05781005a0 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 Processor -- start 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 -- start start 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f05677fe640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f05600a2230 0x7f05600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a447d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5600349b2840 con 0x560034a443f0 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5600349afd70 con 0x560034a4d8f0 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057fc0c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5600349ab550 con 0x560034a44d10 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f0576d9d640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f05600a2230 0x7f05600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f0575d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f0575d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:44090/0 (socket says 172.21.15.116:44090) 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a447d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56362/0 (socket says 172.21.15.116:56362) 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.319+0000 7f0575d9b640 1 -- 172.21.15.116:0/2241991001 learned_addr learned my addr 172.21.15.116:0/2241991001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0576d9d640 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f05677fe640 1 -- 172.21.15.116:0/2691385031 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0568014070 con 0x7f05781005a0 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0575d9b640 1 -- 172.21.15.116:0/2241991001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 msgr2=0x560034a4fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0575d9b640 1 -- 172.21.15.116:0/2241991001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 msgr2=0x560034a447d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a447d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0575d9b640 1 -- 172.21.15.116:0/2241991001 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560034940de0 con 0x560034a44d10 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0576d9d640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f05600a2230 0x7f05600a46f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0570005ab0 tx=0x7f0570005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4d3b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f056c009e20 tx=0x7f056c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf745d6b6373d526 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0564ff9640 1 -- 172.21.15.116:0/2241991001 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c00e670 con 0x560034a44d10 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0564ff9640 1 -- 172.21.15.116:0/2241991001 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f056c00ec80 con 0x560034a44d10 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.320+0000 7f0564ff9640 1 -- 172.21.15.116:0/2241991001 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c019cd0 con 0x560034a44d10 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 -- 172.21.15.116:0/2241991001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 msgr2=0x560034a4d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4d3b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f056c009e20 tx=0x7f056c011810 comp rx=0 tx=0).stop 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 -- 172.21.15.116:0/2241991001 shutdown_connections 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4d3b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2241991001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a447d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 -- 172.21.15.116:0/2241991001 >> 172.21.15.116:0/2241991001 conn(0x560034a3eb20 msgr2=0x560034a3ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 -- 172.21.15.116:0/2241991001 shutdown_connections 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 -- 172.21.15.116:0/2241991001 wait complete. 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 -- start start 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 Processor -- start 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.321+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a4a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56003467e430 con 0x560034a443f0 2024-03-24T09:25:59.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56003463ae00 con 0x560034a4d8f0 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560034930590 con 0x560034a44d10 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 10 client.0 _ll_register_callbacks cb 0x56003499ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a4a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a4a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56374/0 (socket says 172.21.15.116:56374) 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 -- 172.21.15.116:0/2486269419 learned_addr learned my addr 172.21.15.116:0/2486269419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f0576d9d640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 -- 172.21.15.116:0/2486269419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 msgr2=0x560034a4aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 -- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 msgr2=0x560034a4b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 -- 172.21.15.116:0/2486269419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f056c009a80 con 0x560034a443f0 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057659c640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a4a980 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f05680cc5a0 tx=0x7f056809ae40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19a552b5034a471 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f055dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f055dffb640 1 -- 172.21.15.116:0/2486269419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056809b940 con 0x560034a443f0 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f055dffb640 1 -- 172.21.15.116:0/2486269419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f056809b240 con 0x560034a443f0 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5600346aa7e0 con 0x560034a443f0 2024-03-24T09:25:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f055dffb640 1 -- 172.21.15.116:0/2486269419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f056809c570 con 0x560034a443f0 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 10 client.16761 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.322+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600349a9bd0 con 0x560034a443f0 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f055dffb640 1 -- 172.21.15.116:0/2486269419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f056807e020 con 0x560034a443f0 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f055dffb640 1 -- 172.21.15.116:0/2486269419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f05680cc220 con 0x560034a443f0 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f057fc0c740 10 client.16761 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f057fc0c740 10 client.16761 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f057fc0c740 20 client.16761 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034a5da10 0x560034a5fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f057fc0c740 4 client.16761 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600346e7720 con 0x560034a5da10 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034a5da10 0x560034a5fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f055dffb640 10 client.16761 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.323+0000 7f0575d9b640 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034a5da10 0x560034a5fe40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7702a73ac2594a60 server_cookie=6a3efc82f16dbc18 in_seq=0 out_seq=0 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f055dffb640 1 -- 172.21.15.116:0/2486269419 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5600346e7720 con 0x560034a5da10 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f055dffb640 10 client.16761 handle_command_reply: tid=0 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 client.16761 shutdown 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 msgr2=0x560034a4a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a4a980 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f05680cc5a0 tx=0x7f056809ae40 comp rx=0 tx=0).stop 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 shutdown_connections 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034a5da10 0x560034a5fe40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a4d8f0 0x560034a4b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a44d10 0x560034a4aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 --2- 172.21.15.116:0/2486269419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a4a980 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f055dffb640 0 client.16761 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 >> 172.21.15.116:0/2486269419 conn(0x560034a3eb20 msgr2=0x560034a40870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 shutdown_connections 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 1 -- 172.21.15.116:0/2486269419 wait complete. 2024-03-24T09:25:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.324+0000 7f057fc0c740 20 client.16761 trim_cache size 0 max 16384 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 Processor -- start 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 -- start start 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a443f0 0x560034a407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a65710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 0x560034a3db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560034a40d20 con 0x560034a4d8f0 2024-03-24T09:25:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560034a3e650 con 0x560034a44d10 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057fc0c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560034a5d7c0 con 0x560034a443f0 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 0x560034a3db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 0x560034a3db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56378/0 (socket says 172.21.15.116:56378) 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 -- 172.21.15.116:0/1626621276 learned_addr learned my addr 172.21.15.116:0/1626621276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0575d9b640 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a65710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f057659c640 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a443f0 0x560034a407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 -- 172.21.15.116:0/1626621276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a443f0 msgr2=0x560034a407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a443f0 0x560034a407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 -- 172.21.15.116:0/1626621276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 msgr2=0x560034a65710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a65710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 -- 172.21.15.116:0/1626621276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560034945c80 con 0x560034a4d8f0 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.338+0000 7f0576d9d640 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 0x560034a3db40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0570009900 tx=0x7f05700058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38168d66c0e1519b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f055ffff640 1 -- 172.21.15.116:0/1626621276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057002cb00 con 0x560034a4d8f0 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f055ffff640 1 -- 172.21.15.116:0/1626621276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0570003aa0 con 0x560034a4d8f0 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f055ffff640 1 -- 172.21.15.116:0/1626621276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570004fb0 con 0x560034a4d8f0 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 -- 172.21.15.116:0/1626621276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 msgr2=0x560034a3db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 0x560034a3db40 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f0570009900 tx=0x7f05700058d0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 -- 172.21.15.116:0/1626621276 shutdown_connections 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a4d8f0 0x560034a3db40 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a65710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1626621276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a443f0 0x560034a407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 -- 172.21.15.116:0/1626621276 >> 172.21.15.116:0/1626621276 conn(0x560034a3eb20 msgr2=0x560034a3d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 -- 172.21.15.116:0/1626621276 shutdown_connections 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.339+0000 7f057fc0c740 1 -- 172.21.15.116:0/1626621276 wait complete. 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 -- start start 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 Processor -- start 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a53380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a538c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a4d8f0 0x560034a53e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560034ac2f40 con 0x560034a443f0 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560034ac30b0 con 0x560034a44d10 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560034ac3270 con 0x560034a4d8f0 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057fc0c740 10 client.0 _ll_register_callbacks cb 0x7f056801ded0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a53380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a53380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56394/0 (socket says 172.21.15.116:56394) 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 -- 172.21.15.116:0/1782551940 learned_addr learned my addr 172.21.15.116:0/1782551940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f0576d9d640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a4d8f0 0x560034a53e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f0575d9b640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 -- 172.21.15.116:0/1782551940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a4d8f0 msgr2=0x560034a53e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a4d8f0 0x560034a53e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 -- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 msgr2=0x560034a538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 -- 172.21.15.116:0/1782551940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0570048040 con 0x560034a443f0 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f057659c640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a53380 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f056809aef0 tx=0x7f056800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac3a1bbd8505173 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.340+0000 7f055dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f055dffb640 1 -- 172.21.15.116:0/1782551940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05680a39e0 con 0x560034a443f0 2024-03-24T09:25:59.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f055dffb640 1 -- 172.21.15.116:0/1782551940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f056807f080 con 0x560034a443f0 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f056c00ec80 con 0x560034a443f0 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f055dffb640 1 -- 172.21.15.116:0/1782551940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f056809b010 con 0x560034a443f0 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 10 client.16767 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56003496b920 con 0x560034a443f0 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f055dffb640 1 -- 172.21.15.116:0/1782551940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f056c00ec80 con 0x560034a443f0 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f055dffb640 1 -- 172.21.15.116:0/1782551940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f056809a690 con 0x560034a443f0 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 10 client.16767 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 10 client.16767 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:25:59.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 20 client.16767 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034ac4500 0x560034ac68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 4 client.16767 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:59.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560034736890 con 0x560034ac4500 2024-03-24T09:25:59.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f0575d9b640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034ac4500 0x560034ac68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f055dffb640 10 client.16767 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.341+0000 7f0575d9b640 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034ac4500 0x560034ac68f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1b9e72f0909ea900 server_cookie=4c3f213a5843aad in_seq=0 out_seq=0 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f055dffb640 1 -- 172.21.15.116:0/1782551940 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560034736890 con 0x560034ac4500 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f055dffb640 10 client.16767 handle_command_reply: tid=0 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 client.16767 shutdown 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 msgr2=0x560034a53380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a53380 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f056809aef0 tx=0x7f056800cf60 comp rx=0 tx=0).stop 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 shutdown_connections 2024-03-24T09:25:59.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x560034ac4500 0x560034ac68f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560034a4d8f0 0x560034a53e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560034a44d10 0x560034a538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f055dffb640 0 client.16767 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 --2- 172.21.15.116:0/1782551940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560034a443f0 0x560034a53380 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 >> 172.21.15.116:0/1782551940 conn(0x560034a3eb20 msgr2=0x560034a3dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 shutdown_connections 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 1 -- 172.21.15.116:0/1782551940 wait complete. 2024-03-24T09:25:59.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057fc0c740 20 client.16767 trim_cache size 0 max 16384 2024-03-24T09:25:59.356 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:59.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f05600a2230 msgr2=0x7f05600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f05600a2230 0x7f05600a46f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0570005ab0 tx=0x7f0570005fd0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 msgr2=0x7f05781989b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 0x7f05781989b0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f0568009d80 tx=0x7f056800c6a0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 shutdown_connections 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f05600a2230 0x7f05600a46f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0578106660 0x7f05781a3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0578100f60 0x7f0578198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 --2- 172.21.15.116:0/2691385031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05781005a0 0x7f05781989b0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.342+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 >> 172.21.15.116:0/2691385031 conn(0x7f0578077710 msgr2=0x7f05780ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.343+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 shutdown_connections 2024-03-24T09:25:59.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.343+0000 7f057cf5c640 1 -- 172.21.15.116:0/2691385031 wait complete. 2024-03-24T09:25:59.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:25:59.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 -- 172.21.15.116:0/1666494285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68106660 msgr2=0x7fca68106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 --2- 172.21.15.116:0/1666494285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68106660 0x7fca68106a40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009890 tx=0x7fca5c02bb70 comp rx=0 tx=0).stop 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 -- 172.21.15.116:0/1666494285 shutdown_connections 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 --2- 172.21.15.116:0/1666494285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca68100f80 0x7fca6810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 --2- 172.21.15.116:0/1666494285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca681005c0 0x7fca68100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 --2- 172.21.15.116:0/1666494285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68106660 0x7fca68106a40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 -- 172.21.15.116:0/1666494285 >> 172.21.15.116:0/1666494285 conn(0x7fca68076990 msgr2=0x7fca68076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 -- 172.21.15.116:0/1666494285 shutdown_connections 2024-03-24T09:25:59.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.464+0000 7fca6f67c640 1 -- 172.21.15.116:0/1666494285 wait complete. 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 Processor -- start 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 -- start start 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca681005c0 0x7fca6819d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 0x7fca6819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca68106660 0x7fca68197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca681980c0 con 0x7fca68100f80 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca68198230 con 0x7fca68106660 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fca681983f0 con 0x7fca681005c0 2024-03-24T09:25:59.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 0x7fca6819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 0x7fca6819de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56420/0 (socket says 172.21.15.116:56420) 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 -- 172.21.15.116:0/3430719603 learned_addr learned my addr 172.21.15.116:0/3430719603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6d418640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca681005c0 0x7fca6819d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6dc19640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca68106660 0x7fca68197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 -- 172.21.15.116:0/3430719603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca681005c0 msgr2=0x7fca6819d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca681005c0 0x7fca6819d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 -- 172.21.15.116:0/3430719603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca68106660 msgr2=0x7fca68197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca68106660 0x7fca68197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 -- 172.21.15.116:0/3430719603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca60009900 con 0x7fca68100f80 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6cc17640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 0x7fca6819de00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fca6400e9f0 tx=0x7fca6400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca5a7fc640 1 -- 172.21.15.116:0/3430719603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca64009720 con 0x7fca68100f80 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.465+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5c009490 con 0x7fca68100f80 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.466+0000 7fca5a7fc640 1 -- 172.21.15.116:0/3430719603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca6400cd80 con 0x7fca68100f80 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.466+0000 7fca5a7fc640 1 -- 172.21.15.116:0/3430719603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca64010430 con 0x7fca68100f80 2024-03-24T09:25:59.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.466+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca681a4780 con 0x7fca68100f80 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca5a7fc640 1 -- 172.21.15.116:0/3430719603 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fca64005020 con 0x7fca68100f80 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 Processor -- start 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 -- start start 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5460a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f54615ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f54578840 con 0x563f5460ad10 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f54575d70 con 0x563f5460a3f0 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca72257740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f54571550 con 0x563f546138f0 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.468+0000 7fca5a7fc640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fca540a2230 0x7fca540a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6d418640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5460a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f54615ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5460a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:45002/0 (socket says 172.21.15.116:45002) 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 -- 172.21.15.116:0/1091160976 learned_addr learned my addr 172.21.15.116:0/1091160976 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fca540a2230 0x7fca540a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca5a7fc640 1 -- 172.21.15.116:0/3430719603 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fca64014070 con 0x7fca68100f80 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 -- 172.21.15.116:0/1091160976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 msgr2=0x563f54615ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f54615ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 -- 172.21.15.116:0/1091160976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 msgr2=0x563f546133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 -- 172.21.15.116:0/1091160976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f54506de0 con 0x563f5460a3f0 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5460a7d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c4a0 tx=0x7fca6000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af6b874fbb558c9b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca6dc19640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fca540a2230 0x7fca540a46f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca6000ce60 tx=0x7fca60011000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca537fe640 1 -- 172.21.15.116:0/1091160976 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60013030 con 0x563f5460a3f0 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca537fe640 1 -- 172.21.15.116:0/1091160976 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca60014040 con 0x563f5460a3f0 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.469+0000 7fca537fe640 1 -- 172.21.15.116:0/1091160976 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60004620 con 0x563f5460a3f0 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 -- 172.21.15.116:0/1091160976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 msgr2=0x563f5460a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5460a7d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c4a0 tx=0x7fca6000c960 comp rx=0 tx=0).stop 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 -- 172.21.15.116:0/1091160976 shutdown_connections 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f54615ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 --2- 172.21.15.116:0/1091160976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5460a7d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 -- 172.21.15.116:0/1091160976 >> 172.21.15.116:0/1091160976 conn(0x563f54604b20 msgr2=0x563f54604f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 -- 172.21.15.116:0/1091160976 shutdown_connections 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 -- 172.21.15.116:0/1091160976 wait complete. 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 -- start start 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.470+0000 7fca72257740 1 Processor -- start 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f546109e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460ad10 0x563f54610f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 0x563f54611460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f54244430 con 0x563f546138f0 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f54200e00 con 0x563f5460ad10 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f544f6590 con 0x563f5460a3f0 2024-03-24T09:25:59.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca72257740 10 client.0 _ll_register_callbacks cb 0x563f54565fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 0x563f54611460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 0x563f54611460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56436/0 (socket says 172.21.15.116:56436) 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460ad10 0x563f54610f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 -- 172.21.15.116:0/30618251 learned_addr learned my addr 172.21.15.116:0/30618251 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6d418640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f546109e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 -- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 msgr2=0x563f546109e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f546109e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 -- 172.21.15.116:0/30618251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460ad10 msgr2=0x563f54610f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460ad10 0x563f54610f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 -- 172.21.15.116:0/30618251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca6000ec80 con 0x563f546138f0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6cc17640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460ad10 0x563f54610f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca6dc19640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 0x563f54611460 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fca60004260 tx=0x7fca6000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919cbeb14ce654b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.471+0000 7fca50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca50ff9640 1 -- 172.21.15.116:0/30618251 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60013040 con 0x563f546138f0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca50ff9640 1 -- 172.21.15.116:0/30618251 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fca60003d80 con 0x563f546138f0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563f542707e0 con 0x563f546138f0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca50ff9640 1 -- 172.21.15.116:0/30618251 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fca60003f40 con 0x563f546138f0 2024-03-24T09:25:59.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 10 client.16779 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563f5456fbd0 con 0x563f546138f0 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca50ff9640 1 -- 172.21.15.116:0/30618251 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563f542707e0 con 0x563f546138f0 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca50ff9640 1 -- 172.21.15.116:0/30618251 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fca6002d430 con 0x563f546138f0 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 10 client.16779 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 10 client.16779 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 20 client.16779 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f54623a60 0x563f54625e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 4 client.16779 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f542ad720 con 0x563f54623a60 2024-03-24T09:25:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.472+0000 7fca6d418640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f54623a60 0x563f54625e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca50ff9640 10 client.16779 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca6d418640 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f54623a60 0x563f54625e80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=40955c6b1978c0c6 server_cookie=43d5834ddde9e914 in_seq=0 out_seq=0 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca50ff9640 1 -- 172.21.15.116:0/30618251 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f542ad720 con 0x563f54623a60 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca50ff9640 10 client.16779 handle_command_reply: tid=0 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca72257740 1 client.16779 shutdown 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 msgr2=0x563f54611460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.473+0000 7fca72257740 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 0x563f54611460 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fca60004260 tx=0x7fca6000cf50 comp rx=0 tx=0).stop 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 shutdown_connections 2024-03-24T09:25:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f546138f0 0x563f54611460 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f54623a60 0x563f54625e80 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460ad10 0x563f54610f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca50ff9640 0 client.16779 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 --2- 172.21.15.116:0/30618251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f546109e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 >> 172.21.15.116:0/30618251 conn(0x563f54604b20 msgr2=0x563f54606870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 shutdown_connections 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 1 -- 172.21.15.116:0/30618251 wait complete. 2024-03-24T09:25:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.474+0000 7fca72257740 20 client.16779 trim_cache size 0 max 16384 2024-03-24T09:25:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 Processor -- start 2024-03-24T09:25:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 -- start start 2024-03-24T09:25:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f54603f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546102c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f546138f0 0x563f5461f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f54610800 con 0x563f5460ad10 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f54616070 con 0x563f546138f0 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca72257740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f54609b90 con 0x563f5460a3f0 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546102c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546102c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56440/0 (socket says 172.21.15.116:56440) 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 -- 172.21.15.116:0/3460574343 learned_addr learned my addr 172.21.15.116:0/3460574343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6d418640 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f54603f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6dc19640 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f546138f0 0x563f5461f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 -- 172.21.15.116:0/3460574343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 msgr2=0x563f54603f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f54603f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 -- 172.21.15.116:0/3460574343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f546138f0 msgr2=0x563f5461f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f546138f0 0x563f5461f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 -- 172.21.15.116:0/3460574343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f5450bc80 con 0x563f5460ad10 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.487+0000 7fca6cc17640 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546102c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fca64097260 tx=0x7fca6406df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1324e4b0e9b10374 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca52ffd640 1 -- 172.21.15.116:0/3460574343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6409d030 con 0x563f5460ad10 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca52ffd640 1 -- 172.21.15.116:0/3460574343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca640046c0 con 0x563f5460ad10 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca52ffd640 1 -- 172.21.15.116:0/3460574343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6409b060 con 0x563f5460ad10 2024-03-24T09:25:59.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 -- 172.21.15.116:0/3460574343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 msgr2=0x563f546102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546102c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fca64097260 tx=0x7fca6406df30 comp rx=0 tx=0).stop 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 -- 172.21.15.116:0/3460574343 shutdown_connections 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f546138f0 0x563f5461f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f546102c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 --2- 172.21.15.116:0/3460574343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f5460a3f0 0x563f54603f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 -- 172.21.15.116:0/3460574343 >> 172.21.15.116:0/3460574343 conn(0x563f54604b20 msgr2=0x563f54611230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 -- 172.21.15.116:0/3460574343 shutdown_connections 2024-03-24T09:25:59.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 -- 172.21.15.116:0/3460574343 wait complete. 2024-03-24T09:25:59.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 -- start start 2024-03-24T09:25:59.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.488+0000 7fca72257740 1 Processor -- start 2024-03-24T09:25:59.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5461b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f5460d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f5460d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f5460dcd0 con 0x563f5460ad10 2024-03-24T09:25:59.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f5460de40 con 0x563f5460a3f0 2024-03-24T09:25:59.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f5460e000 con 0x563f546138f0 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca72257740 10 client.0 _ll_register_callbacks cb 0x7fca60027070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f5460d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f5460d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56444/0 (socket says 172.21.15.116:56444) 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 -- 172.21.15.116:0/762607018 learned_addr learned my addr 172.21.15.116:0/762607018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6d418640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5461b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6dc19640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f5460d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 -- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 msgr2=0x563f5460d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f5460d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 -- 172.21.15.116:0/762607018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 msgr2=0x563f5461b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5461b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 -- 172.21.15.116:0/762607018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca681a4780 con 0x563f5460ad10 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca6cc17640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f5460d250 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fca6409ab00 tx=0x7fca64097110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ebea2357034c02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca50ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.489+0000 7fca50ff9640 1 -- 172.21.15.116:0/762607018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6409d030 con 0x563f5460ad10 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca50ff9640 1 -- 172.21.15.116:0/762607018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fca6409e070 con 0x563f5460ad10 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca60014040 con 0x563f5460ad10 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca50ff9640 1 -- 172.21.15.116:0/762607018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fca6409b500 con 0x563f5460ad10 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 10 client.16785 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563f54531920 con 0x563f5460ad10 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca50ff9640 1 -- 172.21.15.116:0/762607018 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fca60014040 con 0x563f5460ad10 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca50ff9640 1 -- 172.21.15.116:0/762607018 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fca640cd040 con 0x563f5460ad10 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 10 client.16785 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 10 client.16785 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 20 client.16785 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f5468a0c0 0x563f5468c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 4 client.16785 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f542fc890 con 0x563f5468a0c0 2024-03-24T09:25:59.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.490+0000 7fca6d418640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f5468a0c0 0x563f5468c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca50ff9640 10 client.16785 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca6d418640 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f5468a0c0 0x563f5468c4e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee1fedaa2bdc7788 server_cookie=154bb1ba63d426e8 in_seq=0 out_seq=0 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca50ff9640 1 -- 172.21.15.116:0/762607018 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f542fc890 con 0x563f5468a0c0 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca50ff9640 10 client.16785 handle_command_reply: tid=0 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 client.16785 shutdown 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 msgr2=0x563f5460d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f5460d250 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fca6409ab00 tx=0x7fca64097110 comp rx=0 tx=0).stop 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 shutdown_connections 2024-03-24T09:25:59.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f546138f0 0x563f5460d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f5460ad10 0x563f5460d250 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f5468a0c0 0x563f5468c4e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 --2- 172.21.15.116:0/762607018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f5460a3f0 0x563f5461b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca50ff9640 0 client.16785 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 >> 172.21.15.116:0/762607018 conn(0x563f54604b20 msgr2=0x563f54605c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 shutdown_connections 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 1 -- 172.21.15.116:0/762607018 wait complete. 2024-03-24T09:25:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca72257740 20 client.16785 trim_cache size 0 max 16384 2024-03-24T09:25:59.502 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:59.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fca540a2230 msgr2=0x7fca540a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca6f67c640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fca540a2230 0x7fca540a46f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fca6000ce60 tx=0x7fca60011000 comp rx=0 tx=0).stop 2024-03-24T09:25:59.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 msgr2=0x7fca6819de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.491+0000 7fca6f67c640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 0x7fca6819de00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fca6400e9f0 tx=0x7fca6400edf0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 shutdown_connections 2024-03-24T09:25:59.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca68106660 0x7fca68197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca68100f80 0x7fca6819de00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fca540a2230 0x7fca540a46f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 --2- 172.21.15.116:0/3430719603 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fca681005c0 0x7fca6819d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 >> 172.21.15.116:0/3430719603 conn(0x7fca68076990 msgr2=0x7fca680feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 shutdown_connections 2024-03-24T09:25:59.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.492+0000 7fca6f67c640 1 -- 172.21.15.116:0/3430719603 wait complete. 2024-03-24T09:25:59.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:25:59.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 -- 172.21.15.116:0/3768678631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 msgr2=0x7fc97c100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 --2- 172.21.15.116:0/3768678631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c100ab0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc978009930 tx=0x7fc97802bc30 comp rx=0 tx=0).stop 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 -- 172.21.15.116:0/3768678631 shutdown_connections 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 --2- 172.21.15.116:0/3768678631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc97c100ff0 0x7fc97c10d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 --2- 172.21.15.116:0/3768678631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c100ab0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 --2- 172.21.15.116:0/3768678631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc97c106640 0x7fc97c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 -- 172.21.15.116:0/3768678631 >> 172.21.15.116:0/3768678631 conn(0x7fc97c077770 msgr2=0x7fc97c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 -- 172.21.15.116:0/3768678631 shutdown_connections 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.607+0000 7fc983aeb640 1 -- 172.21.15.116:0/3768678631 wait complete. 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 Processor -- start 2024-03-24T09:25:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 -- start start 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c19c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc97c100ff0 0x7fc97c19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc97c106640 0x7fc97c196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c197080 con 0x7fc97c100630 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c1971f0 con 0x7fc97c106640 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc983aeb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c1973b0 con 0x7fc97c100ff0 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c19c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c19c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56460/0 (socket says 172.21.15.116:56460) 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 -- 172.21.15.116:0/4159792828 learned_addr learned my addr 172.21.15.116:0/4159792828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981086640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc97c100ff0 0x7fc97c19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc982088640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc97c106640 0x7fc97c196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 -- 172.21.15.116:0/4159792828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc97c100ff0 msgr2=0x7fc97c19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc97c100ff0 0x7fc97c19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 -- 172.21.15.116:0/4159792828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc97c106640 msgr2=0x7fc97c196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc97c106640 0x7fc97c196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 -- 172.21.15.116:0/4159792828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc974009840 con 0x7fc97c100630 2024-03-24T09:25:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.608+0000 7fc981887640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c19c830 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00e9f0 tx=0x7fc96c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.609+0000 7fc972ffd640 1 -- 172.21.15.116:0/4159792828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c009700 con 0x7fc97c100630 2024-03-24T09:25:59.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.609+0000 7fc972ffd640 1 -- 172.21.15.116:0/4159792828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc96c00ccc0 con 0x7fc97c100630 2024-03-24T09:25:59.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.609+0000 7fc972ffd640 1 -- 172.21.15.116:0/4159792828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c018790 con 0x7fc97c100630 2024-03-24T09:25:59.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.609+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc978009560 con 0x7fc97c100630 2024-03-24T09:25:59.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.609+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c069ce0 con 0x7fc97c100630 2024-03-24T09:25:59.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.611+0000 7fc972ffd640 1 -- 172.21.15.116:0/4159792828 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc96c010430 con 0x7fc97c100630 2024-03-24T09:25:59.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.611+0000 7fc9867a0740 1 Processor -- start 2024-03-24T09:25:59.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 -- start start 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f9063f0 0x55be8f9067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 0x55be8f911ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55be8f874840 con 0x55be8f906d10 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be8f871d70 con 0x55be8f9063f0 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc9867a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be8f86d550 con 0x55be8f90f8f0 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc972ffd640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9680a22b0 0x7fc9680a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981086640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 0x55be8f911ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc982088640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f9063f0 0x55be8f9067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56468/0 (socket says 172.21.15.116:56468) 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 -- 172.21.15.116:0/1118063294 learned_addr learned my addr 172.21.15.116:0/1118063294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981086640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 0x55be8f911ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:44194/0 (socket says 172.21.15.116:44194) 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc982088640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9680a22b0 0x7fc9680a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc972ffd640 1 -- 172.21.15.116:0/4159792828 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc96c014070 con 0x7fc97c100630 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 -- 172.21.15.116:0/1118063294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 msgr2=0x55be8f911ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 0x55be8f911ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 -- 172.21.15.116:0/1118063294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f9063f0 msgr2=0x55be8f9067d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f9063f0 0x55be8f9067d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 -- 172.21.15.116:0/1118063294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be8f802de0 con 0x55be8f906d10 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc982088640 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f9063f0 0x55be8f9067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981086640 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 0x55be8f911ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.612+0000 7fc981887640 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90f3b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc96c097720 tx=0x7fc96c06ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5236ad603e87da53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc982088640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9680a22b0 0x7fc9680a4770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc974009b80 tx=0x7fc974009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc967fff640 1 -- 172.21.15.116:0/1118063294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c09b450 con 0x55be8f906d10 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc967fff640 1 -- 172.21.15.116:0/1118063294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc96c09ad50 con 0x55be8f906d10 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc967fff640 1 -- 172.21.15.116:0/1118063294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0973c0 con 0x55be8f906d10 2024-03-24T09:25:59.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 -- 172.21.15.116:0/1118063294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 msgr2=0x55be8f90f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90f3b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fc96c097720 tx=0x7fc96c06ddc0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 -- 172.21.15.116:0/1118063294 shutdown_connections 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f90f8f0 0x55be8f911ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90f3b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1118063294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f9063f0 0x55be8f9067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 -- 172.21.15.116:0/1118063294 >> 172.21.15.116:0/1118063294 conn(0x55be8f900b20 msgr2=0x55be8f900f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 -- 172.21.15.116:0/1118063294 shutdown_connections 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.613+0000 7fc9867a0740 1 -- 172.21.15.116:0/1118063294 wait complete. 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 -- start start 2024-03-24T09:25:59.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 Processor -- start 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f90d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55be8f540430 con 0x55be8f906d10 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be8f4fce00 con 0x55be8f90f8f0 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be8f7f2590 con 0x55be8f9063f0 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc9867a0740 10 client.0 _ll_register_callbacks cb 0x55be8f861fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc981086640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc981887640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc981086640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56470/0 (socket says 172.21.15.116:56470) 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc981086640 1 -- 172.21.15.116:0/2021333890 learned_addr learned my addr 172.21.15.116:0/2021333890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.614+0000 7fc982088640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f90d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc981086640 1 -- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 msgr2=0x55be8f90c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc981086640 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc981086640 1 -- 172.21.15.116:0/2021333890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 msgr2=0x55be8f90d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc981086640 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f90d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc981086640 1 -- 172.21.15.116:0/2021333890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c069ce0 con 0x55be8f906d10 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc981086640 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90ce90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc978009930 tx=0x7fc97802c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ee5aaca34d79ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9657fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9657fa640 1 -- 172.21.15.116:0/2021333890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978004040 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9657fa640 1 -- 172.21.15.116:0/2021333890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc9780041a0 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be8f56c7e0 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9657fa640 1 -- 172.21.15.116:0/2021333890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc978004360 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9867a0740 10 client.16797 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55be8f86bbd0 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9657fa640 1 -- 172.21.15.116:0/2021333890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55be8f56c7e0 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9657fa640 1 -- 172.21.15.116:0/2021333890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc97803c040 con 0x55be8f906d10 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9867a0740 10 client.16797 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9867a0740 10 client.16797 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.615+0000 7fc9867a0740 20 client.16797 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.616+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f91fa30 0x55be8f921e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.616+0000 7fc9867a0740 4 client.16797 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.616+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be8f5a9720 con 0x55be8f91fa30 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.616+0000 7fc981887640 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f91fa30 0x55be8f921e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.616+0000 7fc9657fa640 10 client.16797 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.616+0000 7fc981887640 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f91fa30 0x55be8f921e50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b2acb2f20580546b server_cookie=f9c0d40f5e450d4d in_seq=0 out_seq=0 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9657fa640 1 -- 172.21.15.116:0/2021333890 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55be8f5a9720 con 0x55be8f91fa30 2024-03-24T09:25:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9657fa640 10 client.16797 handle_command_reply: tid=0 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 client.16797 shutdown 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 msgr2=0x55be8f90ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90ce90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc978009930 tx=0x7fc97802c9a0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 shutdown_connections 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f91fa30 0x55be8f921e50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f90d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f90ce90 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2021333890 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9657fa640 0 client.16797 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 >> 172.21.15.116:0/2021333890 conn(0x55be8f900b20 msgr2=0x55be8f902870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 shutdown_connections 2024-03-24T09:25:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 1 -- 172.21.15.116:0/2021333890 wait complete. 2024-03-24T09:25:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.617+0000 7fc9867a0740 20 client.16797 trim_cache size 0 max 16384 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 Processor -- start 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 -- start start 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f906d10 0x55be8f927720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 0x55be8f917740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55be8f901d50 con 0x55be8f90f8f0 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be8f901b20 con 0x55be8f906d10 2024-03-24T09:25:59.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc9867a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be8f911ce0 con 0x55be8f9063f0 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc982088640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 0x55be8f917740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc981887640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc982088640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 0x55be8f917740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56486/0 (socket says 172.21.15.116:56486) 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc982088640 1 -- 172.21.15.116:0/1992763033 learned_addr learned my addr 172.21.15.116:0/1992763033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.630+0000 7fc981086640 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f906d10 0x55be8f927720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc982088640 1 -- 172.21.15.116:0/1992763033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 msgr2=0x55be8f90e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc982088640 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc982088640 1 -- 172.21.15.116:0/1992763033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f906d10 msgr2=0x55be8f927720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc982088640 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f906d10 0x55be8f927720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc982088640 1 -- 172.21.15.116:0/1992763033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be8f807c80 con 0x55be8f90f8f0 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc982088640 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 0x55be8f917740 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc97400fdf0 tx=0x7fc974009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6a4690a4d819db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9677fe640 1 -- 172.21.15.116:0/1992763033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97400fee0 con 0x55be8f90f8f0 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9677fe640 1 -- 172.21.15.116:0/1992763033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc974004d20 con 0x55be8f90f8f0 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9677fe640 1 -- 172.21.15.116:0/1992763033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9740054a0 con 0x55be8f90f8f0 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 -- 172.21.15.116:0/1992763033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 msgr2=0x55be8f917740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 0x55be8f917740 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc97400fdf0 tx=0x7fc974009af0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 -- 172.21.15.116:0/1992763033 shutdown_connections 2024-03-24T09:25:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f90f8f0 0x55be8f917740 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f906d10 0x55be8f927720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 --2- 172.21.15.116:0/1992763033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f90e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.631+0000 7fc9867a0740 1 -- 172.21.15.116:0/1992763033 >> 172.21.15.116:0/1992763033 conn(0x55be8f900b20 msgr2=0x55be8f90e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 -- 172.21.15.116:0/1992763033 shutdown_connections 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 -- 172.21.15.116:0/1992763033 wait complete. 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 -- start start 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 Processor -- start 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f917a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f914720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f914c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55be8f915360 con 0x55be8f906d10 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be8f9154d0 con 0x55be8f90f8f0 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55be8f985d60 con 0x55be8f9063f0 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc9867a0740 10 client.0 _ll_register_callbacks cb 0x7fc978009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc981086640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f914720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc981086640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f914720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56490/0 (socket says 172.21.15.116:56490) 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc981086640 1 -- 172.21.15.116:0/2532172594 learned_addr learned my addr 172.21.15.116:0/2532172594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc981887640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f917a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.632+0000 7fc982088640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f914c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc981086640 1 -- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 msgr2=0x55be8f917a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc981086640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f917a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc981086640 1 -- 172.21.15.116:0/2532172594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 msgr2=0x55be8f914c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc981086640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f914c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc981086640 1 -- 172.21.15.116:0/2532172594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc974019040 con 0x55be8f906d10 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc981086640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f914720 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x55be8f906160 tx=0x7fc97802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4469e18675045127 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9657fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9657fa640 1 -- 172.21.15.116:0/2532172594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978004440 con 0x55be8f906d10 2024-03-24T09:25:59.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9657fa640 1 -- 172.21.15.116:0/2532172594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc978000e40 con 0x55be8f906d10 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc96c09ad50 con 0x55be8f906d10 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9657fa640 1 -- 172.21.15.116:0/2532172594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc978004fd0 con 0x55be8f906d10 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 10 client.16803 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55be8f82d920 con 0x55be8f906d10 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9657fa640 1 -- 172.21.15.116:0/2532172594 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc96c09ad50 con 0x55be8f906d10 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9657fa640 1 -- 172.21.15.116:0/2532172594 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc97803c040 con 0x55be8f906d10 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 10 client.16803 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 10 client.16803 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 20 client.16803 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f9872e0 0x55be8f989700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 4 client.16803 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.633+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be8f5f8890 con 0x55be8f9872e0 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc981887640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f9872e0 0x55be8f989700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9657fa640 10 client.16803 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc981887640 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f9872e0 0x55be8f989700 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74ef7e715af06180 server_cookie=3aabe3ce6b4344ac in_seq=0 out_seq=0 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9657fa640 1 -- 172.21.15.116:0/2532172594 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be8f5f8890 con 0x55be8f9872e0 2024-03-24T09:25:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9657fa640 10 client.16803 handle_command_reply: tid=0 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 client.16803 shutdown 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 msgr2=0x55be8f914720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f914720 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x55be8f906160 tx=0x7fc97802c640 comp rx=0 tx=0).stop 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 shutdown_connections 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55be8f9872e0 0x55be8f989700 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be8f90f8f0 0x55be8f914c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55be8f906d10 0x55be8f914720 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 --2- 172.21.15.116:0/2532172594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55be8f9063f0 0x55be8f917a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9657fa640 0 client.16803 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:25:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 >> 172.21.15.116:0/2532172594 conn(0x55be8f900b20 msgr2=0x55be8f90e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 shutdown_connections 2024-03-24T09:25:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 1 -- 172.21.15.116:0/2532172594 wait complete. 2024-03-24T09:25:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.634+0000 7fc9867a0740 20 client.16803 trim_cache size 0 max 16384 2024-03-24T09:25:59.645 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9680a22b0 msgr2=0x7fc9680a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9680a22b0 0x7fc9680a4770 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc974009b80 tx=0x7fc974009500 comp rx=0 tx=0).stop 2024-03-24T09:25:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 msgr2=0x7fc97c19c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c19c830 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fc96c00e9f0 tx=0x7fc96c00edf0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 shutdown_connections 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc97c106640 0x7fc97c196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9680a22b0 0x7fc9680a4770 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc97c100ff0 0x7fc97c19cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 --2- 172.21.15.116:0/4159792828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c100630 0x7fc97c19c830 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 >> 172.21.15.116:0/4159792828 conn(0x7fc97c077770 msgr2=0x7fc97c103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 shutdown_connections 2024-03-24T09:25:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.635+0000 7fc983aeb640 1 -- 172.21.15.116:0/4159792828 wait complete. 2024-03-24T09:25:59.647 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:25:59 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3453556284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:25:59.648 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:25:59.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 -- 172.21.15.116:0/1827483025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 msgr2=0x7fdcd410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 --2- 172.21.15.116:0/1827483025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd410d460 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009ca0 tx=0x7fdccc02c180 comp rx=0 tx=0).stop 2024-03-24T09:25:59.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 -- 172.21.15.116:0/1827483025 shutdown_connections 2024-03-24T09:25:59.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 --2- 172.21.15.116:0/1827483025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd410d460 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 --2- 172.21.15.116:0/1827483025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4100d20 0x7fdcd41011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 --2- 172.21.15.116:0/1827483025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41066a0 0x7fdcd4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 -- 172.21.15.116:0/1827483025 >> 172.21.15.116:0/1827483025 conn(0x7fdcd4076bd0 msgr2=0x7fdcd40fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 -- 172.21.15.116:0/1827483025 shutdown_connections 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.751+0000 7fdcda347640 1 -- 172.21.15.116:0/1827483025 wait complete. 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 Processor -- start 2024-03-24T09:25:59.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 -- start start 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4100d20 0x7fdcd419d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41066a0 0x7fdcd4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4198070 con 0x7fdcd41016e0 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd41981e0 con 0x7fdcd4100d20 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcda347640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd41983a0 con 0x7fdcd41066a0 2024-03-24T09:25:59.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd419ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56506/0 (socket says 172.21.15.116:56506) 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcd37fe640 1 -- 172.21.15.116:0/3006194336 learned_addr learned my addr 172.21.15.116:0/3006194336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4100d20 0x7fdcd419d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.752+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41066a0 0x7fdcd4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd37fe640 1 -- 172.21.15.116:0/3006194336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41066a0 msgr2=0x7fdcd4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41066a0 0x7fdcd4197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd37fe640 1 -- 172.21.15.116:0/3006194336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4100d20 msgr2=0x7fdcd419d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4100d20 0x7fdcd419d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd37fe640 1 -- 172.21.15.116:0/3006194336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc4009490 con 0x7fdcd41016e0 2024-03-24T09:25:59.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd419ddb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800e9f0 tx=0x7fdcc800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd17fa640 1 -- 172.21.15.116:0/3006194336 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8009700 con 0x7fdcd41016e0 2024-03-24T09:25:59.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd17fa640 1 -- 172.21.15.116:0/3006194336 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc800cd80 con 0x7fdcd41016e0 2024-03-24T09:25:59.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdccc009900 con 0x7fdcd41016e0 2024-03-24T09:25:59.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcd17fa640 1 -- 172.21.15.116:0/3006194336 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8018840 con 0x7fdcd41016e0 2024-03-24T09:25:59.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.753+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41a4730 con 0x7fdcd41016e0 2024-03-24T09:25:59.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.755+0000 7fdcd17fa640 1 -- 172.21.15.116:0/3006194336 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fdcc80189a0 con 0x7fdcd41016e0 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 Processor -- start 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 -- start start 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccc2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74cccf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a74cc30840 con 0x55a74cccfd00 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a74cc2dd70 con 0x55a74ccc2a40 2024-03-24T09:25:59.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a74cc29550 con 0x55a74ccc2120 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd17fa640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcbc0a2230 0x7fdcbc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74cccf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74cccf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:45094/0 (socket says 172.21.15.116:45094) 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcbc0a2230 0x7fdcbc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 -- 172.21.15.116:0/4012920448 learned_addr learned my addr 172.21.15.116:0/4012920448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56514/0 (socket says 172.21.15.116:56514) 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccc2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd17fa640 1 -- 172.21.15.116:0/3006194336 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdcc8014070 con 0x7fdcd41016e0 2024-03-24T09:25:59.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 -- 172.21.15.116:0/4012920448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 msgr2=0x55a74ccc2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccc2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 -- 172.21.15.116:0/4012920448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 msgr2=0x55a74ccd20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 -- 172.21.15.116:0/4012920448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a74cbbede0 con 0x55a74ccc2a40 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcbc0a2230 0x7fdcbc0a46f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdccc004af0 tx=0x7fdccc02e040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.756+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74cccf7c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009da0 tx=0x7fdcc4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f9715f9f1ebaa52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcba7fc640 1 -- 172.21.15.116:0/4012920448 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc40042f0 con 0x55a74ccc2a40 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcba7fc640 1 -- 172.21.15.116:0/4012920448 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc4004450 con 0x55a74ccc2a40 2024-03-24T09:25:59.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcba7fc640 1 -- 172.21.15.116:0/4012920448 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4019800 con 0x55a74ccc2a40 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 -- 172.21.15.116:0/4012920448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 msgr2=0x55a74cccf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74cccf7c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009da0 tx=0x7fdcc4011810 comp rx=0 tx=0).stop 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 -- 172.21.15.116:0/4012920448 shutdown_connections 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74cccf7c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/4012920448 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccc2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 -- 172.21.15.116:0/4012920448 >> 172.21.15.116:0/4012920448 conn(0x55a74ccbcb20 msgr2=0x55a74ccbcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 -- 172.21.15.116:0/4012920448 shutdown_connections 2024-03-24T09:25:59.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.757+0000 7fdcdcffd740 1 -- 172.21.15.116:0/4012920448 wait complete. 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 -- start start 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 Processor -- start 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 0x55a74cccd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74cccfd00 0x55a74cccd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a74c8fc430 con 0x55a74ccc2a40 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a74c8b8e00 con 0x55a74cccfd00 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a74cbae590 con 0x55a74ccc2120 2024-03-24T09:25:59.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcdcffd740 10 client.0 _ll_register_callbacks cb 0x55a74cc1dfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 0x55a74cccd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 0x55a74cccd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56522/0 (socket says 172.21.15.116:56522) 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd88e4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74cccfd00 0x55a74cccd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 -- 172.21.15.116:0/19830948 learned_addr learned my addr 172.21.15.116:0/19830948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 -- 172.21.15.116:0/19830948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 msgr2=0x55a74ccccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 -- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74cccfd00 msgr2=0x55a74cccd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74cccfd00 0x55a74cccd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd37fe640 1 -- 172.21.15.116:0/19830948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc4009a00 con 0x55a74ccc2a40 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.758+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74cccfd00 0x55a74cccd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 0x55a74cccd270 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8097850 tx=0x7fdcc806a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9ce2012b01e414 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcb37fe640 1 -- 172.21.15.116:0/19830948 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80a3570 con 0x55a74ccc2a40 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a74c9287e0 con 0x55a74ccc2a40 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcb37fe640 1 -- 172.21.15.116:0/19830948 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdcc80b91c0 con 0x55a74ccc2a40 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcb37fe640 1 -- 172.21.15.116:0/19830948 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdcc80974f0 con 0x55a74ccc2a40 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 10 client.16815 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a74cc27bd0 con 0x55a74ccc2a40 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcb37fe640 1 -- 172.21.15.116:0/19830948 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a74c9287e0 con 0x55a74ccc2a40 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcb37fe640 1 -- 172.21.15.116:0/19830948 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fdcc809b0e0 con 0x55a74ccc2a40 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 10 client.16815 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 10 client.16815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 20 client.16815 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74ccdfef0 0x55a74cce2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 4 client.16815 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.759+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a74c965720 con 0x55a74ccdfef0 2024-03-24T09:25:59.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.760+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74ccdfef0 0x55a74cce2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.760+0000 7fdcb37fe640 10 client.16815 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.760+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74ccdfef0 0x55a74cce2310 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=878a3a91e9effac7 server_cookie=cacb38901cb36a21 in_seq=0 out_seq=0 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcb37fe640 1 -- 172.21.15.116:0/19830948 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a74c965720 con 0x55a74ccdfef0 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcb37fe640 10 client.16815 handle_command_reply: tid=0 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 client.16815 shutdown 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 msgr2=0x55a74cccd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 0x55a74cccd270 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8097850 tx=0x7fdcc806a0e0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 shutdown_connections 2024-03-24T09:25:59.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74ccdfef0 0x55a74cce2310 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74cccfd00 0x55a74cccd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2a40 0x55a74cccd270 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcb37fe640 0 client.16815 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/19830948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 >> 172.21.15.116:0/19830948 conn(0x55a74ccbcb20 msgr2=0x55a74ccbe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 shutdown_connections 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 1 -- 172.21.15.116:0/19830948 wait complete. 2024-03-24T09:25:59.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.761+0000 7fdcdcffd740 20 client.16815 trim_cache size 0 max 16384 2024-03-24T09:25:59.776 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:25:59 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3453556284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:25:59.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 Processor -- start 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 -- start start 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 0x55a74cccf460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccbbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74cccfd00 0x55a74cccd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a74ccdc890 con 0x55a74ccc2120 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a74ccb83e0 con 0x55a74ccc2a40 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a74ccc8930 con 0x55a74cccfd00 2024-03-24T09:25:59.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 0x55a74cccf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 0x55a74cccf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56528/0 (socket says 172.21.15.116:56528) 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.774+0000 7fdcd3fff640 1 -- 172.21.15.116:0/2881412616 learned_addr learned my addr 172.21.15.116:0/2881412616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74cccfd00 0x55a74cccd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccbbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd3fff640 1 -- 172.21.15.116:0/2881412616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74cccfd00 msgr2=0x55a74cccd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74cccfd00 0x55a74cccd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd3fff640 1 -- 172.21.15.116:0/2881412616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 msgr2=0x55a74ccbbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccbbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd3fff640 1 -- 172.21.15.116:0/2881412616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a74ccbc3c0 con 0x55a74ccc2120 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 0x55a74cccf460 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x55a74ccc3300 tx=0x7fdcc4009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710f92bfa69a2612 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcb9ffb640 1 -- 172.21.15.116:0/2881412616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4005240 con 0x55a74ccc2120 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcb9ffb640 1 -- 172.21.15.116:0/2881412616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc40042c0 con 0x55a74ccc2120 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.775+0000 7fdcb9ffb640 1 -- 172.21.15.116:0/2881412616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4003850 con 0x55a74ccc2120 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- 172.21.15.116:0/2881412616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 msgr2=0x55a74cccf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 0x55a74cccf460 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x55a74ccc3300 tx=0x7fdcc4009930 comp rx=0 tx=0).stop 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- 172.21.15.116:0/2881412616 shutdown_connections 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74cccfd00 0x55a74cccd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccbbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/2881412616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74ccc2120 0x55a74cccf460 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- 172.21.15.116:0/2881412616 >> 172.21.15.116:0/2881412616 conn(0x55a74ccbcb20 msgr2=0x55a74ccbdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- 172.21.15.116:0/2881412616 shutdown_connections 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- 172.21.15.116:0/2881412616 wait complete. 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- start start 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 Processor -- start 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccd3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccd3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a74cd45330 con 0x55a74cccfd00 2024-03-24T09:25:59.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a74cd454a0 con 0x55a74ccc2a40 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a74cd45660 con 0x55a74ccc2120 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcdcffd740 10 client.0 _ll_register_callbacks cb 0x7fdcc800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.776+0000 7fdcd88e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd41d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56536/0 (socket says 172.21.15.116:56536) 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccd3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 -- 172.21.15.116:0/3866063914 learned_addr learned my addr 172.21.15.116:0/3866063914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd37fe640 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccd3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 -- 172.21.15.116:0/3866063914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 msgr2=0x55a74ccd3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccd3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 -- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 msgr2=0x55a74ccd3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccd3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 -- 172.21.15.116:0/3866063914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc401e040 con 0x55a74cccfd00 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcd88e4640 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd41d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009ca0 tx=0x7fdccc02c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928eac4bdf6b13d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcb37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcb37fe640 1 -- 172.21.15.116:0/3866063914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdccc032e80 con 0x55a74cccfd00 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcb37fe640 1 -- 172.21.15.116:0/3866063914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdccc003e20 con 0x55a74cccfd00 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcc4004450 con 0x55a74cccfd00 2024-03-24T09:25:59.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcb37fe640 1 -- 172.21.15.116:0/3866063914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdccc004250 con 0x55a74cccfd00 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcdcffd740 10 client.16821 fetch_fsmap learned FSMap version 32 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a74cbc3c80 con 0x55a74cccfd00 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcb37fe640 1 -- 172.21.15.116:0/3866063914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc4004450 con 0x55a74cccfd00 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.777+0000 7fdcb37fe640 1 -- 172.21.15.116:0/3866063914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fdccc04b040 con 0x55a74cccfd00 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 10 client.16821 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 10 client.16821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 20 client.16821 populate_metadata read hostname 'smithi116' 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74cd46fd0 0x55a74cd493f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 4 client.16821 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:25:59.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a74c9b4890 con 0x55a74cd46fd0 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74cd46fd0 0x55a74cd493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcb37fe640 10 client.16821 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74cd46fd0 0x55a74cd493f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d871519e0d88e02d server_cookie=dc057d1ae89e6cfb in_seq=0 out_seq=0 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcb37fe640 1 -- 172.21.15.116:0/3866063914 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a74c9b4890 con 0x55a74cd46fd0 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcb37fe640 10 client.16821 handle_command_reply: tid=0 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 client.16821 shutdown 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 msgr2=0x55a74ccd41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd41d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fdccc009ca0 tx=0x7fdccc02c180 comp rx=0 tx=0).stop 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 shutdown_connections 2024-03-24T09:25:59.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a74cccfd00 0x55a74ccd41d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a74cd46fd0 0x55a74cd493f0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a74ccc2a40 0x55a74ccd3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 --2- 172.21.15.116:0/3866063914 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a74ccc2120 0x55a74ccd3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcb37fe640 0 client.16821 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.778+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 >> 172.21.15.116:0/3866063914 conn(0x55a74ccbcb20 msgr2=0x55a74ccbe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 shutdown_connections 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcdcffd740 1 -- 172.21.15.116:0/3866063914 wait complete. 2024-03-24T09:25:59.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcdcffd740 20 client.16821 trim_cache size 0 max 16384 2024-03-24T09:25:59.791 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:25:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcbc0a2230 msgr2=0x7fdcbc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcbc0a2230 0x7fdcbc0a46f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdccc004af0 tx=0x7fdccc02e040 comp rx=0 tx=0).stop 2024-03-24T09:25:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 msgr2=0x7fdcd419ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:25:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd419ddb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800e9f0 tx=0x7fdcc800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 shutdown_connections 2024-03-24T09:25:59.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcbc0a2230 0x7fdcbc0a46f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41066a0 0x7fdcd4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016e0 0x7fdcd419ddb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 --2- 172.21.15.116:0/3006194336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4100d20 0x7fdcd419d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:25:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 >> 172.21.15.116:0/3006194336 conn(0x7fdcd4076bd0 msgr2=0x7fdcd40ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:25:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 shutdown_connections 2024-03-24T09:25:59.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:25:59.779+0000 7fdcda347640 1 -- 172.21.15.116:0/3006194336 wait complete. 2024-03-24T09:25:59.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:25:59 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3453556284' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:26:00.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:00 smithi143 ceph-mon[35965]: pgmap v120: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:00.794 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:26:00.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:26:00.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:26:00.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:00 smithi117 ceph-mon[35788]: pgmap v120: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:00.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:00 smithi116 ceph-mon[32207]: pgmap v120: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:00.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 -- 172.21.15.116:0/533210400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0068910 msgr2=0x7f08d0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/533210400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0068910 0x7f08d0068d90 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009930 tx=0x7f08c402bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:00.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 -- 172.21.15.116:0/533210400 shutdown_connections 2024-03-24T09:26:00.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/533210400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08d00692d0 0x7f08d010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/533210400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0068910 0x7f08d0068d90 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/533210400 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d0107760 0x7f08d0107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.901+0000 7f08d6aa4640 1 -- 172.21.15.116:0/533210400 >> 172.21.15.116:0/533210400 conn(0x7f08d0077710 msgr2=0x7f08d0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 -- 172.21.15.116:0/533210400 shutdown_connections 2024-03-24T09:26:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 -- 172.21.15.116:0/533210400 wait complete. 2024-03-24T09:26:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 Processor -- start 2024-03-24T09:26:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 -- start start 2024-03-24T09:26:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d0068910 0x7f08d019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08d00692d0 0x7f08d019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 0x7f08d0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08d01980b0 con 0x7f08d0107760 2024-03-24T09:26:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f08d0198220 con 0x7f08d00692d0 2024-03-24T09:26:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d6aa4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f08d01983e0 con 0x7f08d0068910 2024-03-24T09:26:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d5041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 0x7f08d0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d5041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 0x7f08d0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56556/0 (socket says 172.21.15.116:56556) 2024-03-24T09:26:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.902+0000 7f08d5041640 1 -- 172.21.15.116:0/2489205148 learned_addr learned my addr 172.21.15.116:0/2489205148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08cffff640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08d00692d0 0x7f08d019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d4840640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d0068910 0x7f08d019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d5041640 1 -- 172.21.15.116:0/2489205148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d0068910 msgr2=0x7f08d019d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d5041640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d0068910 0x7f08d019d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d5041640 1 -- 172.21.15.116:0/2489205148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08d00692d0 msgr2=0x7f08d019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d5041640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08d00692d0 0x7f08d019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d5041640 1 -- 172.21.15.116:0/2489205148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c4009560 con 0x7f08d0107760 2024-03-24T09:26:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d5041640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 0x7f08d0197a60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f08c800eb10 tx=0x7f08c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08cdffb640 1 -- 172.21.15.116:0/2489205148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8004b20 con 0x7f08d0107760 2024-03-24T09:26:00.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08cdffb640 1 -- 172.21.15.116:0/2489205148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c8004c80 con 0x7f08d0107760 2024-03-24T09:26:00.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d01986c0 con 0x7f08d0107760 2024-03-24T09:26:00.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08cdffb640 1 -- 172.21.15.116:0/2489205148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c8010690 con 0x7f08d0107760 2024-03-24T09:26:00.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.903+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d01a4950 con 0x7f08d0107760 2024-03-24T09:26:00.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.905+0000 7f08cdffb640 1 -- 172.21.15.116:0/2489205148 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f08c8005020 con 0x7f08d0107760 2024-03-24T09:26:00.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08cdffb640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08b80a22b0 0x7f08b80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 Processor -- start 2024-03-24T09:26:00.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 -- start start 2024-03-24T09:26:00.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08b80a22b0 0x7f08b80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08cdffb640 1 -- 172.21.15.116:0/2489205148 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f08c8014070 con 0x7f08d0107760 2024-03-24T09:26:00.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93550b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93559720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93559c60 0x560d9355c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d934be530 con 0x560d93550760 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560d934bba60 con 0x560d93551080 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d9765740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560d934b7240 con 0x560d93559c60 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93550b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93550b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56564/0 (socket says 172.21.15.116:56564) 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 -- 172.21.15.116:0/2739100030 learned_addr learned my addr 172.21.15.116:0/2739100030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08cffff640 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93559720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d5041640 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93559c60 0x560d9355c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 -- 172.21.15.116:0/2739100030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93559c60 msgr2=0x560d9355c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93559c60 0x560d9355c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 -- 172.21.15.116:0/2739100030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 msgr2=0x560d93559720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93559720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 -- 172.21.15.116:0/2739100030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d934b58c0 con 0x560d93550760 2024-03-24T09:26:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08cffff640 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93559720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.906+0000 7f08d4840640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08b80a22b0 0x7f08b80a4770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f08c00139d0 tx=0x7f08c0009b60 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d4840640 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93550b40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f08c000f790 tx=0x7f08c001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d5ff9e79ac68a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08b6ffd640 1 -- 172.21.15.116:0/2739100030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0013170 con 0x560d93550760 2024-03-24T09:26:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08b6ffd640 1 -- 172.21.15.116:0/2739100030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c00044e0 con 0x560d93550760 2024-03-24T09:26:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08b6ffd640 1 -- 172.21.15.116:0/2739100030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c000ee70 con 0x560d93550760 2024-03-24T09:26:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 -- 172.21.15.116:0/2739100030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 msgr2=0x560d93550b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93550b40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f08c000f790 tx=0x7f08c001d000 comp rx=0 tx=0).stop 2024-03-24T09:26:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 -- 172.21.15.116:0/2739100030 shutdown_connections 2024-03-24T09:26:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93559c60 0x560d9355c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93559720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 --2- 172.21.15.116:0/2739100030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93550b40 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 -- 172.21.15.116:0/2739100030 >> 172.21.15.116:0/2739100030 conn(0x560d9354ae90 msgr2=0x560d9354b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 -- 172.21.15.116:0/2739100030 shutdown_connections 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.907+0000 7f08d9765740 1 -- 172.21.15.116:0/2739100030 wait complete. 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 -- start start 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 Processor -- start 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93550760 0x560d93556d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93557240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93557780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d9345bf80 con 0x560d93559c60 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560d9318a4b0 con 0x560d93551080 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560d9343cad0 con 0x560d93550760 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d9765740 10 client.0 _ll_register_callbacks cb 0x560d9346fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93557780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93557780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56572/0 (socket says 172.21.15.116:56572) 2024-03-24T09:26:00.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93557240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d4840640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93550760 0x560d93556d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 -- 172.21.15.116:0/293178671 learned_addr learned my addr 172.21.15.116:0/293178671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 -- 172.21.15.116:0/293178671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93550760 msgr2=0x560d93556d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93550760 0x560d93556d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 -- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 msgr2=0x560d93557240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.908+0000 7f08d5041640 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93557240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d5041640 1 -- 172.21.15.116:0/293178671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c001b040 con 0x560d93559c60 2024-03-24T09:26:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d5041640 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93557780 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f08c80cc370 tx=0x7f08c8099880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30a7cef7ef96029 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08affff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08affff640 1 -- 172.21.15.116:0/293178671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c809b710 con 0x560d93559c60 2024-03-24T09:26:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08affff640 1 -- 172.21.15.116:0/293178671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f08c809b010 con 0x560d93559c60 2024-03-24T09:26:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560d934db7a0 con 0x560d93559c60 2024-03-24T09:26:00.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08affff640 1 -- 172.21.15.116:0/293178671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08c809f160 con 0x560d93559c60 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 10 client.16833 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d9346d6d0 con 0x560d93559c60 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08affff640 1 -- 172.21.15.116:0/293178671 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560d934db7a0 con 0x560d93559c60 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08affff640 1 -- 172.21.15.116:0/293178671 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f08c801f060 con 0x560d93559c60 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 10 client.16833 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 10 client.16833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:00.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 20 client.16833 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d93569de0 0x560d9356c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 4 client.16833 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.909+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d934ad1f0 con 0x560d93569de0 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.910+0000 7f08d4840640 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d93569de0 0x560d9356c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.910+0000 7f08affff640 10 client.16833 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.910+0000 7f08d4840640 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d93569de0 0x560d9356c200 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eee7ebcbbcc1d7c server_cookie=48d866ec4b39d589 in_seq=0 out_seq=0 2024-03-24T09:26:00.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08affff640 1 -- 172.21.15.116:0/293178671 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d934ad1f0 con 0x560d93569de0 2024-03-24T09:26:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08affff640 10 client.16833 handle_command_reply: tid=0 2024-03-24T09:26:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 client.16833 shutdown 2024-03-24T09:26:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 msgr2=0x560d93557780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93557780 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f08c80cc370 tx=0x7f08c8099880 comp rx=0 tx=0).stop 2024-03-24T09:26:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 shutdown_connections 2024-03-24T09:26:00.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93557780 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d93569de0 0x560d9356c200 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93551080 0x560d93557240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 --2- 172.21.15.116:0/293178671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93550760 0x560d93556d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08affff640 0 client.16833 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 >> 172.21.15.116:0/293178671 conn(0x560d9354ae90 msgr2=0x560d9354cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 shutdown_connections 2024-03-24T09:26:00.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 1 -- 172.21.15.116:0/293178671 wait complete. 2024-03-24T09:26:00.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.911+0000 7f08d9765740 20 client.16833 trim_cache size 0 max 16384 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 Processor -- start 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 -- start start 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93550760 0x560d93559430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d935575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93571ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d9356c900 con 0x560d93559c60 2024-03-24T09:26:00.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560d93550160 con 0x560d93550760 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d9765740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560d93566770 con 0x560d93551080 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d5041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93571ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d5041640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93571ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56580/0 (socket says 172.21.15.116:56580) 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d5041640 1 -- 172.21.15.116:0/1183922297 learned_addr learned my addr 172.21.15.116:0/1183922297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.928+0000 7f08d4840640 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93550760 0x560d93559430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08cffff640 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d935575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d5041640 1 -- 172.21.15.116:0/1183922297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 msgr2=0x560d935575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d5041640 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d935575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d5041640 1 -- 172.21.15.116:0/1183922297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93550760 msgr2=0x560d93559430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d5041640 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93550760 0x560d93559430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d5041640 1 -- 172.21.15.116:0/1183922297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d93451c60 con 0x560d93559c60 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08cffff640 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d935575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d5041640 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93571ad0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f08c80979e0 tx=0x7f08c8099520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644cd2884df22270 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:00.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08b67fc640 1 -- 172.21.15.116:0/1183922297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c80a2270 con 0x560d93559c60 2024-03-24T09:26:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08b67fc640 1 -- 172.21.15.116:0/1183922297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c809ade0 con 0x560d93559c60 2024-03-24T09:26:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08b67fc640 1 -- 172.21.15.116:0/1183922297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c80a3560 con 0x560d93559c60 2024-03-24T09:26:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d9765740 1 -- 172.21.15.116:0/1183922297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 msgr2=0x560d93571ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.929+0000 7f08d9765740 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93571ad0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f08c80979e0 tx=0x7f08c8099520 comp rx=0 tx=0).stop 2024-03-24T09:26:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- 172.21.15.116:0/1183922297 shutdown_connections 2024-03-24T09:26:00.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93559c60 0x560d93571ad0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d935575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 --2- 172.21.15.116:0/1183922297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93550760 0x560d93559430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- 172.21.15.116:0/1183922297 >> 172.21.15.116:0/1183922297 conn(0x560d9354ae90 msgr2=0x560d93549660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- 172.21.15.116:0/1183922297 shutdown_connections 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- 172.21.15.116:0/1183922297 wait complete. 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- start start 2024-03-24T09:26:00.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 Processor -- start 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93561c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d9355e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93559c60 0x560d9355e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560d9355eba0 con 0x560d93550760 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560d9355ed10 con 0x560d93559c60 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560d9355eed0 con 0x560d93551080 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d9765740 10 client.0 _ll_register_callbacks cb 0x7f08c809a970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:00.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d4840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93561c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d4840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93561c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56586/0 (socket says 172.21.15.116:56586) 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d4840640 1 -- 172.21.15.116:0/2410583903 learned_addr learned my addr 172.21.15.116:0/2410583903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08cffff640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d9355e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.930+0000 7f08d5041640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93559c60 0x560d9355e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d4840640 1 -- 172.21.15.116:0/2410583903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 msgr2=0x560d9355e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d4840640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d9355e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d4840640 1 -- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93559c60 msgr2=0x560d9355e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d4840640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93559c60 0x560d9355e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d4840640 1 -- 172.21.15.116:0/2410583903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d9346d6d0 con 0x560d93550760 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d5041640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93559c60 0x560d9355e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d4840640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93561c80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f08c0015010 tx=0x7f08c0009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec9dadc915b7dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08affff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08affff640 1 -- 172.21.15.116:0/2410583903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0031070 con 0x560d93550760 2024-03-24T09:26:00.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08affff640 1 -- 172.21.15.116:0/2410583903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f08c000e780 con 0x560d93550760 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c00044e0 con 0x560d93550760 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08affff640 1 -- 172.21.15.116:0/2410583903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08c0009850 con 0x560d93550760 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 10 client.16839 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d93477970 con 0x560d93550760 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08affff640 1 -- 172.21.15.116:0/2410583903 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f08c00044e0 con 0x560d93550760 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08affff640 1 -- 172.21.15.116:0/2410583903 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f08c0005040 con 0x560d93550760 2024-03-24T09:26:00.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 10 client.16839 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 20 client.16839 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d935d1830 0x560d935d3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 4 client.16839 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.931+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560d931b6860 con 0x560d935d1830 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.932+0000 7f08cffff640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d935d1830 0x560d935d3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.932+0000 7f08affff640 10 client.16839 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:00.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.932+0000 7f08cffff640 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d935d1830 0x560d935d3c50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3added5d949e76db server_cookie=6fc3fcaf18f65279 in_seq=0 out_seq=0 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08affff640 1 -- 172.21.15.116:0/2410583903 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560d931b6860 con 0x560d935d1830 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08affff640 10 client.16839 handle_command_reply: tid=0 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 client.16839 shutdown 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 msgr2=0x560d93561c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93561c80 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f08c0015010 tx=0x7f08c0009620 comp rx=0 tx=0).stop 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 shutdown_connections 2024-03-24T09:26:00.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560d935d1830 0x560d935d3c50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d93559c60 0x560d9355e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560d93551080 0x560d9355e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 --2- 172.21.15.116:0/2410583903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560d93550760 0x560d93561c80 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 >> 172.21.15.116:0/2410583903 conn(0x560d9354ae90 msgr2=0x560d93549ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08affff640 0 client.16839 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 shutdown_connections 2024-03-24T09:26:00.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 1 -- 172.21.15.116:0/2410583903 wait complete. 2024-03-24T09:26:00.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.933+0000 7f08d9765740 20 client.16839 trim_cache size 0 max 16384 2024-03-24T09:26:00.951 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:26:00.951 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:26:00.951 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "95e1ca74-f150-4429-9498-2dce9c980442", 2024-03-24T09:26:00.951 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:26:00.952 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:26:00.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08b80a22b0 msgr2=0x7f08b80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08b80a22b0 0x7f08b80a4770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f08c00139d0 tx=0x7f08c0009b60 comp rx=0 tx=0).stop 2024-03-24T09:26:00.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 msgr2=0x7f08d0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 0x7f08d0197a60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f08c800eb10 tx=0x7f08c800c2d0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 shutdown_connections 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08d0107760 0x7f08d0197a60 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08d00692d0 0x7f08d019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08b80a22b0 0x7f08b80a4770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 --2- 172.21.15.116:0/2489205148 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f08d0068910 0x7f08d019d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:00.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 >> 172.21.15.116:0/2489205148 conn(0x7f08d0077710 msgr2=0x7f08d00fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:00.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 shutdown_connections 2024-03-24T09:26:00.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:00.934+0000 7f08d6aa4640 1 -- 172.21.15.116:0/2489205148 wait complete. 2024-03-24T09:26:00.955 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:26:00.955 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:26:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3050576256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 msgr2=0x7f4cf81059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3050576256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf81059b0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009930 tx=0x7f4ce802bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3050576256 shutdown_connections 2024-03-24T09:26:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3050576256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cf8100f70 0x7f4cf810d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3050576256 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81005b0 0x7f4cf8100a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3050576256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf81059b0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3050576256 >> 172.21.15.116:0/3050576256 conn(0x7f4cf80776a0 msgr2=0x7f4cf8077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:01.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3050576256 shutdown_connections 2024-03-24T09:26:01.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.120+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3050576256 wait complete. 2024-03-24T09:26:01.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 Processor -- start 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 -- start start 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81005b0 0x7f4cf819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cf8100f70 0x7f4cf819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf8196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8196fa0 con 0x7f4cf81055d0 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf8197110 con 0x7f4cf8100f70 2024-03-24T09:26:01.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cfdc9d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf81972d0 con 0x7f4cf81005b0 2024-03-24T09:26:01.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf8196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf8196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56606/0 (socket says 172.21.15.116:56606) 2024-03-24T09:26:01.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 -- 172.21.15.116:0/3234455924 learned_addr learned my addr 172.21.15.116:0/3234455924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:01.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4ceffff640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cf8100f70 0x7f4cf819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 -- 172.21.15.116:0/3234455924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81005b0 msgr2=0x7f4cf819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81005b0 0x7f4cf819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 -- 172.21.15.116:0/3234455924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cf8100f70 msgr2=0x7f4cf819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cf8100f70 0x7f4cf819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.121+0000 7f4cf7fff640 1 -- 172.21.15.116:0/3234455924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce8009560 con 0x7f4cf81055d0 2024-03-24T09:26:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.122+0000 7f4cf7fff640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf8196950 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000eb10 tx=0x7f4cf000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.122+0000 7f4cf57fa640 1 -- 172.21.15.116:0/3234455924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0004b20 con 0x7f4cf81055d0 2024-03-24T09:26:01.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.122+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf81975b0 con 0x7f4cf81055d0 2024-03-24T09:26:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.122+0000 7f4cf57fa640 1 -- 172.21.15.116:0/3234455924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4cf0004c80 con 0x7f4cf81055d0 2024-03-24T09:26:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.122+0000 7f4cf57fa640 1 -- 172.21.15.116:0/3234455924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0010690 con 0x7f4cf81055d0 2024-03-24T09:26:01.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.122+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf81a37f0 con 0x7f4cf81055d0 2024-03-24T09:26:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4cf57fa640 1 -- 172.21.15.116:0/3234455924 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f4cf0005020 con 0x7f4cf81055d0 2024-03-24T09:26:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 Processor -- start 2024-03-24T09:26:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 -- start start 2024-03-24T09:26:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 0x5593757d52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d5820 0x5593757e03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757e08e0 0x5593757e2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5593756f9f30 con 0x5593757d4f00 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559375743810 con 0x5593757e08e0 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4d00882740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559375740d40 con 0x5593757d5820 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 0x5593757d52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 0x5593757d52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56614/0 (socket says 172.21.15.116:56614) 2024-03-24T09:26:01.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.125+0000 7f4ceffff640 1 -- 172.21.15.116:0/3012033613 learned_addr learned my addr 172.21.15.116:0/3012033613 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cf57fa640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4ce40a2230 0x7f4ce40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757e08e0 0x5593757e2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d5820 0x5593757e03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 -- 172.21.15.116:0/3012033613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d5820 msgr2=0x5593757e03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cf57fa640 1 -- 172.21.15.116:0/3234455924 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4cf0014070 con 0x7f4cf81055d0 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d5820 0x5593757e03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cf7fff640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4ce40a2230 0x7f4ce40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 -- 172.21.15.116:0/3012033613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757e08e0 msgr2=0x5593757e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757e08e0 0x5593757e2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 -- 172.21.15.116:0/3012033613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593756fdea0 con 0x5593757d4f00 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4ceffff640 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 0x5593757d52e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00058a0 tx=0x7f4ce000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d0cdedefbc4e8e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cf7fff640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4ce40a2230 0x7f4ce40a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00049d0 tx=0x7f4cf001bb30 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cedffb640 1 -- 172.21.15.116:0/3012033613 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0012da0 con 0x5593757d4f00 2024-03-24T09:26:01.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cedffb640 1 -- 172.21.15.116:0/3012033613 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ce00044e0 con 0x5593757d4f00 2024-03-24T09:26:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.126+0000 7f4cedffb640 1 -- 172.21.15.116:0/3012033613 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000ed00 con 0x5593757d4f00 2024-03-24T09:26:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 -- 172.21.15.116:0/3012033613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 msgr2=0x5593757d52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 0x5593757d52e0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4ce00058a0 tx=0x7f4ce000b390 comp rx=0 tx=0).stop 2024-03-24T09:26:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 -- 172.21.15.116:0/3012033613 shutdown_connections 2024-03-24T09:26:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757e08e0 0x5593757e2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d5820 0x5593757e03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 --2- 172.21.15.116:0/3012033613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d4f00 0x5593757d52e0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 -- 172.21.15.116:0/3012033613 >> 172.21.15.116:0/3012033613 conn(0x5593757cf900 msgr2=0x5593757cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 -- 172.21.15.116:0/3012033613 shutdown_connections 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.127+0000 7f4d00882740 1 -- 172.21.15.116:0/3012033613 wait complete. 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.128+0000 7f4d00882740 1 -- start start 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.128+0000 7f4d00882740 1 Processor -- start 2024-03-24T09:26:01.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.128+0000 7f4d00882740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757dba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.128+0000 7f4d00882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 0x5593757dbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.128+0000 7f4d00882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757e08e0 0x5593757dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4d00882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55937573c520 con 0x5593757e08e0 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4d00882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55937540f430 con 0x5593757d5820 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4d00882740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5593753cbe00 con 0x5593757d4f00 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 0x5593757dbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757dba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 0x5593757dbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:45198/0 (socket says 172.21.15.116:45198) 2024-03-24T09:26:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 -- 172.21.15.116:0/4293026544 learned_addr learned my addr 172.21.15.116:0/4293026544 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4ceffff640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757e08e0 0x5593757dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4d00882740 10 client.0 _ll_register_callbacks cb 0x5593756dad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 -- 172.21.15.116:0/4293026544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 msgr2=0x5593757dba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757dba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 -- 172.21.15.116:0/4293026544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757e08e0 msgr2=0x5593757dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757e08e0 0x5593757dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4cf77fe640 1 -- 172.21.15.116:0/4293026544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce001a040 con 0x5593757d5820 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.129+0000 7f4ceffff640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757e08e0 0x5593757dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 0x5593757dbfc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80683c0 tx=0x7f4ce802c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6372fb3b15cd7e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4cd6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:26:01.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/4293026544 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8004240 con 0x5593757d5820 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/4293026544 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4ce802de40 con 0x5593757d5820 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559375478720 con 0x5593757d5820 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/4293026544 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ce802c5d0 con 0x5593757d5820 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4d00882740 10 client.26678 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.130+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55937573aba0 con 0x5593757d5820 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/4293026544 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559375478720 con 0x5593757d5820 2024-03-24T09:26:01.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/4293026544 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f4ce803c040 con 0x5593757d5820 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4d00882740 10 client.26678 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4d00882740 10 client.26678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4d00882740 20 client.26678 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4d00882740 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757eebc0 0x5593757f1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4d00882740 4 client.26678 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593754c7890 con 0x5593757eebc0 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4ceffff640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757eebc0 0x5593757f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4ceffff640 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757eebc0 0x5593757f1060 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e2bb3e2b5b0a039 server_cookie=323e5319ea8b0a5a in_seq=0 out_seq=0 2024-03-24T09:26:01.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.131+0000 7f4cd6ffd640 10 client.26678 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:01.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.132+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/4293026544 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5593754c7890 con 0x5593757eebc0 2024-03-24T09:26:01.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.132+0000 7f4cd6ffd640 10 client.26678 handle_command_reply: tid=0 2024-03-24T09:26:01.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 client.26678 shutdown 2024-03-24T09:26:01.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 msgr2=0x5593757dbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 0x5593757dbfc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80683c0 tx=0x7f4ce802c980 comp rx=0 tx=0).stop 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 shutdown_connections 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757e08e0 0x5593757dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757eebc0 0x5593757f1060 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d5820 0x5593757dbfc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4cd6ffd640 0 client.26678 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 --2- 172.21.15.116:0/4293026544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757dba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 >> 172.21.15.116:0/4293026544 conn(0x5593757cf900 msgr2=0x5593757d1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:01.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 shutdown_connections 2024-03-24T09:26:01.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.133+0000 7f4d00882740 1 -- 172.21.15.116:0/4293026544 wait complete. 2024-03-24T09:26:01.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.134+0000 7f4d00882740 20 client.26678 trim_cache size 0 max 16384 2024-03-24T09:26:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.153+0000 7f4d00882740 1 Processor -- start 2024-03-24T09:26:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 -- start start 2024-03-24T09:26:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757cf900 0x5593757dc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757f6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757d8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5593756c1430 con 0x5593757d5820 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5593757ed310 con 0x5593757cf900 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4d00882740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5593757c14f0 con 0x5593757d4f00 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757d8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757d8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56644/0 (socket says 172.21.15.116:56644) 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 -- 172.21.15.116:0/1512116310 learned_addr learned my addr 172.21.15.116:0/1512116310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:01.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757cf900 0x5593757dc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 -- 172.21.15.116:0/1512116310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 msgr2=0x5593757f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 -- 172.21.15.116:0/1512116310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757cf900 msgr2=0x5593757dc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757cf900 0x5593757dc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 -- 172.21.15.116:0/1512116310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593756d6b20 con 0x5593757d5820 2024-03-24T09:26:01.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.154+0000 7f4ceffff640 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757d8d60 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e960 tx=0x7f4ce0014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b0f5a92adfe56e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.155+0000 7f4ced7fa640 1 -- 172.21.15.116:0/1512116310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0031450 con 0x5593757d5820 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.155+0000 7f4ced7fa640 1 -- 172.21.15.116:0/1512116310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4ce0038070 con 0x5593757d5820 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.155+0000 7f4ced7fa640 1 -- 172.21.15.116:0/1512116310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000e990 con 0x5593757d5820 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 -- 172.21.15.116:0/1512116310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 msgr2=0x5593757d8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757d8d60 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e960 tx=0x7f4ce0014040 comp rx=0 tx=0).stop 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 -- 172.21.15.116:0/1512116310 shutdown_connections 2024-03-24T09:26:01.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757d8d60 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757d4f00 0x5593757f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- 172.21.15.116:0/1512116310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757cf900 0x5593757dc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 -- 172.21.15.116:0/1512116310 >> 172.21.15.116:0/1512116310 conn(0x5593757da720 msgr2=0x5593757ceb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 -- 172.21.15.116:0/1512116310 shutdown_connections 2024-03-24T09:26:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 -- 172.21.15.116:0/1512116310 wait complete. 2024-03-24T09:26:01.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 -- start start 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 Processor -- start 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757cf900 0x5593757e5610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d4f00 0x5593757e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.156+0000 7f4d00882740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757e6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4d00882740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559375854030 con 0x5593757d5820 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4d00882740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5593758541a0 con 0x5593757d4f00 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4d00882740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559375854360 con 0x5593757cf900 2024-03-24T09:26:01.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4d00882740 10 client.0 _ll_register_callbacks cb 0x7f4cf00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757e6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757e6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56650/0 (socket says 172.21.15.116:56650) 2024-03-24T09:26:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4cf77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d4f00 0x5593757e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 -- 172.21.15.116:0/649090261 learned_addr learned my addr 172.21.15.116:0/649090261 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757cf900 0x5593757e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 -- 172.21.15.116:0/649090261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757cf900 msgr2=0x5593757e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757cf900 0x5593757e5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 -- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d4f00 msgr2=0x5593757e5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d4f00 0x5593757e5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 -- 172.21.15.116:0/649090261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce0027040 con 0x5593757d5820 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d4f00 0x5593757e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4ceffff640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757e6090 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0005870 tx=0x7f4ce000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550686f86af58b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4cd6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:01.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/649090261 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0031530 con 0x5593757d5820 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/649090261 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4ce0031690 con 0x5593757d5820 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.157+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55937543b7e0 con 0x5593757d5820 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/649090261 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4ce000e990 con 0x5593757d5820 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 10 client.16857 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5593756fc700 con 0x5593757d5820 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/649090261 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55937543b7e0 con 0x5593757d5820 2024-03-24T09:26:01.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/649090261 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f4ce0048070 con 0x5593757d5820 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 10 client.16857 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 20 client.16857 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757e08e0 0x559375857990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 4 client.16857 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5593753e7700 con 0x5593757e08e0 2024-03-24T09:26:01.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.158+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757e08e0 0x559375857990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4cf77fe640 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757e08e0 0x559375857990 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b26bd630c1ec1e5 server_cookie=a4ee6d262ec8b55a in_seq=0 out_seq=0 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4cd6ffd640 10 client.16857 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4cd6ffd640 1 -- 172.21.15.116:0/649090261 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5593753e7700 con 0x5593757e08e0 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4cd6ffd640 10 client.16857 handle_command_reply: tid=0 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 client.16857 shutdown 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 msgr2=0x5593757e6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757e6090 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0005870 tx=0x7f4ce000b390 comp rx=0 tx=0).stop 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 shutdown_connections 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5593757d5820 0x5593757e6090 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5593757e08e0 0x559375857990 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5593757d4f00 0x5593757e5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 --2- 172.21.15.116:0/649090261 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5593757cf900 0x5593757e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 >> 172.21.15.116:0/649090261 conn(0x5593757da720 msgr2=0x5593757d0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:01.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4cd6ffd640 0 client.16857 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 shutdown_connections 2024-03-24T09:26:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 1 -- 172.21.15.116:0/649090261 wait complete. 2024-03-24T09:26:01.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.159+0000 7f4d00882740 20 client.16857 trim_cache size 0 max 16384 2024-03-24T09:26:01.173 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:26:01.173 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "95e1ca74-f150-4429-9498-2dce9c980442": { 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "95e1ca74-f150-4429-9498-2dce9c980442", 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:26:01.174 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:26:01.175 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:26:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4ce40a2230 msgr2=0x7f4ce40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4ce40a2230 0x7f4ce40a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00049d0 tx=0x7f4cf001bb30 comp rx=0 tx=0).stop 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 msgr2=0x7f4cf8196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf8196950 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000eb10 tx=0x7f4cf000c2d0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 shutdown_connections 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cf81055d0 0x7f4cf8196950 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cf8100f70 0x7f4cf819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4ce40a2230 0x7f4ce40a46f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 --2- 172.21.15.116:0/3234455924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81005b0 0x7f4cf819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:01.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 >> 172.21.15.116:0/3234455924 conn(0x7f4cf80776a0 msgr2=0x7f4cf80fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 shutdown_connections 2024-03-24T09:26:01.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:01.160+0000 7f4cfdc9d640 1 -- 172.21.15.116:0/3234455924 wait complete. 2024-03-24T09:26:01.178 INFO:tasks.cephfs.filesystem:scrub status for tag:95e1ca74-f150-4429-9498-2dce9c980442 - {'path': '/', 'tag': '95e1ca74-f150-4429-9498-2dce9c980442', 'options': 'recursive,force'} 2024-03-24T09:26:01.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:26:01.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:01.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:01.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:01.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:01.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:01 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:26:01.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:01.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:01.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:01.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:01.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:01 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:26:01.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:01.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:01.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:01.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:01.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:01.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:01 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:02.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:02.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.720 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.721 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:02 smithi116 ceph-mon[32207]: pgmap v121: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:02.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:02 smithi143 ceph-mon[35965]: pgmap v121: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:02.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:02.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:02.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:02.817 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:02 smithi117 ceph-mon[35788]: pgmap v121: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:04.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:04 smithi143 ceph-mon[35965]: pgmap v122: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:04.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:04 smithi117 ceph-mon[35788]: pgmap v122: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:04.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:04 smithi116 ceph-mon[32207]: pgmap v122: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:06.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:06 smithi143 ceph-mon[35965]: pgmap v123: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:06.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:06 smithi117 ceph-mon[35788]: pgmap v123: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:06.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:06 smithi116 ceph-mon[32207]: pgmap v123: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:08.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:08 smithi143 ceph-mon[35965]: pgmap v124: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:08.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:08 smithi117 ceph-mon[35788]: pgmap v124: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:08.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:08 smithi116 ceph-mon[32207]: pgmap v124: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:10.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:10 smithi143 ceph-mon[35965]: pgmap v125: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:10.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:10 smithi117 ceph-mon[35788]: pgmap v125: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:10.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:10 smithi116 ceph-mon[32207]: pgmap v125: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:26:12.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:12 smithi143 ceph-mon[35965]: pgmap v126: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:12.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:12 smithi117 ceph-mon[35788]: pgmap v126: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:12.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:12 smithi116 ceph-mon[32207]: pgmap v126: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:26:13.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:13 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:13.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:13 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:13.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:13 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:13.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:13 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:13.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:13 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:13.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:13 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:13.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:13 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:13.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:13 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:13.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:13 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:14.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:14 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:14.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:14 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:14.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:14 smithi143 ceph-mon[35965]: pgmap v127: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:26:14.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:14 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:26:14.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:14 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:14.867 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:14 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:14.867 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:14 smithi117 ceph-mon[35788]: pgmap v127: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:26:14.867 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:14 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:26:14.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:14 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:14.867 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:14 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:14.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:14 smithi116 ceph-mon[32207]: pgmap v127: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:26:14.868 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:14 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:26:15.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:26:15.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:26:15.775 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:26:15.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:26:15.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:26:15.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:26:15.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:26:15.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:26:15.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:26:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:16 smithi143 ceph-mon[35965]: pgmap v128: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:26:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:16 smithi117 ceph-mon[35788]: pgmap v128: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:26:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:16 smithi116 ceph-mon[32207]: pgmap v128: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:26:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:18 smithi143 ceph-mon[35965]: pgmap v129: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:26:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:18 smithi117 ceph-mon[35788]: pgmap v129: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:26:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:18 smithi116 ceph-mon[32207]: pgmap v129: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:26:21.118 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:20 smithi117 ceph-mon[35788]: pgmap v130: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:26:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:20 smithi143 ceph-mon[35965]: pgmap v130: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:26:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:20 smithi116 ceph-mon[32207]: pgmap v130: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:26:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:22 smithi143 ceph-mon[35965]: pgmap v131: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:26:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:22 smithi117 ceph-mon[35788]: pgmap v131: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:26:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:22 smithi116 ceph-mon[32207]: pgmap v131: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:26:25.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:24 smithi143 ceph-mon[35965]: pgmap v132: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:24 smithi117 ceph-mon[35788]: pgmap v132: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:24 smithi116 ceph-mon[32207]: pgmap v132: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:26 smithi143 ceph-mon[35965]: pgmap v133: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:26 smithi117 ceph-mon[35788]: pgmap v133: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:26 smithi116 ceph-mon[32207]: pgmap v133: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:28 smithi143 ceph-mon[35965]: pgmap v134: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:28 smithi117 ceph-mon[35788]: pgmap v134: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:29.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:28 smithi116 ceph-mon[32207]: pgmap v134: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:31.179 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:26:31.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:26:31.207 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:30 smithi116 ceph-mon[32207]: pgmap v135: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:30 smithi143 ceph-mon[35965]: pgmap v135: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 -- 172.21.15.116:0/3404314704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c102980 msgr2=0x7fc62c102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 --2- 172.21.15.116:0/3404314704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c102980 0x7fc62c102d60 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc61c009900 tx=0x7fc61c02bcd0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 -- 172.21.15.116:0/3404314704 shutdown_connections 2024-03-24T09:26:31.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 --2- 172.21.15.116:0/3404314704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c10bd50 0x7fc62c10e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 --2- 172.21.15.116:0/3404314704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c1032a0 0x7fc62c10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 --2- 172.21.15.116:0/3404314704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c102980 0x7fc62c102d60 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.293+0000 7fc631bc9640 1 -- 172.21.15.116:0/3404314704 >> 172.21.15.116:0/3404314704 conn(0x7fc62c077770 msgr2=0x7fc62c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.294+0000 7fc631bc9640 1 -- 172.21.15.116:0/3404314704 shutdown_connections 2024-03-24T09:26:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.294+0000 7fc631bc9640 1 -- 172.21.15.116:0/3404314704 wait complete. 2024-03-24T09:26:31.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.294+0000 7fc631bc9640 1 Processor -- start 2024-03-24T09:26:31.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.294+0000 7fc631bc9640 1 -- start start 2024-03-24T09:26:31.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc631bc9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c102980 0x7fc62c19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc631bc9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c1032a0 0x7fc62c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc631bc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 0x7fc62c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc631bc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c198100 con 0x7fc62c10bd50 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc631bc9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c198270 con 0x7fc62c1032a0 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc631bc9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc62c198430 con 0x7fc62c102980 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 0x7fc62c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 0x7fc62c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54198/0 (socket says 172.21.15.116:54198) 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62bfff640 1 -- 172.21.15.116:0/2487809436 learned_addr learned my addr 172.21.15.116:0/2487809436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c1032a0 0x7fc62c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c102980 0x7fc62c19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 -- 172.21.15.116:0/2487809436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c1032a0 msgr2=0x7fc62c19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c1032a0 0x7fc62c19de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 -- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 msgr2=0x7fc62c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 0x7fc62c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 -- 172.21.15.116:0/2487809436 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc620009490 con 0x7fc62c102980 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62bfff640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 0x7fc62c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62affd640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c1032a0 0x7fc62c19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c102980 0x7fc62c19d900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc61c036040 tx=0x7fc61c005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.295+0000 7fc628ff9640 1 -- 172.21.15.116:0/2487809436 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c0043d0 con 0x7fc62c102980 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.296+0000 7fc628ff9640 1 -- 172.21.15.116:0/2487809436 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc61c003aa0 con 0x7fc62c102980 2024-03-24T09:26:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.296+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c009560 con 0x7fc62c102980 2024-03-24T09:26:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.296+0000 7fc628ff9640 1 -- 172.21.15.116:0/2487809436 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c0053c0 con 0x7fc62c102980 2024-03-24T09:26:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.296+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62c100500 con 0x7fc62c102980 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc628ff9640 1 -- 172.21.15.116:0/2487809436 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc61c005520 con 0x7fc62c102980 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 Processor -- start 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 -- start start 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f577cdf30 con 0x564f578a91d0 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f57817810 con 0x564f578b26d0 2024-03-24T09:26:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc6347af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f57814d40 con 0x564f578a9af0 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc62bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc62b7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc62bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578a95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54210/0 (socket says 172.21.15.116:54210) 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc62bfff640 1 -- 172.21.15.116:0/3454089833 learned_addr learned my addr 172.21.15.116:0/3454089833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.298+0000 7fc62affd640 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc628ff9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc6140a2230 0x7fc6140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 -- 172.21.15.116:0/3454089833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 msgr2=0x564f578b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 -- 172.21.15.116:0/3454089833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 msgr2=0x564f578b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 -- 172.21.15.116:0/3454089833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f577d1ea0 con 0x564f578a91d0 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62affd640 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc628ff9640 1 -- 172.21.15.116:0/2487809436 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc61c0f0a80 con 0x7fc62c102980 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc6140a2230 0x7fc6140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc62bfff640 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578a95b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc624011c70 tx=0x7fc62400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f997248a7d6c62b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc611ffb640 1 -- 172.21.15.116:0/3454089833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62402a440 con 0x564f578a91d0 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc611ffb640 1 -- 172.21.15.116:0/3454089833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc62402aa50 con 0x564f578a91d0 2024-03-24T09:26:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.299+0000 7fc611ffb640 1 -- 172.21.15.116:0/3454089833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc624017030 con 0x564f578a91d0 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 -- 172.21.15.116:0/3454089833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 msgr2=0x564f578a95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578a95b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fc624011c70 tx=0x7fc62400c6a0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc62bfff640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc6140a2230 0x7fc6140a46f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc624027000 tx=0x7fc624028000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 -- 172.21.15.116:0/3454089833 shutdown_connections 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 --2- 172.21.15.116:0/3454089833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578a95b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 -- 172.21.15.116:0/3454089833 >> 172.21.15.116:0/3454089833 conn(0x564f578a3900 msgr2=0x564f578a3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 -- 172.21.15.116:0/3454089833 shutdown_connections 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 -- 172.21.15.116:0/3454089833 wait complete. 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.300+0000 7fc6347af740 1 -- start start 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 Processor -- start 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578afc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f57810520 con 0x564f578a91d0 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f574e3430 con 0x564f578b26d0 2024-03-24T09:26:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f5749fe00 con 0x564f578a9af0 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc6347af740 10 client.0 _ll_register_callbacks cb 0x564f577aed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54220/0 (socket says 172.21.15.116:54220) 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 -- 172.21.15.116:0/4053885519 learned_addr learned my addr 172.21.15.116:0/4053885519 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62affd640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62bfff640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 -- 172.21.15.116:0/4053885519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 msgr2=0x564f578afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 -- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 msgr2=0x564f578b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62b7fe640 1 -- 172.21.15.116:0/4053885519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc624011860 con 0x564f578a91d0 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62bfff640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.301+0000 7fc62affd640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578af6c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0edf30 tx=0x7fc61c004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46978443f07c56aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc60affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc60affd640 1 -- 172.21.15.116:0/4053885519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c0bd870 con 0x564f578a91d0 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc60affd640 1 -- 172.21.15.116:0/4053885519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc61c0c5910 con 0x564f578a91d0 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f5750f7e0 con 0x564f578a91d0 2024-03-24T09:26:31.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc60affd640 1 -- 172.21.15.116:0/4053885519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc61c0c12c0 con 0x564f578a91d0 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 10 client.16869 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564f5780eba0 con 0x564f578a91d0 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc60affd640 1 -- 172.21.15.116:0/4053885519 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564f5750f7e0 con 0x564f578a91d0 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc60affd640 1 -- 172.21.15.116:0/4053885519 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc61c03c070 con 0x564f578a91d0 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 10 client.16869 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 10 client.16869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 20 client.16869 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f578c2990 0x564f578c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 4 client.16869 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f5754c720 con 0x564f578c2990 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.302+0000 7fc62affd640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f578c2990 0x564f578c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.303+0000 7fc60affd640 10 client.16869 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.303+0000 7fc62affd640 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f578c2990 0x564f578c4db0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e9515f54156717e server_cookie=cf4dbde6b4a93d42 in_seq=0 out_seq=0 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.303+0000 7fc60affd640 1 -- 172.21.15.116:0/4053885519 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f5754c720 con 0x564f578c2990 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.303+0000 7fc60affd640 10 client.16869 handle_command_reply: tid=0 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.303+0000 7fc6347af740 1 client.16869 shutdown 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 msgr2=0x564f578af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578af6c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fc61c0edf30 tx=0x7fc61c004dc0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 shutdown_connections 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f578c2990 0x564f578c4db0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578b26d0 0x564f578b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc60affd640 0 client.16869 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 --2- 172.21.15.116:0/4053885519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a91d0 0x564f578af6c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 >> 172.21.15.116:0/4053885519 conn(0x564f578a3900 msgr2=0x564f578a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 shutdown_connections 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 1 -- 172.21.15.116:0/4053885519 wait complete. 2024-03-24T09:26:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.304+0000 7fc6347af740 20 client.16869 trim_cache size 0 max 16384 2024-03-24T09:26:31.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:30 smithi117 ceph-mon[35788]: pgmap v135: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 Processor -- start 2024-03-24T09:26:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 -- start start 2024-03-24T09:26:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 0x564f578aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578b26d0 0x564f578ca680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f57795430 con 0x564f578a9af0 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f578a31d0 con 0x564f578a91d0 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6347af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f5754c720 con 0x564f578b26d0 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 0x564f578aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 0x564f578aff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54236/0 (socket says 172.21.15.116:54236) 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 -- 172.21.15.116:0/1032273784 learned_addr learned my addr 172.21.15.116:0/1032273784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62bfff640 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578b26d0 0x564f578ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 -- 172.21.15.116:0/1032273784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578b26d0 msgr2=0x564f578ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578b26d0 0x564f578ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 -- 172.21.15.116:0/1032273784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 msgr2=0x564f578b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 -- 172.21.15.116:0/1032273784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f577aab20 con 0x564f578a9af0 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc62affd640 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 0x564f578aff60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc620000c00 tx=0x7fc620005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c731752363406393 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.322+0000 7fc6117fa640 1 -- 172.21.15.116:0/1032273784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620026030 con 0x564f578a9af0 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6117fa640 1 -- 172.21.15.116:0/1032273784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc620018040 con 0x564f578a9af0 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6117fa640 1 -- 172.21.15.116:0/1032273784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620017040 con 0x564f578a9af0 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 -- 172.21.15.116:0/1032273784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 msgr2=0x564f578aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 0x564f578aff60 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fc620000c00 tx=0x7fc620005f90 comp rx=0 tx=0).stop 2024-03-24T09:26:31.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 -- 172.21.15.116:0/1032273784 shutdown_connections 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578b26d0 0x564f578ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578a9af0 0x564f578aff60 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 --2- 172.21.15.116:0/1032273784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 -- 172.21.15.116:0/1032273784 >> 172.21.15.116:0/1032273784 conn(0x564f578a3900 msgr2=0x564f578a4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 -- 172.21.15.116:0/1032273784 shutdown_connections 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.323+0000 7fc6347af740 1 -- 172.21.15.116:0/1032273784 wait complete. 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 -- start start 2024-03-24T09:26:31.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 Processor -- start 2024-03-24T09:26:31.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578ba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 0x564f578b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f578bac90 con 0x564f578b26d0 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f578bae00 con 0x564f578a91d0 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f578b8010 con 0x564f578a9af0 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc6347af740 10 client.0 _ll_register_callbacks cb 0x7fc61c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 0x564f578b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 0x564f578b7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54252/0 (socket says 172.21.15.116:54252) 2024-03-24T09:26:31.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 -- 172.21.15.116:0/3525299480 learned_addr learned my addr 172.21.15.116:0/3525299480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578ba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62affd640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 -- 172.21.15.116:0/3525299480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 msgr2=0x564f578b7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 -- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 msgr2=0x564f578ba590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578ba590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.324+0000 7fc62bfff640 1 -- 172.21.15.116:0/3525299480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc620012c00 con 0x564f578b26d0 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc62bfff640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 0x564f578b7aa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc624011fd0 tx=0x7fc624011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcd86883d41d3c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc60affd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc60affd640 1 -- 172.21.15.116:0/3525299480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62400cc50 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc60affd640 1 -- 172.21.15.116:0/3525299480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc624018070 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc62402aa50 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc60affd640 1 -- 172.21.15.116:0/3525299480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc62402a850 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 10 client.16875 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564f577d0700 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc60affd640 1 -- 172.21.15.116:0/3525299480 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc62402aa50 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc60affd640 1 -- 172.21.15.116:0/3525299480 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc624028070 con 0x564f578b26d0 2024-03-24T09:26:31.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 10 client.16875 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 20 client.16875 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f5792a0b0 0x564f5792c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 4 client.16875 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f5759b890 con 0x564f5792a0b0 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.325+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f5792a0b0 0x564f5792c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc60affd640 10 client.16875 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc62b7fe640 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f5792a0b0 0x564f5792c4d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf96905fed69ad76 server_cookie=a47c7e21c08a5018 in_seq=0 out_seq=0 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc60affd640 1 -- 172.21.15.116:0/3525299480 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564f5759b890 con 0x564f5792a0b0 2024-03-24T09:26:31.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc60affd640 10 client.16875 handle_command_reply: tid=0 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 client.16875 shutdown 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 msgr2=0x564f578b7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 0x564f578b7aa0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fc624011fd0 tx=0x7fc624011ec0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 shutdown_connections 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f578b26d0 0x564f578b7aa0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f578a9af0 0x564f578b7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f5792a0b0 0x564f5792c4d0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 --2- 172.21.15.116:0/3525299480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f578a91d0 0x564f578ba590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc60affd640 0 client.16875 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 >> 172.21.15.116:0/3525299480 conn(0x564f578a3900 msgr2=0x564f578a54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 shutdown_connections 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 1 -- 172.21.15.116:0/3525299480 wait complete. 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.326+0000 7fc6347af740 20 client.16875 trim_cache size 0 max 16384 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:26:31.337 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:26:31.338 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc6140a2230 msgr2=0x7fc6140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc6140a2230 0x7fc6140a46f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc624027000 tx=0x7fc624028000 comp rx=0 tx=0).stop 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c102980 msgr2=0x7fc62c19d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c102980 0x7fc62c19d900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc61c036040 tx=0x7fc61c005e20 comp rx=0 tx=0).stop 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 shutdown_connections 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc62c10bd50 0x7fc62c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc62c1032a0 0x7fc62c19de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc6140a2230 0x7fc6140a46f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 --2- 172.21.15.116:0/2487809436 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc62c102980 0x7fc62c19d900 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 >> 172.21.15.116:0/2487809436 conn(0x7fc62c077770 msgr2=0x7fc62c0fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 shutdown_connections 2024-03-24T09:26:31.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.327+0000 7fc631bc9640 1 -- 172.21.15.116:0/2487809436 wait complete. 2024-03-24T09:26:31.340 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:26:31.340 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 -- 172.21.15.116:0/169050057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54100d40 msgr2=0x7fec5410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 --2- 172.21.15.116:0/169050057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54100d40 0x7fec5410d4e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fec4c009be0 tx=0x7fec4c02bf80 comp rx=0 tx=0).stop 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 -- 172.21.15.116:0/169050057 shutdown_connections 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 --2- 172.21.15.116:0/169050057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54100d40 0x7fec5410d4e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 --2- 172.21.15.116:0/169050057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54106fd0 0x7fec54100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 --2- 172.21.15.116:0/169050057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec541066b0 0x7fec54106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 -- 172.21.15.116:0/169050057 >> 172.21.15.116:0/169050057 conn(0x7fec54077710 msgr2=0x7fec54077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 -- 172.21.15.116:0/169050057 shutdown_connections 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.480+0000 7fec5973e640 1 -- 172.21.15.116:0/169050057 wait complete. 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 Processor -- start 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 -- start start 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 0x7fec5419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec541066b0 0x7fec5419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54106fd0 0x7fec54196990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec54196fe0 con 0x7fec54100d40 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec54197150 con 0x7fec541066b0 2024-03-24T09:26:31.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec5973e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fec54197310 con 0x7fec54106fd0 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 0x7fec5419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 0x7fec5419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54278/0 (socket says 172.21.15.116:54278) 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec52ffd640 1 -- 172.21.15.116:0/686104928 learned_addr learned my addr 172.21.15.116:0/686104928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec4bfff640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec541066b0 0x7fec5419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.481+0000 7fec537fe640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54106fd0 0x7fec54196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec52ffd640 1 -- 172.21.15.116:0/686104928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54106fd0 msgr2=0x7fec54196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec52ffd640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54106fd0 0x7fec54196990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec52ffd640 1 -- 172.21.15.116:0/686104928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec541066b0 msgr2=0x7fec5419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec52ffd640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec541066b0 0x7fec5419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec52ffd640 1 -- 172.21.15.116:0/686104928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c009840 con 0x7fec54100d40 2024-03-24T09:26:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec52ffd640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 0x7fec5419c7e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fec4400eb00 tx=0x7fec44009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44013070 con 0x7fec54100d40 2024-03-24T09:26:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec541975f0 con 0x7fec54100d40 2024-03-24T09:26:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec440044e0 con 0x7fec54100d40 2024-03-24T09:26:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44003c50 con 0x7fec54100d40 2024-03-24T09:26:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.482+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec54068a50 con 0x7fec54100d40 2024-03-24T09:26:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.483+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec54101d50 con 0x7fec54100d40 2024-03-24T09:26:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.484+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fec44003df0 con 0x7fec54100d40 2024-03-24T09:26:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.484+0000 7fec50ff9640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec400a2230 0x7fec400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.485+0000 7fec4bfff640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec400a2230 0x7fec400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.485+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fec440cfc40 con 0x7fec54100d40 2024-03-24T09:26:31.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.485+0000 7fec4bfff640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec400a2230 0x7fec400a46f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fec54105cc0 tx=0x7fec3c005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.489+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fec440985d0 con 0x7fec54100d40 2024-03-24T09:26:31.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.701+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec54068500 con 0x7fec54100d40 2024-03-24T09:26:31.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.702+0000 7fec50ff9640 1 -- 172.21.15.116:0/686104928 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7fec4409bc90 con 0x7fec54100d40 2024-03-24T09:26:31.703 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:26:31.704 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:26:31.704 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:26:31.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec400a2230 msgr2=0x7fec400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec400a2230 0x7fec400a46f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fec54105cc0 tx=0x7fec3c005f90 comp rx=0 tx=0).stop 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 msgr2=0x7fec5419c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 0x7fec5419c7e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fec4400eb00 tx=0x7fec44009e00 comp rx=0 tx=0).stop 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 shutdown_connections 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec400a2230 0x7fec400a46f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec54106fd0 0x7fec54196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec541066b0 0x7fec5419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 --2- 172.21.15.116:0/686104928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec54100d40 0x7fec5419c7e0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 >> 172.21.15.116:0/686104928 conn(0x7fec54077710 msgr2=0x7fec540ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 shutdown_connections 2024-03-24T09:26:31.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.704+0000 7fec5973e640 1 -- 172.21.15.116:0/686104928 wait complete. 2024-03-24T09:26:31.716 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:26:31.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 -- 172.21.15.116:0/960722976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 msgr2=0x7fb5f0100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 --2- 172.21.15.116:0/960722976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f0100450 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80098c0 tx=0x7fb5d802bbe0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 -- 172.21.15.116:0/960722976 shutdown_connections 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 --2- 172.21.15.116:0/960722976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f0100990 0x7fb5f010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 --2- 172.21.15.116:0/960722976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f0100450 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 --2- 172.21.15.116:0/960722976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0107910 0x7fb5f0107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 -- 172.21.15.116:0/960722976 >> 172.21.15.116:0/960722976 conn(0x7fb5f0075950 msgr2=0x7fb5f0075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 -- 172.21.15.116:0/960722976 shutdown_connections 2024-03-24T09:26:31.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.819+0000 7fb5f5884640 1 -- 172.21.15.116:0/960722976 wait complete. 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 Processor -- start 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 -- start start 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 0x7fb5f019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f0107910 0x7fb5f0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f0198130 con 0x7fb5f0100990 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f01982a0 con 0x7fb5f00fffd0 2024-03-24T09:26:31.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5f5884640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f0198460 con 0x7fb5f0107910 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 0x7fb5f019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 0x7fb5f019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54294/0 (socket says 172.21.15.116:54294) 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 -- 172.21.15.116:0/1104333554 learned_addr learned my addr 172.21.15.116:0/1104333554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f0107910 0x7fb5f0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 -- 172.21.15.116:0/1104333554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f0107910 msgr2=0x7fb5f0197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f0107910 0x7fb5f0197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 -- 172.21.15.116:0/1104333554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 msgr2=0x7fb5f019d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f019d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 -- 172.21.15.116:0/1104333554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e0009560 con 0x7fb5f0100990 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.820+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 0x7fb5f019de70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8005940 tx=0x7fb5d8034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.821+0000 7fb5f4882640 1 -- 172.21.15.116:0/1104333554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d802c580 con 0x7fb5f0100990 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.821+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d8009490 con 0x7fb5f0100990 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.821+0000 7fb5f4882640 1 -- 172.21.15.116:0/1104333554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5d802cb90 con 0x7fb5f0100990 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.821+0000 7fb5f4882640 1 -- 172.21.15.116:0/1104333554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d803d3f0 con 0x7fb5f0100990 2024-03-24T09:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.821+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f01a47f0 con 0x7fb5f0100990 2024-03-24T09:26:31.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f4882640 1 -- 172.21.15.116:0/1104333554 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fb5d803b040 con 0x7fb5f0100990 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 Processor -- start 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 -- start start 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f4882640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb5dc0a2230 0x7fb5dc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 0x55e9bc9377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc9403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb5dc0a2230 0x7fb5dc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc942ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc8a5840 con 0x55e9bc9408f0 2024-03-24T09:26:31.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc8a2d70 con 0x55e9bc937d10 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f8464740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc89e550 con 0x55e9bc9373f0 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.823+0000 7fb5f4882640 1 -- 172.21.15.116:0/1104333554 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb5d80efd80 con 0x7fb5f0100990 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 0x55e9bc9377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 0x55e9bc9377d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:58968/0 (socket says 172.21.15.116:58968) 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 -- 172.21.15.116:0/3824654021 learned_addr learned my addr 172.21.15.116:0/3824654021 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc942ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc9403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 -- 172.21.15.116:0/3824654021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 msgr2=0x55e9bc9403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc9403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 -- 172.21.15.116:0/3824654021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 msgr2=0x55e9bc942ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc942ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb5dc0a2230 0x7fb5dc0a46f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8005fd0 tx=0x7fb5e8013b60 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 -- 172.21.15.116:0/3824654021 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9bc833de0 con 0x55e9bc9373f0 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc9403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 0x55e9bc9377d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0011680 tx=0x7fb5e0011b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=282aceac04226e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5e5ffb640 1 -- 172.21.15.116:0/3824654021 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e000fb20 con 0x55e9bc9373f0 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5e5ffb640 1 -- 172.21.15.116:0/3824654021 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5e00054d0 con 0x55e9bc9373f0 2024-03-24T09:26:31.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.824+0000 7fb5e5ffb640 1 -- 172.21.15.116:0/3824654021 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0004d10 con 0x55e9bc9373f0 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 -- 172.21.15.116:0/3824654021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 msgr2=0x55e9bc9377d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 0x55e9bc9377d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0011680 tx=0x7fb5e0011b40 comp rx=0 tx=0).stop 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 -- 172.21.15.116:0/3824654021 shutdown_connections 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc942ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc9403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 --2- 172.21.15.116:0/3824654021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9373f0 0x55e9bc9377d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 -- 172.21.15.116:0/3824654021 >> 172.21.15.116:0/3824654021 conn(0x55e9bc931b20 msgr2=0x55e9bc931f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 -- 172.21.15.116:0/3824654021 shutdown_connections 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 -- 172.21.15.116:0/3824654021 wait complete. 2024-03-24T09:26:31.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.825+0000 7fb5f8464740 1 -- start start 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 Processor -- start 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 0x55e9bc93d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc93de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9408f0 0x55e9bc93e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc571430 con 0x55e9bc9373f0 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc52de00 con 0x55e9bc937d10 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc823590 con 0x55e9bc9408f0 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 0x55e9bc93d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5f8464740 10 client.0 _ll_register_callbacks cb 0x55e9bc892fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 0x55e9bc93d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54310/0 (socket says 172.21.15.116:54310) 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 -- 172.21.15.116:0/2719103330 learned_addr learned my addr 172.21.15.116:0/2719103330 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc93de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9408f0 0x55e9bc93e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 -- 172.21.15.116:0/2719103330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9408f0 msgr2=0x55e9bc93e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9408f0 0x55e9bc93e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 -- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 msgr2=0x55e9bc93de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc93de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 -- 172.21.15.116:0/2719103330 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e0009e70 con 0x55e9bc9373f0 2024-03-24T09:26:31.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 0x55e9bc93d910 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb5e00048f0 tx=0x7fb5e0005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624a4d046ee4fc38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.826+0000 7fb5ceffd640 1 -- 172.21.15.116:0/2719103330 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0004e60 con 0x55e9bc9373f0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5ceffd640 1 -- 172.21.15.116:0/2719103330 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb5e0004fc0 con 0x55e9bc9373f0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e9bc59d7e0 con 0x55e9bc9373f0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5ceffd640 1 -- 172.21.15.116:0/2719103330 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb5e0005180 con 0x55e9bc9373f0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 10 client.16893 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9bc89cbd0 con 0x55e9bc9373f0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5ceffd640 1 -- 172.21.15.116:0/2719103330 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e9bc59d7e0 con 0x55e9bc9373f0 2024-03-24T09:26:31.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5ceffd640 1 -- 172.21.15.116:0/2719103330 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fb5e002b020 con 0x55e9bc9373f0 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 10 client.16893 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 10 client.16893 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 20 client.16893 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc950990 0x55e9bc952da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 4 client.16893 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9bc5da720 con 0x55e9bc950990 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.827+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc950990 0x55e9bc952da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5ceffd640 10 client.16893 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc950990 0x55e9bc952da0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=28637fb2171bac2e server_cookie=90ee7803ec5b5f1 in_seq=0 out_seq=0 2024-03-24T09:26:31.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5ceffd640 1 -- 172.21.15.116:0/2719103330 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e9bc5da720 con 0x55e9bc950990 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5ceffd640 10 client.16893 handle_command_reply: tid=0 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 client.16893 shutdown 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 msgr2=0x55e9bc93d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 0x55e9bc93d910 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb5e00048f0 tx=0x7fb5e0005e70 comp rx=0 tx=0).stop 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 shutdown_connections 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc9408f0 0x55e9bc93e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc937d10 0x55e9bc93de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc950990 0x55e9bc952da0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 --2- 172.21.15.116:0/2719103330 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9373f0 0x55e9bc93d910 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5ceffd640 0 client.16893 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:26:31.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.828+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 >> 172.21.15.116:0/2719103330 conn(0x55e9bc931b20 msgr2=0x55e9bc933870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.829+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 shutdown_connections 2024-03-24T09:26:31.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.829+0000 7fb5f8464740 1 -- 172.21.15.116:0/2719103330 wait complete. 2024-03-24T09:26:31.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.829+0000 7fb5f8464740 20 client.16893 trim_cache size 0 max 16384 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 Processor -- start 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 -- start start 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc940040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc94c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc9586b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc932b20 con 0x55e9bc9408f0 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc9367f0 con 0x55e9bc9373f0 2024-03-24T09:26:31.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5f8464740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc939520 con 0x55e9bc937d10 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc9586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc9586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54316/0 (socket says 172.21.15.116:54316) 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/504716892 learned_addr learned my addr 172.21.15.116:0/504716892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc940040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc94c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/504716892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 msgr2=0x55e9bc94c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc94c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/504716892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 msgr2=0x55e9bc940040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc940040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/504716892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9bc838c80 con 0x55e9bc9408f0 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc9586b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8012420 tx=0x7fb5e801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e27d4f8f98909d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.842+0000 7fb5e57fa640 1 -- 172.21.15.116:0/504716892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8025070 con 0x55e9bc9408f0 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5e57fa640 1 -- 172.21.15.116:0/504716892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5e8003970 con 0x55e9bc9408f0 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5e57fa640 1 -- 172.21.15.116:0/504716892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8020d40 con 0x55e9bc9408f0 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 -- 172.21.15.116:0/504716892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 msgr2=0x55e9bc9586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc9586b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8012420 tx=0x7fb5e801f000 comp rx=0 tx=0).stop 2024-03-24T09:26:31.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 -- 172.21.15.116:0/504716892 shutdown_connections 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc9586b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc94c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 --2- 172.21.15.116:0/504716892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc940040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 -- 172.21.15.116:0/504716892 >> 172.21.15.116:0/504716892 conn(0x55e9bc931b20 msgr2=0x55e9bc930cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 -- 172.21.15.116:0/504716892 shutdown_connections 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 -- 172.21.15.116:0/504716892 wait complete. 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.843+0000 7fb5f8464740 1 -- start start 2024-03-24T09:26:31.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 Processor -- start 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc9487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc945780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc945cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc948cf0 con 0x55e9bc9408f0 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc948e60 con 0x55e9bc9373f0 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e9bc949020 con 0x55e9bc937d10 2024-03-24T09:26:31.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5f8464740 10 client.0 _ll_register_callbacks cb 0x7fb5e8009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc945cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc945cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54318/0 (socket says 172.21.15.116:54318) 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5eeffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc9487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/517291374 learned_addr learned my addr 172.21.15.116:0/517291374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc945780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/517291374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 msgr2=0x55e9bc945780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc945780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 msgr2=0x55e9bc9487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc9487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 -- 172.21.15.116:0/517291374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e80121b0 con 0x55e9bc9408f0 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc9487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5e7fff640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc945780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ef7fe640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc945cc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009b40 tx=0x7fb5e800ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759ef8f3595a51c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.844+0000 7fb5ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5ceffd640 1 -- 172.21.15.116:0/517291374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8025040 con 0x55e9bc9408f0 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5ceffd640 1 -- 172.21.15.116:0/517291374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb5e8005000 con 0x55e9bc9408f0 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5e00054d0 con 0x55e9bc9408f0 2024-03-24T09:26:31.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5ceffd640 1 -- 172.21.15.116:0/517291374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb5e8020e50 con 0x55e9bc9408f0 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 10 client.16899 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e9bc85e920 con 0x55e9bc9408f0 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5ceffd640 1 -- 172.21.15.116:0/517291374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb5e803c020 con 0x55e9bc9408f0 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5ceffd640 1 -- 172.21.15.116:0/517291374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fb5e8010880 con 0x55e9bc9408f0 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 10 client.16899 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 10 client.16899 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 20 client.16899 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc9b7e90 0x55e9bc9ba2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 4 client.16899 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e9bc629890 con 0x55e9bc9b7e90 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.845+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc9b7e90 0x55e9bc9ba2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5ceffd640 10 client.16899 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5eeffd640 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc9b7e90 0x55e9bc9ba2b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=de73f542a409a30a server_cookie=31ec32f54c5b11f9 in_seq=0 out_seq=0 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5ceffd640 1 -- 172.21.15.116:0/517291374 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e9bc629890 con 0x55e9bc9b7e90 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5ceffd640 10 client.16899 handle_command_reply: tid=0 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 client.16899 shutdown 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 msgr2=0x55e9bc945cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc945cc0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009b40 tx=0x7fb5e800ff90 comp rx=0 tx=0).stop 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 shutdown_connections 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e9bc9b7e90 0x55e9bc9ba2b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e9bc9408f0 0x55e9bc945cc0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e9bc937d10 0x55e9bc945780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5ceffd640 0 client.16899 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 --2- 172.21.15.116:0/517291374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e9bc9373f0 0x55e9bc9487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 >> 172.21.15.116:0/517291374 conn(0x55e9bc931b20 msgr2=0x55e9bc930430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 shutdown_connections 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 1 -- 172.21.15.116:0/517291374 wait complete. 2024-03-24T09:26:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.846+0000 7fb5f8464740 20 client.16899 trim_cache size 0 max 16384 2024-03-24T09:26:31.858 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:26:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb5dc0a2230 msgr2=0x7fb5dc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb5dc0a2230 0x7fb5dc0a46f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8005fd0 tx=0x7fb5e8013b60 comp rx=0 tx=0).stop 2024-03-24T09:26:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 msgr2=0x7fb5f019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 0x7fb5f019de70 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8005940 tx=0x7fb5d8034440 comp rx=0 tx=0).stop 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 shutdown_connections 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb5dc0a2230 0x7fb5dc0a46f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb5f0107910 0x7fb5f0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5f0100990 0x7fb5f019de70 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 --2- 172.21.15.116:0/1104333554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5f00fffd0 0x7fb5f019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 >> 172.21.15.116:0/1104333554 conn(0x7fb5f0075950 msgr2=0x7fb5f00fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 shutdown_connections 2024-03-24T09:26:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.847+0000 7fb5f5884640 1 -- 172.21.15.116:0/1104333554 wait complete. 2024-03-24T09:26:31.861 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- 172.21.15.116:0/3675630500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a8102060 msgr2=0x7fb3a810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- 172.21.15.116:0/3675630500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a8102060 0x7fb3a810e570 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0009930 tx=0x7fb3a002bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- 172.21.15.116:0/3675630500 shutdown_connections 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- 172.21.15.116:0/3675630500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a8102060 0x7fb3a810e570 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- 172.21.15.116:0/3675630500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a81016a0 0x7fb3a8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- 172.21.15.116:0/3675630500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81076b0 0x7fb3a8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- 172.21.15.116:0/3675630500 >> 172.21.15.116:0/3675630500 conn(0x7fb3a80fd1b0 msgr2=0x7fb3a80ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- 172.21.15.116:0/3675630500 shutdown_connections 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- 172.21.15.116:0/3675630500 wait complete. 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 Processor -- start 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- start start 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81016a0 0x7fb3a819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a8102060 0x7fb3a819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 0x7fb3a8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a81980d0 con 0x7fb3a81076b0 2024-03-24T09:26:31.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8198240 con 0x7fb3a8102060 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.963+0000 7fb3b075e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a8198400 con 0x7fb3a81016a0 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 0x7fb3a8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81016a0 0x7fb3a819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3adcf9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a8102060 0x7fb3a819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81016a0 0x7fb3a819d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:59010/0 (socket says 172.21.15.116:59010) 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3ae4fa640 1 -- 172.21.15.116:0/333133153 learned_addr learned my addr 172.21.15.116:0/333133153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3adcf9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a8102060 0x7fb3a819de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48612/0 (socket says 172.21.15.116:48612) 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 0x7fb3a8197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54332/0 (socket says 172.21.15.116:54332) 2024-03-24T09:26:31.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 -- 172.21.15.116:0/333133153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81016a0 msgr2=0x7fb3a819d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81016a0 0x7fb3a819d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 -- 172.21.15.116:0/333133153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a8102060 msgr2=0x7fb3a819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a8102060 0x7fb3a819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 -- 172.21.15.116:0/333133153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a0009560 con 0x7fb3a81076b0 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 0x7fb3a8197a80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb3a002c170 tx=0x7fb3a002cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3a77fe640 1 -- 172.21.15.116:0/333133153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0004350 con 0x7fb3a81076b0 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3a77fe640 1 -- 172.21.15.116:0/333133153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3a00044b0 con 0x7fb3a81076b0 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a8198680 con 0x7fb3a81076b0 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3a77fe640 1 -- 172.21.15.116:0/333133153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00053c0 con 0x7fb3a81076b0 2024-03-24T09:26:31.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.964+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a81a4970 con 0x7fb3a81076b0 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.966+0000 7fb3a77fe640 1 -- 172.21.15.116:0/333133153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fb3a0005520 con 0x7fb3a81076b0 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 Processor -- start 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 -- start start 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3a77fe640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3900a2230 0x7fb3900a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 0x55dcf423b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423bd10 0x55dcf42443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf42448f0 0x55dcf4246ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dcf41a9840 con 0x55dcf423b3f0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dcf41a6d70 con 0x55dcf42448f0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3b341a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dcf41a2550 con 0x55dcf423bd10 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3900a2230 0x7fb3900a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 0x55dcf423b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3a77fe640 1 -- 172.21.15.116:0/333133153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb3a002de40 con 0x7fb3a81076b0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3aecfb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf42448f0 0x55dcf4246ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 0x55dcf423b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54336/0 (socket says 172.21.15.116:54336) 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3ae4fa640 1 -- 172.21.15.116:0/792598275 learned_addr learned my addr 172.21.15.116:0/792598275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423bd10 0x55dcf42443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3aecfb640 1 -- 172.21.15.116:0/792598275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423bd10 msgr2=0x55dcf42443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.967+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423bd10 0x55dcf42443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3aecfb640 1 -- 172.21.15.116:0/792598275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 msgr2=0x55dcf423b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 0x55dcf423b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3aecfb640 1 -- 172.21.15.116:0/792598275 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcf4137de0 con 0x55dcf42448f0 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 0x55dcf423b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3900a2230 0x7fb3900a46f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80ff820 tx=0x7fb39c009ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf42448f0 0x55dcf4246ce0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00edde0 tx=0x7fb3a00bce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1e802aba6714bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3a4ff9640 1 -- 172.21.15.116:0/792598275 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00a0e80 con 0x55dcf42448f0 2024-03-24T09:26:31.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3a4ff9640 1 -- 172.21.15.116:0/792598275 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3a0046310 con 0x55dcf42448f0 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.968+0000 7fb3a4ff9640 1 -- 172.21.15.116:0/792598275 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00c5820 con 0x55dcf42448f0 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- 172.21.15.116:0/792598275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf42448f0 msgr2=0x55dcf4246ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf42448f0 0x55dcf4246ce0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00edde0 tx=0x7fb3a00bce50 comp rx=0 tx=0).stop 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- 172.21.15.116:0/792598275 shutdown_connections 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf42448f0 0x55dcf4246ce0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423bd10 0x55dcf42443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- 172.21.15.116:0/792598275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423b3f0 0x55dcf423b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- 172.21.15.116:0/792598275 >> 172.21.15.116:0/792598275 conn(0x55dcf4235b20 msgr2=0x55dcf4235f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- 172.21.15.116:0/792598275 shutdown_connections 2024-03-24T09:26:31.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- 172.21.15.116:0/792598275 wait complete. 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- start start 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 Processor -- start 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf4241800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf4241d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4242280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3e75430 con 0x55dcf42448f0 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3e31e00 con 0x55dcf423bd10 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dcf4127590 con 0x55dcf423b3f0 2024-03-24T09:26:31.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3b341a740 10 client.0 _ll_register_callbacks cb 0x55dcf4196fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3aecfb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4242280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3aecfb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4242280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54350/0 (socket says 172.21.15.116:54350) 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf4241800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3aecfb640 1 -- 172.21.15.116:0/3047216209 learned_addr learned my addr 172.21.15.116:0/3047216209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.969+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf4241d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3aecfb640 1 -- 172.21.15.116:0/3047216209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 msgr2=0x55dcf4241800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf4241800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3aecfb640 1 -- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 msgr2=0x55dcf4241d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf4241d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3aecfb640 1 -- 172.21.15.116:0/3047216209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a81a4970 con 0x55dcf42448f0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4242280 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00ee2d0 tx=0x7fb3a00098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89cd3a01da41f0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb38dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb38dffb640 1 -- 172.21.15.116:0/3047216209 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a00d3070 con 0x55dcf42448f0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb38dffb640 1 -- 172.21.15.116:0/3047216209 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb3a003cba0 con 0x55dcf42448f0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dcf3ea17e0 con 0x55dcf42448f0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb38dffb640 1 -- 172.21.15.116:0/3047216209 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb3a00bd780 con 0x55dcf42448f0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 10 client.16911 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dcf41a0bd0 con 0x55dcf42448f0 2024-03-24T09:26:31.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb38dffb640 1 -- 172.21.15.116:0/3047216209 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55dcf3ea17e0 con 0x55dcf42448f0 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb38dffb640 1 -- 172.21.15.116:0/3047216209 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fb3a003c070 con 0x55dcf42448f0 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 10 client.16911 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 10 client.16911 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 20 client.16911 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf4254230 0x55dcf4256650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 4 client.16911 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.970+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcf3ede720 con 0x55dcf4254230 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.971+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf4254230 0x55dcf4256650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.971+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf4254230 0x55dcf4256650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=def871b29602ebf2 server_cookie=e5eb2c934dcdd977 in_seq=0 out_seq=0 2024-03-24T09:26:31.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.971+0000 7fb38dffb640 10 client.16911 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb38dffb640 1 -- 172.21.15.116:0/3047216209 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dcf3ede720 con 0x55dcf4254230 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb38dffb640 10 client.16911 handle_command_reply: tid=0 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 client.16911 shutdown 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 msgr2=0x55dcf4242280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4242280 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00ee2d0 tx=0x7fb3a00098c0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 shutdown_connections 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf4254230 0x55dcf4256650 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4242280 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf4241d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 --2- 172.21.15.116:0/3047216209 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf4241800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 >> 172.21.15.116:0/3047216209 conn(0x55dcf4235b20 msgr2=0x55dcf4237870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb38dffb640 0 client.16911 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:26:31.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 shutdown_connections 2024-03-24T09:26:31.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 1 -- 172.21.15.116:0/3047216209 wait complete. 2024-03-24T09:26:31.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.972+0000 7fb3b341a740 20 client.16911 trim_cache size 0 max 16384 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 Processor -- start 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 -- start start 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423b3f0 0x55dcf424ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 0x55dcf4240960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf42448f0 0x55dcf42420a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dcf423d240 con 0x55dcf423bd10 2024-03-24T09:26:31.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dcf3ede750 con 0x55dcf423b3f0 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3b341a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dcf42472b0 con 0x55dcf42448f0 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 0x55dcf4240960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 0x55dcf4240960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54362/0 (socket says 172.21.15.116:54362) 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 -- 172.21.15.116:0/140215386 learned_addr learned my addr 172.21.15.116:0/140215386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423b3f0 0x55dcf424ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3aecfb640 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf42448f0 0x55dcf42420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 -- 172.21.15.116:0/140215386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf42448f0 msgr2=0x55dcf42420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf42448f0 0x55dcf42420a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 -- 172.21.15.116:0/140215386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423b3f0 msgr2=0x55dcf424ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423b3f0 0x55dcf424ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3adcf9640 1 -- 172.21.15.116:0/140215386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcf413cc80 con 0x55dcf423bd10 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.985+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423b3f0 0x55dcf424ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 0x55dcf4240960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009fd0 tx=0x7fb39c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292aea36dc42866d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb38ffff640 1 -- 172.21.15.116:0/140215386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c026070 con 0x55dcf423bd10 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb38ffff640 1 -- 172.21.15.116:0/140215386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb39c004400 con 0x55dcf423bd10 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb38ffff640 1 -- 172.21.15.116:0/140215386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c004cc0 con 0x55dcf423bd10 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 -- 172.21.15.116:0/140215386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 msgr2=0x55dcf4240960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 0x55dcf4240960 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009fd0 tx=0x7fb39c021000 comp rx=0 tx=0).stop 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 -- 172.21.15.116:0/140215386 shutdown_connections 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf42448f0 0x55dcf42420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf423bd10 0x55dcf4240960 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 --2- 172.21.15.116:0/140215386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423b3f0 0x55dcf424ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 -- 172.21.15.116:0/140215386 >> 172.21.15.116:0/140215386 conn(0x55dcf4235b20 msgr2=0x55dcf4234430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 -- 172.21.15.116:0/140215386 shutdown_connections 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.986+0000 7fb3b341a740 1 -- 172.21.15.116:0/140215386 wait complete. 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 -- start start 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 Processor -- start 2024-03-24T09:26:31.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf42b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf42b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4247690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55dcf4247c50 con 0x55dcf42448f0 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dcf4247dc0 con 0x55dcf423bd10 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55dcf4247f80 con 0x55dcf423b3f0 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3b341a740 10 client.0 _ll_register_callbacks cb 0x7fb3a00b9320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3aecfb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4247690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf42b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf42b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:59050/0 (socket says 172.21.15.116:59050) 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 -- 172.21.15.116:0/867627012 learned_addr learned my addr 172.21.15.116:0/867627012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf42b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 -- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 msgr2=0x55dcf42b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf42b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 -- 172.21.15.116:0/867627012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 msgr2=0x55dcf4247690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4247690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 -- 172.21.15.116:0/867627012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb398012450 con 0x55dcf423b3f0 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb3ae4fa640 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf42b28c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3980129c0 tx=0x7fb398012e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5105158a057f3198 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.987+0000 7fb38dffb640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:26:31.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb38dffb640 1 -- 172.21.15.116:0/867627012 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb398017040 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb3a0046310 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb38dffb640 1 -- 172.21.15.116:0/867627012 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb39801ec70 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb38dffb640 1 -- 172.21.15.116:0/867627012 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb39801ee30 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 10 client.26617 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7fb39c01b040 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb38dffb640 1 -- 172.21.15.116:0/867627012 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb3a0046310 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb38dffb640 1 -- 172.21.15.116:0/867627012 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fb398010790 con 0x55dcf423b3f0 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 10 client.26617 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 10 client.26617 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:26:31.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 20 client.26617 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf42ba8f0 0x55dcf42bcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 4 client.26617 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcf3f2d890 con 0x55dcf42ba8f0 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.988+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf42ba8f0 0x55dcf42bcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb38dffb640 10 client.26617 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3adcf9640 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf42ba8f0 0x55dcf42bcd10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9b4e4357f117ea49 server_cookie=e50d1164e5b131a2 in_seq=0 out_seq=0 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb38dffb640 1 -- 172.21.15.116:0/867627012 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcf3f2d890 con 0x55dcf42ba8f0 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb38dffb640 10 client.26617 handle_command_reply: tid=0 2024-03-24T09:26:31.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 client.26617 shutdown 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 msgr2=0x55dcf42b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf42b28c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3980129c0 tx=0x7fb398012e80 comp rx=0 tx=0).stop 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 shutdown_connections 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55dcf42ba8f0 0x55dcf42bcd10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55dcf42448f0 0x55dcf4247690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcf423bd10 0x55dcf42b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb38dffb640 0 client.26617 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 --2- 172.21.15.116:0/867627012 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55dcf423b3f0 0x55dcf42b28c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:31.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 >> 172.21.15.116:0/867627012 conn(0x55dcf4235b20 msgr2=0x55dcf42420e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 shutdown_connections 2024-03-24T09:26:32.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 1 -- 172.21.15.116:0/867627012 wait complete. 2024-03-24T09:26:32.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b341a740 20 client.26617 trim_cache size 0 max 16384 2024-03-24T09:26:32.000 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:26:32.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3900a2230 msgr2=0x7fb3900a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b075e640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3900a2230 0x7fb3900a46f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80ff820 tx=0x7fb39c009ab0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 msgr2=0x7fb3a8197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b075e640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 0x7fb3a8197a80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fb3a002c170 tx=0x7fb3a002cd10 comp rx=0 tx=0).stop 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.989+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 shutdown_connections 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a81076b0 0x7fb3a8197a80 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3a8102060 0x7fb3a819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3900a2230 0x7fb3900a46f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 --2- 172.21.15.116:0/333133153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3a81016a0 0x7fb3a819d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 >> 172.21.15.116:0/333133153 conn(0x7fb3a80fd1b0 msgr2=0x7fb3a80fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 shutdown_connections 2024-03-24T09:26:32.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:31.990+0000 7fb3b075e640 1 -- 172.21.15.116:0/333133153 wait complete. 2024-03-24T09:26:32.003 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:31 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/686104928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:26:32.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:26:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 -- 172.21.15.116:0/1132219554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574107690 msgr2=0x7fc574107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1132219554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574107690 0x7fc574107a70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc568009930 tx=0x7fc56802bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 -- 172.21.15.116:0/1132219554 shutdown_connections 2024-03-24T09:26:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1132219554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5740692b0 0x7fc57410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1132219554 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5740688f0 0x7fc574068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1132219554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc574107690 0x7fc574107a70 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 -- 172.21.15.116:0/1132219554 >> 172.21.15.116:0/1132219554 conn(0x7fc574077710 msgr2=0x7fc574077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 -- 172.21.15.116:0/1132219554 shutdown_connections 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 -- 172.21.15.116:0/1132219554 wait complete. 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 Processor -- start 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.107+0000 7fc578bd7640 1 -- start start 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 0x7fc57419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5740692b0 0x7fc57419de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc574107690 0x7fc574197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc574198120 con 0x7fc5740688f0 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc574198290 con 0x7fc574107690 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc574198450 con 0x7fc5740692b0 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 0x7fc57419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 0x7fc57419d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54390/0 (socket says 172.21.15.116:54390) 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 -- 172.21.15.116:0/1009053235 learned_addr learned my addr 172.21.15.116:0/1009053235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc571d9b640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5740692b0 0x7fc57419de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc572d9d640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc574107690 0x7fc574197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 -- 172.21.15.116:0/1009053235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5740692b0 msgr2=0x7fc57419de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5740692b0 0x7fc57419de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 -- 172.21.15.116:0/1009053235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc574107690 msgr2=0x7fc574197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc574107690 0x7fc574197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 -- 172.21.15.116:0/1009053235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc568009560 con 0x7fc5740688f0 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc57259c640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 0x7fc57419d920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc568009230 tx=0x7fc568005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc5637fe640 1 -- 172.21.15.116:0/1009053235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568004350 con 0x7fc5740688f0 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5741986d0 con 0x7fc5740688f0 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc5637fe640 1 -- 172.21.15.116:0/1009053235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc56802de90 con 0x7fc5740688f0 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc5637fe640 1 -- 172.21.15.116:0/1009053235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568005220 con 0x7fc5740688f0 2024-03-24T09:26:32.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.108+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5741a49c0 con 0x7fc5740688f0 2024-03-24T09:26:32.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc5637fe640 1 -- 172.21.15.116:0/1009053235 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc568003c30 con 0x7fc5740688f0 2024-03-24T09:26:32.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 Processor -- start 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 -- start start 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 0x55caa96cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cfd10 0x55caa96d83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96dace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55caa963d840 con 0x55caa96d88f0 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55caa963ad70 con 0x55caa96cf3f0 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55caa9636550 con 0x55caa96cfd10 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc572d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 0x55caa96cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc571d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96dace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc572d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 0x55caa96cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48664/0 (socket says 172.21.15.116:48664) 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc572d9d640 1 -- 172.21.15.116:0/3362122046 learned_addr learned my addr 172.21.15.116:0/3362122046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc57259c640 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cfd10 0x55caa96d83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc5637fe640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc55c0a2230 0x7fc55c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.111+0000 7fc572d9d640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc55c0a2230 0x7fc55c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc571d9b640 1 -- 172.21.15.116:0/3362122046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cfd10 msgr2=0x55caa96d83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc571d9b640 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cfd10 0x55caa96d83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc571d9b640 1 -- 172.21.15.116:0/3362122046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 msgr2=0x55caa96cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc571d9b640 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 0x55caa96cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc571d9b640 1 -- 172.21.15.116:0/3362122046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caa95cbde0 con 0x55caa96d88f0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc572d9d640 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 0x55caa96cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc5637fe640 1 -- 172.21.15.116:0/1009053235 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc5680f0280 con 0x7fc5740688f0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc571d9b640 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96dace0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc574198c10 tx=0x7fc564011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d173d16ad0b67f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc572d9d640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc55c0a2230 0x7fc55c0a46f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0049d0 tx=0x7fc56c004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc560ff9640 1 -- 172.21.15.116:0/3362122046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564004230 con 0x55caa96d88f0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc560ff9640 1 -- 172.21.15.116:0/3362122046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc564009e10 con 0x55caa96d88f0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.112+0000 7fc560ff9640 1 -- 172.21.15.116:0/3362122046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564004bc0 con 0x55caa96d88f0 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3362122046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 msgr2=0x55caa96dace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96dace0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fc574198c10 tx=0x7fc564011810 comp rx=0 tx=0).stop 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3362122046 shutdown_connections 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96dace0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cfd10 0x55caa96d83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3362122046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cf3f0 0x55caa96cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3362122046 >> 172.21.15.116:0/3362122046 conn(0x55caa96c9b20 msgr2=0x55caa96c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3362122046 shutdown_connections 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3362122046 wait complete. 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- start start 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 Processor -- start 2024-03-24T09:26:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55caa9309430 con 0x55caa96d88f0 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55caa92c5e00 con 0x55caa96cfd10 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55caa95bb590 con 0x55caa96cf3f0 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.113+0000 7fc57b7b8740 10 client.0 _ll_register_callbacks cb 0x55caa962afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc57259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc571d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54396/0 (socket says 172.21.15.116:54396) 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 -- 172.21.15.116:0/1679357441 learned_addr learned my addr 172.21.15.116:0/1679357441 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 -- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 msgr2=0x55caa96d5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 -- 172.21.15.116:0/1679357441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 msgr2=0x55caa96d5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 -- 172.21.15.116:0/1679357441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc564009830 con 0x55caa96d88f0 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc571d9b640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc57259c640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc572d9d640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d6380 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc56c00fc20 tx=0x7fc56c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec21f51ba2778b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc559ffb640 1 -- 172.21.15.116:0/1679357441 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c022650 con 0x55caa96d88f0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc559ffb640 1 -- 172.21.15.116:0/1679357441 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc56c0227b0 con 0x55caa96d88f0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55caa93357e0 con 0x55caa96d88f0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc559ffb640 1 -- 172.21.15.116:0/1679357441 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc56c010e80 con 0x55caa96d88f0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc57b7b8740 10 client.16929 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55caa9634bd0 con 0x55caa96d88f0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.114+0000 7fc559ffb640 1 -- 172.21.15.116:0/1679357441 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55caa93357e0 con 0x55caa96d88f0 2024-03-24T09:26:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc559ffb640 1 -- 172.21.15.116:0/1679357441 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc56c028020 con 0x55caa96d88f0 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57b7b8740 10 client.16929 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57b7b8740 10 client.16929 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57b7b8740 20 client.16929 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa96e8ae0 0x55caa96eaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57b7b8740 4 client.16929 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55caa9372720 con 0x55caa96e8ae0 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57259c640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa96e8ae0 0x55caa96eaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc559ffb640 10 client.16929 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:26:32.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.115+0000 7fc57259c640 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa96e8ae0 0x55caa96eaf00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69b2ee4c9dabbc47 server_cookie=a8a069de548dee54 in_seq=0 out_seq=0 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc559ffb640 1 -- 172.21.15.116:0/1679357441 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55caa9372720 con 0x55caa96e8ae0 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc559ffb640 10 client.16929 handle_command_reply: tid=0 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 client.16929 shutdown 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 msgr2=0x55caa96d6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d6380 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fc56c00fc20 tx=0x7fc56c004a40 comp rx=0 tx=0).stop 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 shutdown_connections 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d6380 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa96e8ae0 0x55caa96eaf00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc559ffb640 0 client.16929 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:26:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/1679357441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 >> 172.21.15.116:0/1679357441 conn(0x55caa96c9b20 msgr2=0x55caa96cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 shutdown_connections 2024-03-24T09:26:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 1 -- 172.21.15.116:0/1679357441 wait complete. 2024-03-24T09:26:32.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.116+0000 7fc57b7b8740 20 client.16929 trim_cache size 0 max 16384 2024-03-24T09:26:32.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 Processor -- start 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 -- start start 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 0x55caa96d8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96d88f0 0x55caa96f0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55caa96ca6f0 con 0x55caa96cf3f0 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55caa96dace0 con 0x55caa96cfd10 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55caa96c53e0 con 0x55caa96d88f0 2024-03-24T09:26:32.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 0x55caa96d8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 0x55caa96d8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54404/0 (socket says 172.21.15.116:54404) 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc57259c640 1 -- 172.21.15.116:0/3814405260 learned_addr learned my addr 172.21.15.116:0/3814405260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.129+0000 7fc571d9b640 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc572d9d640 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96d88f0 0x55caa96f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57259c640 1 -- 172.21.15.116:0/3814405260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96d88f0 msgr2=0x55caa96f0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57259c640 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96d88f0 0x55caa96f0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57259c640 1 -- 172.21.15.116:0/3814405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 msgr2=0x55caa96d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57259c640 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57259c640 1 -- 172.21.15.116:0/3814405260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55caa95d0c80 con 0x55caa96cf3f0 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57259c640 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 0x55caa96d8030 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc5680f2750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf2b41e90544d4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc55bfff640 1 -- 172.21.15.116:0/3814405260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5680bc510 con 0x55caa96cf3f0 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc55bfff640 1 -- 172.21.15.116:0/3814405260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5680dc040 con 0x55caa96cf3f0 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc55bfff640 1 -- 172.21.15.116:0/3814405260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5680c5170 con 0x55caa96cf3f0 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3814405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 msgr2=0x55caa96d8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.130+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 0x55caa96d8030 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc5680f2750 comp rx=0 tx=0).stop 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3814405260 shutdown_connections 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96d88f0 0x55caa96f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/3814405260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96cf3f0 0x55caa96d8030 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3814405260 >> 172.21.15.116:0/3814405260 conn(0x55caa96c9b20 msgr2=0x55caa96cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3814405260 shutdown_connections 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- 172.21.15.116:0/3814405260 wait complete. 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- start start 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 Processor -- start 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55caa96e0ff0 con 0x55caa96d88f0 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55caa974de40 con 0x55caa96cfd10 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55caa974dfb0 con 0x55caa96cf3f0 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57b7b8740 10 client.0 _ll_register_callbacks cb 0x7fc56c013620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc572d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc57259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc572d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54416/0 (socket says 172.21.15.116:54416) 2024-03-24T09:26:32.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc572d9d640 1 -- 172.21.15.116:0/853550932 learned_addr learned my addr 172.21.15.116:0/853550932 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.131+0000 7fc571d9b640 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc572d9d640 1 -- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 msgr2=0x55caa96d23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc572d9d640 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc572d9d640 1 -- 172.21.15.116:0/853550932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 msgr2=0x55caa96d2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc572d9d640 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc572d9d640 1 -- 172.21.15.116:0/853550932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5741a49c0 con 0x55caa96d88f0 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc572d9d640 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d2e50 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc56c006010 tx=0x7fc56c02a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c5e68ebc3297ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:32.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc559ffb640 1 -- 172.21.15.116:0/853550932 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c010e80 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc559ffb640 1 -- 172.21.15.116:0/853550932 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc56c00fe60 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc564009e10 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc559ffb640 1 -- 172.21.15.116:0/853550932 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc56c022450 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 10 client.16935 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55caa95f6920 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc559ffb640 1 -- 172.21.15.116:0/853550932 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc564009e10 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc559ffb640 1 -- 172.21.15.116:0/853550932 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc56c028040 con 0x55caa96d88f0 2024-03-24T09:26:32.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 10 client.16935 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 10 client.16935 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 20 client.16935 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa974f710 0x55caa9751b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 4 client.16935 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55caa93c1890 con 0x55caa974f710 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.132+0000 7fc57259c640 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa974f710 0x55caa9751b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57259c640 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa974f710 0x55caa9751b30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7c132fda2f0a6c2d server_cookie=6715e105fe7acfb8 in_seq=0 out_seq=0 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc559ffb640 10 client.16935 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:26:32.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc559ffb640 1 -- 172.21.15.116:0/853550932 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55caa93c1890 con 0x55caa974f710 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc559ffb640 10 client.16935 handle_command_reply: tid=0 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 client.16935 shutdown 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 msgr2=0x55caa96d2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d2e50 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc56c006010 tx=0x7fc56c02a630 comp rx=0 tx=0).stop 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 shutdown_connections 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55caa96d88f0 0x55caa96d2e50 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55caa974f710 0x55caa9751b30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55caa96cfd10 0x55caa96d2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc559ffb640 0 client.16935 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 --2- 172.21.15.116:0/853550932 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55caa96cf3f0 0x55caa96d23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 >> 172.21.15.116:0/853550932 conn(0x55caa96c9b20 msgr2=0x55caa96cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 shutdown_connections 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 1 -- 172.21.15.116:0/853550932 wait complete. 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.133+0000 7fc57b7b8740 20 client.16935 trim_cache size 0 max 16384 2024-03-24T09:26:32.145 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:26:32.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc55c0a2230 msgr2=0x7fc55c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc55c0a2230 0x7fc55c0a46f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc56c0049d0 tx=0x7fc56c004ab0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 msgr2=0x7fc57419d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 0x7fc57419d920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fc568009230 tx=0x7fc568005dd0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 shutdown_connections 2024-03-24T09:26:32.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc574107690 0x7fc574197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc55c0a2230 0x7fc55c0a46f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5740692b0 0x7fc57419de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 --2- 172.21.15.116:0/1009053235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5740688f0 0x7fc57419d920 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 >> 172.21.15.116:0/1009053235 conn(0x7fc574077710 msgr2=0x7fc5740fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 shutdown_connections 2024-03-24T09:26:32.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.134+0000 7fc578bd7640 1 -- 172.21.15.116:0/1009053235 wait complete. 2024-03-24T09:26:32.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 -- 172.21.15.116:0/632618391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c1005a0 msgr2=0x7f6e8c100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 --2- 172.21.15.116:0/632618391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c100a20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009900 tx=0x7f6e8802bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 -- 172.21.15.116:0/632618391 shutdown_connections 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 --2- 172.21.15.116:0/632618391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c100f60 0x7f6e8c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 --2- 172.21.15.116:0/632618391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c100a20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 --2- 172.21.15.116:0/632618391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e8c106660 0x7f6e8c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 -- 172.21.15.116:0/632618391 >> 172.21.15.116:0/632618391 conn(0x7f6e8c077710 msgr2=0x7f6e8c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 -- 172.21.15.116:0/632618391 shutdown_connections 2024-03-24T09:26:32.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.251+0000 7f6e93664640 1 -- 172.21.15.116:0/632618391 wait complete. 2024-03-24T09:26:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 Processor -- start 2024-03-24T09:26:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 -- start start 2024-03-24T09:26:32.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c1989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 0x7f6e8c198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e8c106660 0x7f6e8c1a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c199790 con 0x7f6e8c100f60 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c1a3fb0 con 0x7f6e8c1005a0 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e93664640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c1a40f0 con 0x7f6e8c106660 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 0x7f6e8c198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 0x7f6e8c198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54438/0 (socket says 172.21.15.116:54438) 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 -- 172.21.15.116:0/3681423402 learned_addr learned my addr 172.21.15.116:0/3681423402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e91400640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c1989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e91c01640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e8c106660 0x7f6e8c1a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 -- 172.21.15.116:0/3681423402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e8c106660 msgr2=0x7f6e8c1a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e8c106660 0x7f6e8c1a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 -- 172.21.15.116:0/3681423402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c1005a0 msgr2=0x7f6e8c1989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c1989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e90bff640 1 -- 172.21.15.116:0/3681423402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c009490 con 0x7f6e8c100f60 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.252+0000 7f6e91400640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c1989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.253+0000 7f6e90bff640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 0x7f6e8c198f30 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6e88005940 tx=0x7f6e88004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.253+0000 7f6e827fc640 1 -- 172.21.15.116:0/3681423402 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88004440 con 0x7f6e8c100f60 2024-03-24T09:26:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.253+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e88009560 con 0x7f6e8c100f60 2024-03-24T09:26:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.253+0000 7f6e827fc640 1 -- 172.21.15.116:0/3681423402 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e88003aa0 con 0x7f6e8c100f60 2024-03-24T09:26:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.253+0000 7f6e827fc640 1 -- 172.21.15.116:0/3681423402 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88005460 con 0x7f6e8c100f60 2024-03-24T09:26:32.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.253+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8c1a46c0 con 0x7f6e8c100f60 2024-03-24T09:26:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.255+0000 7f6e827fc640 1 -- 172.21.15.116:0/3681423402 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6e88003c10 con 0x7f6e8c100f60 2024-03-24T09:26:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.255+0000 7f6e96244740 1 Processor -- start 2024-03-24T09:26:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.255+0000 7f6e96244740 1 -- start start 2024-03-24T09:26:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e96244740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656752a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e96244740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x5565675333b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e96244740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x556567535ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e96244740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556567498840 con 0x5565675338f0 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e96244740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556567495d70 con 0x55656752a3f0 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e96244740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556567491550 con 0x55656752ad10 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x556567535ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91c01640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656752a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x556567535ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54452/0 (socket says 172.21.15.116:54452) 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91c01640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656752a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48696/0 (socket says 172.21.15.116:48696) 2024-03-24T09:26:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e90bff640 1 -- 172.21.15.116:0/3726529805 learned_addr learned my addr 172.21.15.116:0/3726529805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e827fc640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6e780a2230 0x7f6e780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x5565675333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91c01640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6e780a2230 0x7f6e780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 -- 172.21.15.116:0/3726529805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 msgr2=0x55656752a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656752a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 -- 172.21.15.116:0/3726529805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 msgr2=0x556567535ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x556567535ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e827fc640 1 -- 172.21.15.116:0/3681423402 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6e880f04c0 con 0x7f6e8c100f60 2024-03-24T09:26:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 -- 172.21.15.116:0/3726529805 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556567426de0 con 0x55656752ad10 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91400640 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x5565675333b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c011520 tx=0x7f6e7c0119e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bd0d3a306ac954 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e91c01640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6e780a2230 0x7f6e780a46f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e84009fd0 tx=0x7f6e84004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e777fe640 1 -- 172.21.15.116:0/3726529805 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c004020 con 0x55656752ad10 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e777fe640 1 -- 172.21.15.116:0/3726529805 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e7c00f6a0 con 0x55656752ad10 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.256+0000 7f6e777fe640 1 -- 172.21.15.116:0/3726529805 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c003810 con 0x55656752ad10 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 -- 172.21.15.116:0/3726529805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 msgr2=0x5565675333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x5565675333b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c011520 tx=0x7f6e7c0119e0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 -- 172.21.15.116:0/3726529805 shutdown_connections 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x556567535ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x5565675333b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 --2- 172.21.15.116:0/3726529805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656752a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 -- 172.21.15.116:0/3726529805 >> 172.21.15.116:0/3726529805 conn(0x556567524b20 msgr2=0x556567524f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 -- 172.21.15.116:0/3726529805 shutdown_connections 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.257+0000 7f6e96244740 1 -- 172.21.15.116:0/3726529805 wait complete. 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 -- start start 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 Processor -- start 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752a3f0 0x556567530950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567530e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5565675338f0 0x5565675313d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556567164430 con 0x55656752ad10 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556567120e00 con 0x5565675338f0 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556567416590 con 0x55656752a3f0 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e96244740 10 client.0 _ll_register_callbacks cb 0x556567485fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567530e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e91400640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752a3f0 0x556567530950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567530e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54466/0 (socket says 172.21.15.116:54466) 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 -- 172.21.15.116:0/1475712305 learned_addr learned my addr 172.21.15.116:0/1475712305 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e91c01640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5565675338f0 0x5565675313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 -- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752a3f0 msgr2=0x556567530950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752a3f0 0x556567530950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 -- 172.21.15.116:0/1475712305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5565675338f0 msgr2=0x5565675313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5565675338f0 0x5565675313d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e90bff640 1 -- 172.21.15.116:0/1475712305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7c009d90 con 0x55656752ad10 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e91400640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752a3f0 0x556567530950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.258+0000 7f6e91c01640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5565675338f0 0x5565675313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e90bff640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567530e90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6e8802cd60 tx=0x7f6e880bb1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429b98d56030777d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e74ff9640 1 -- 172.21.15.116:0/1475712305 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880bd3b0 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e74ff9640 1 -- 172.21.15.116:0/1475712305 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6e880bccb0 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5565671907e0 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e74ff9640 1 -- 172.21.15.116:0/1475712305 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e880c0e00 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 10 client.16947 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55656748fbd0 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e74ff9640 1 -- 172.21.15.116:0/1475712305 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5565671907e0 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e74ff9640 1 -- 172.21.15.116:0/1475712305 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f6e8803c070 con 0x55656752ad10 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 10 client.16947 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 10 client.16947 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:26:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 20 client.16947 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556567543820 0x556567545c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 4 client.16947 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565671cd720 con 0x556567543820 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.259+0000 7f6e91400640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556567543820 0x556567545c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.260+0000 7f6e91400640 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556567543820 0x556567545c10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7ceddf47d630fe8 server_cookie=8e6d20b5034a70f5 in_seq=0 out_seq=0 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.260+0000 7f6e74ff9640 10 client.16947 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e74ff9640 1 -- 172.21.15.116:0/1475712305 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565671cd720 con 0x556567543820 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e74ff9640 10 client.16947 handle_command_reply: tid=0 2024-03-24T09:26:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 client.16947 shutdown 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 msgr2=0x556567530e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567530e90 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f6e8802cd60 tx=0x7f6e880bb1c0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 shutdown_connections 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556567543820 0x556567545c10 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5565675338f0 0x5565675313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e74ff9640 0 client.16947 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567530e90 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 --2- 172.21.15.116:0/1475712305 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752a3f0 0x556567530950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 >> 172.21.15.116:0/1475712305 conn(0x556567524b20 msgr2=0x556567526870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 shutdown_connections 2024-03-24T09:26:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 1 -- 172.21.15.116:0/1475712305 wait complete. 2024-03-24T09:26:32.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.261+0000 7f6e96244740 20 client.16947 trim_cache size 0 max 16384 2024-03-24T09:26:32.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:31 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/686104928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 Processor -- start 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 -- start start 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x556567523b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x55656754b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556567546270 con 0x5565675338f0 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55656752eea0 con 0x55656752a3f0 2024-03-24T09:26:32.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e96244740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556567531920 con 0x55656752ad10 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e91c01640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x55656754b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e91400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.274+0000 7f6e91c01640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x55656754b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54478/0 (socket says 172.21.15.116:54478) 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 -- 172.21.15.116:0/473786041 learned_addr learned my addr 172.21.15.116:0/473786041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e90bff640 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x556567523b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 -- 172.21.15.116:0/473786041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 msgr2=0x556567523b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x556567523b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 -- 172.21.15.116:0/473786041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 msgr2=0x55656753f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 -- 172.21.15.116:0/473786041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55656742bc80 con 0x5565675338f0 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91400640 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e91c01640 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x55656754b4a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6e84012610 tx=0x7f6e84009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dda89747582b973 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e76ffd640 1 -- 172.21.15.116:0/473786041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e840235c0 con 0x5565675338f0 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e76ffd640 1 -- 172.21.15.116:0/473786041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e84023bd0 con 0x5565675338f0 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.275+0000 7f6e76ffd640 1 -- 172.21.15.116:0/473786041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e84010a50 con 0x5565675338f0 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- 172.21.15.116:0/473786041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 msgr2=0x55656754b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x55656754b4a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6e84012610 tx=0x7f6e84009920 comp rx=0 tx=0).stop 2024-03-24T09:26:32.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- 172.21.15.116:0/473786041 shutdown_connections 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5565675338f0 0x55656754b4a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55656752ad10 0x556567523b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- 172.21.15.116:0/473786041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- 172.21.15.116:0/473786041 >> 172.21.15.116:0/473786041 conn(0x556567524b20 msgr2=0x556567523430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- 172.21.15.116:0/473786041 shutdown_connections 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- 172.21.15.116:0/473786041 wait complete. 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- start start 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 Processor -- start 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567536d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5565675338f0 0x5565675372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5565675377f0 con 0x55656752ad10 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.276+0000 7f6e96244740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556567537960 con 0x55656752a3f0 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e96244740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556567537b20 con 0x5565675338f0 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e96244740 10 client.0 _ll_register_callbacks cb 0x7f6e880ed9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567536d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e91400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567536d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54486/0 (socket says 172.21.15.116:54486) 2024-03-24T09:26:32.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 -- 172.21.15.116:0/2991688684 learned_addr learned my addr 172.21.15.116:0/2991688684 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e91c01640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5565675338f0 0x5565675372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 -- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5565675338f0 msgr2=0x5565675372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5565675338f0 0x5565675372b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 -- 172.21.15.116:0/2991688684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 msgr2=0x55656753b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 -- 172.21.15.116:0/2991688684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556567426de0 con 0x55656752ad10 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e91400640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e90bff640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567536d70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x55656752a0d0 tx=0x7f6e880bc610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1c8fea33ea6d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e74ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e74ff9640 1 -- 172.21.15.116:0/2991688684 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8802ce70 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e74ff9640 1 -- 172.21.15.116:0/2991688684 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f6e88004dc0 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e7c00f6a0 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e74ff9640 1 -- 172.21.15.116:0/2991688684 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6e880bc7f0 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e96244740 10 client.16953 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f6e8401a040 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.277+0000 7f6e74ff9640 1 -- 172.21.15.116:0/2991688684 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6e7c00f6a0 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e74ff9640 1 -- 172.21.15.116:0/2991688684 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f6e8803c070 con 0x55656752ad10 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 10 client.16953 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:32.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 10 client.16953 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 20 client.16953 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5565675a9c10 0x5565675ac000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 4 client.16953 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55656721c890 con 0x5565675a9c10 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e91400640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5565675a9c10 0x5565675ac000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e91400640 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5565675a9c10 0x5565675ac000 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0a26b3ecbd751a6 server_cookie=bfe6b48287128c4 in_seq=0 out_seq=0 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e74ff9640 10 client.16953 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e74ff9640 1 -- 172.21.15.116:0/2991688684 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55656721c890 con 0x5565675a9c10 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e74ff9640 10 client.16953 handle_command_reply: tid=0 2024-03-24T09:26:32.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 client.16953 shutdown 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 msgr2=0x556567536d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567536d70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x55656752a0d0 tx=0x7f6e880bc610 comp rx=0 tx=0).stop 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 shutdown_connections 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5565675a9c10 0x5565675ac000 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5565675338f0 0x5565675372b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55656752ad10 0x556567536d70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 --2- 172.21.15.116:0/2991688684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55656752a3f0 0x55656753b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e74ff9640 0 client.16953 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.278+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 >> 172.21.15.116:0/2991688684 conn(0x556567524b20 msgr2=0x556567531180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 shutdown_connections 2024-03-24T09:26:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e96244740 1 -- 172.21.15.116:0/2991688684 wait complete. 2024-03-24T09:26:32.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e96244740 20 client.16953 trim_cache size 0 max 16384 2024-03-24T09:26:32.290 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:26:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6e780a2230 msgr2=0x7f6e780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6e780a2230 0x7f6e780a46f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6e84009fd0 tx=0x7f6e84004ab0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 msgr2=0x7f6e8c198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 0x7f6e8c198f30 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6e88005940 tx=0x7f6e88004040 comp rx=0 tx=0).stop 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 shutdown_connections 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6e780a2230 0x7f6e780a46f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6e8c106660 0x7f6e8c1a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8c100f60 0x7f6e8c198f30 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 --2- 172.21.15.116:0/3681423402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e8c1005a0 0x7f6e8c1989f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 >> 172.21.15.116:0/3681423402 conn(0x7f6e8c077710 msgr2=0x7f6e8c0ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 shutdown_connections 2024-03-24T09:26:32.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.279+0000 7f6e93664640 1 -- 172.21.15.116:0/3681423402 wait complete. 2024-03-24T09:26:32.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:26:32.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:31 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/686104928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:26:32.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 -- 172.21.15.116:0/3814535347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 msgr2=0x7f8690100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 --2- 172.21.15.116:0/3814535347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f8690100ab0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8684009930 tx=0x7f868402bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 -- 172.21.15.116:0/3814535347 shutdown_connections 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 --2- 172.21.15.116:0/3814535347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8690100ff0 0x7f869010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 --2- 172.21.15.116:0/3814535347 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f8690100ab0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 --2- 172.21.15.116:0/3814535347 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 0x7f8690106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 -- 172.21.15.116:0/3814535347 >> 172.21.15.116:0/3814535347 conn(0x7f8690077770 msgr2=0x7f8690077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 -- 172.21.15.116:0/3814535347 shutdown_connections 2024-03-24T09:26:32.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 -- 172.21.15.116:0/3814535347 wait complete. 2024-03-24T09:26:32.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 Processor -- start 2024-03-24T09:26:32.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.396+0000 7f8697581640 1 -- start start 2024-03-24T09:26:32.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f869019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8690100ff0 0x7f869019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 0x7f86901969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8690197040 con 0x7f8690100630 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f86901971b0 con 0x7f8690100ff0 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8690197370 con 0x7f8690106640 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f869019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f869019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54506/0 (socket says 172.21.15.116:54506) 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 -- 172.21.15.116:0/3875333354 learned_addr learned my addr 172.21.15.116:0/3875333354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8694b1c640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8690100ff0 0x7f869019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8695b1e640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 0x7f86901969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 -- 172.21.15.116:0/3875333354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 msgr2=0x7f86901969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 0x7f86901969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 -- 172.21.15.116:0/3875333354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8690100ff0 msgr2=0x7f869019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8690100ff0 0x7f869019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 -- 172.21.15.116:0/3875333354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8688009490 con 0x7f8690100630 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8695b1e640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 0x7f86901969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f869531d640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f869019c840 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f868800ea60 tx=0x7f868800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f868e7fc640 1 -- 172.21.15.116:0/3875333354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868800cde0 con 0x7f8690100630 2024-03-24T09:26:32.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f868e7fc640 1 -- 172.21.15.116:0/3875333354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86880044d0 con 0x7f8690100630 2024-03-24T09:26:32.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8684009560 con 0x7f8690100630 2024-03-24T09:26:32.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f868e7fc640 1 -- 172.21.15.116:0/3875333354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8688010690 con 0x7f8690100630 2024-03-24T09:26:32.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.397+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86901a3700 con 0x7f8690100630 2024-03-24T09:26:32.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.399+0000 7f868e7fc640 1 -- 172.21.15.116:0/3875333354 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f8688003e40 con 0x7f8690100630 2024-03-24T09:26:32.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 Processor -- start 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 -- start start 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec2c3f0 0x55af1ec2c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec353b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec358f0 0x55af1ec37ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af1eb9a840 con 0x55af1ec2cd10 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af1eb97d70 con 0x55af1ec2c3f0 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8699dfd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55af1eb93550 con 0x55af1ec358f0 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f868e7fc640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86780a2230 0x7f86780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f869531d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec353b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f869531d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec353b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54518/0 (socket says 172.21.15.116:54518) 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f869531d640 1 -- 172.21.15.116:0/327209333 learned_addr learned my addr 172.21.15.116:0/327209333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8694b1c640 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec358f0 0x55af1ec37ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.400+0000 7f8695b1e640 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec2c3f0 0x55af1ec2c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8695b1e640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86780a2230 0x7f86780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8694b1c640 1 -- 172.21.15.116:0/327209333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec2c3f0 msgr2=0x55af1ec2c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f868e7fc640 1 -- 172.21.15.116:0/3875333354 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8688014070 con 0x7f8690100630 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8694b1c640 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec2c3f0 0x55af1ec2c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8694b1c640 1 -- 172.21.15.116:0/327209333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 msgr2=0x55af1ec353b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8694b1c640 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec353b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8694b1c640 1 -- 172.21.15.116:0/327209333 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1eb28de0 con 0x55af1ec358f0 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8695b1e640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86780a2230 0x7f86780a46f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8680005c40 tx=0x7f8680009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f8694b1c640 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec358f0 0x55af1ec37ce0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8684009230 tx=0x7f8684005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8de9ff10c32b8961 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f86777fe640 1 -- 172.21.15.116:0/327209333 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684004800 con 0x55af1ec358f0 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f86777fe640 1 -- 172.21.15.116:0/327209333 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f868402de90 con 0x55af1ec358f0 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.401+0000 7f86777fe640 1 -- 172.21.15.116:0/327209333 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86840035f0 con 0x55af1ec358f0 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 -- 172.21.15.116:0/327209333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec358f0 msgr2=0x55af1ec37ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec358f0 0x55af1ec37ce0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8684009230 tx=0x7f8684005d50 comp rx=0 tx=0).stop 2024-03-24T09:26:32.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 -- 172.21.15.116:0/327209333 shutdown_connections 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec358f0 0x55af1ec37ce0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec353b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 --2- 172.21.15.116:0/327209333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec2c3f0 0x55af1ec2c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 -- 172.21.15.116:0/327209333 >> 172.21.15.116:0/327209333 conn(0x55af1ec26b20 msgr2=0x55af1ec26f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 -- 172.21.15.116:0/327209333 shutdown_connections 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 -- 172.21.15.116:0/327209333 wait complete. 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 -- start start 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.402+0000 7f8699dfd740 1 Processor -- start 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 0x55af1ec328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2cd10 0x55af1ec32e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec33370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af1e866430 con 0x55af1ec2c3f0 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af1e822e00 con 0x55af1ec358f0 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55af1eb18590 con 0x55af1ec2cd10 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 10 client.0 _ll_register_callbacks cb 0x55af1eb87fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 0x55af1ec328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 0x55af1ec328f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54524/0 (socket says 172.21.15.116:54524) 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 -- 172.21.15.116:0/3717469431 learned_addr learned my addr 172.21.15.116:0/3717469431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8695b1e640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec33370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8694b1c640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2cd10 0x55af1ec32e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 -- 172.21.15.116:0/3717469431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2cd10 msgr2=0x55af1ec32e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2cd10 0x55af1ec32e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 -- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 msgr2=0x55af1ec33370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec33370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 -- 172.21.15.116:0/3717469431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f869010de00 con 0x55af1ec2c3f0 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f869531d640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 0x55af1ec328f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f868801f000 tx=0x7f8688096ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e735e7e76b5c59b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8674ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8674ff9640 1 -- 172.21.15.116:0/3717469431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868809b650 con 0x55af1ec2c3f0 2024-03-24T09:26:32.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8674ff9640 1 -- 172.21.15.116:0/3717469431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f868809af50 con 0x55af1ec2c3f0 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.403+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55af1e8927e0 con 0x55af1ec2c3f0 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8674ff9640 1 -- 172.21.15.116:0/3717469431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f868809f0a0 con 0x55af1ec2c3f0 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 10 client.16965 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55af1eb91bd0 con 0x55af1ec2c3f0 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8674ff9640 1 -- 172.21.15.116:0/3717469431 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8688064020 con 0x55af1ec2c3f0 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8674ff9640 1 -- 172.21.15.116:0/3717469431 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f86880a3260 con 0x55af1ec2c3f0 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 10 client.16965 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 10 client.16965 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 20 client.16965 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:32.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ec459b0 0x55af1ec47dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 4 client.16965 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55af1e8cf720 con 0x55af1ec459b0 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8694b1c640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ec459b0 0x55af1ec47dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8674ff9640 10 client.16965 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.404+0000 7f8694b1c640 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ec459b0 0x55af1ec47dd0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b9118f357b680e0 server_cookie=cb60067a6d15f003 in_seq=0 out_seq=0 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8674ff9640 1 -- 172.21.15.116:0/3717469431 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55af1e8cf720 con 0x55af1ec459b0 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8674ff9640 10 client.16965 handle_command_reply: tid=0 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 client.16965 shutdown 2024-03-24T09:26:32.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 msgr2=0x55af1ec328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 0x55af1ec328f0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f868801f000 tx=0x7f8688096ad0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 shutdown_connections 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ec459b0 0x55af1ec47dd0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec33370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2cd10 0x55af1ec32e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 --2- 172.21.15.116:0/3717469431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2c3f0 0x55af1ec328f0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8674ff9640 0 client.16965 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 >> 172.21.15.116:0/3717469431 conn(0x55af1ec26b20 msgr2=0x55af1ec28870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 shutdown_connections 2024-03-24T09:26:32.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 1 -- 172.21.15.116:0/3717469431 wait complete. 2024-03-24T09:26:32.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.405+0000 7f8699dfd740 20 client.16965 trim_cache size 0 max 16384 2024-03-24T09:26:32.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.418+0000 7f8699dfd740 1 Processor -- start 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.418+0000 7f8699dfd740 1 -- start start 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec21b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec41120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec4d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8699dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af1ec44110 con 0x55af1ec2cd10 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8699dfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af1ec2d790 con 0x55af1ec358f0 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8699dfd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55af1ec22090 con 0x55af1ec2c3f0 2024-03-24T09:26:32.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec41120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec41120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54536/0 (socket says 172.21.15.116:54536) 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 -- 172.21.15.116:0/201171934 learned_addr learned my addr 172.21.15.116:0/201171934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8695b1e640 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec4d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f869531d640 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec21b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 -- 172.21.15.116:0/201171934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 msgr2=0x55af1ec21b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec21b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 -- 172.21.15.116:0/201171934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 msgr2=0x55af1ec4d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec4d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 -- 172.21.15.116:0/201171934 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55af1eb2dc80 con 0x55af1ec2cd10 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8694b1c640 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec41120 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8684009230 tx=0x7f8684000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e0c6b698aca299 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8676ffd640 1 -- 172.21.15.116:0/201171934 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684053d80 con 0x55af1ec2cd10 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8676ffd640 1 -- 172.21.15.116:0/201171934 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86840044e0 con 0x55af1ec2cd10 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.419+0000 7f8676ffd640 1 -- 172.21.15.116:0/201171934 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86840533b0 con 0x55af1ec2cd10 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 -- 172.21.15.116:0/201171934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 msgr2=0x55af1ec41120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec41120 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8684009230 tx=0x7f8684000bc0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 -- 172.21.15.116:0/201171934 shutdown_connections 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ec4d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec41120 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 --2- 172.21.15.116:0/201171934 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec21b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 -- 172.21.15.116:0/201171934 >> 172.21.15.116:0/201171934 conn(0x55af1ec26b20 msgr2=0x55af1ec25cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 -- 172.21.15.116:0/201171934 shutdown_connections 2024-03-24T09:26:32.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 -- 172.21.15.116:0/201171934 wait complete. 2024-03-24T09:26:32.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 -- start start 2024-03-24T09:26:32.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.420+0000 7f8699dfd740 1 Processor -- start 2024-03-24T09:26:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec393e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec39920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ecaad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55af1ec3a1f0 con 0x55af1ec2cd10 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55af1ecab250 con 0x55af1ec358f0 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55af1ecab390 con 0x55af1ec2c3f0 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 10 client.0 _ll_register_callbacks cb 0x7f8688003b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec39920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec39920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54546/0 (socket says 172.21.15.116:54546) 2024-03-24T09:26:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 -- 172.21.15.116:0/2542188517 learned_addr learned my addr 172.21.15.116:0/2542188517 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8695b1e640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ecaad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f869531d640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 -- 172.21.15.116:0/2542188517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 msgr2=0x55af1ec393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec393e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 -- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 msgr2=0x55af1ecaad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ecaad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 -- 172.21.15.116:0/2542188517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55af1eb28de0 con 0x55af1ec2cd10 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8694b1c640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec39920 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f868402c710 tx=0x7f8684005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f15b124b5eb95a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8674ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8674ff9640 1 -- 172.21.15.116:0/2542188517 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684053d80 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8674ff9640 1 -- 172.21.15.116:0/2542188517 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f868404e040 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.421+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f868402de90 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8674ff9640 1 -- 172.21.15.116:0/2542188517 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f868402c770 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 10 client.16971 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55af1eb53920 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8674ff9640 1 -- 172.21.15.116:0/2542188517 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f868406b020 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8674ff9640 1 -- 172.21.15.116:0/2542188517 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f8684052070 con 0x55af1ec2cd10 2024-03-24T09:26:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 10 client.16971 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 10 client.16971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 20 client.16971 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ecac8d0 0x55af1ecaecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 4 client.16971 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55af1e91e890 con 0x55af1ecac8d0 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f869531d640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ecac8d0 0x55af1ecaecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f869531d640 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ecac8d0 0x55af1ecaecf0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf0e5e3fb46521bb server_cookie=c3af0991a28191c5 in_seq=0 out_seq=0 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.422+0000 7f8674ff9640 10 client.16971 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8674ff9640 1 -- 172.21.15.116:0/2542188517 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55af1e91e890 con 0x55af1ecac8d0 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8674ff9640 10 client.16971 handle_command_reply: tid=0 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 client.16971 shutdown 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 msgr2=0x55af1ec39920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec39920 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f868402c710 tx=0x7f8684005dd0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 shutdown_connections 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55af1ecac8d0 0x55af1ecaecf0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55af1ec358f0 0x55af1ecaad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55af1ec2cd10 0x55af1ec39920 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8674ff9640 0 client.16971 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 --2- 172.21.15.116:0/2542188517 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55af1ec2c3f0 0x55af1ec393e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 >> 172.21.15.116:0/2542188517 conn(0x55af1ec26b20 msgr2=0x55af1ec27c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 shutdown_connections 2024-03-24T09:26:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 1 -- 172.21.15.116:0/2542188517 wait complete. 2024-03-24T09:26:32.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8699dfd740 20 client.16971 trim_cache size 0 max 16384 2024-03-24T09:26:32.434 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:26:32.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86780a2230 msgr2=0x7f86780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86780a2230 0x7f86780a46f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8680005c40 tx=0x7f8680009500 comp rx=0 tx=0).stop 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 msgr2=0x7f869019c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f869019c840 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f868800ea60 tx=0x7f868800ef20 comp rx=0 tx=0).stop 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 shutdown_connections 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86780a2230 0x7f86780a46f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8690106640 0x7f86901969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8690100ff0 0x7f869019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 --2- 172.21.15.116:0/3875333354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8690100630 0x7f869019c840 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.423+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 >> 172.21.15.116:0/3875333354 conn(0x7f8690077770 msgr2=0x7f86901032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.424+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 shutdown_connections 2024-03-24T09:26:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:32.424+0000 7f8697581640 1 -- 172.21.15.116:0/3875333354 wait complete. 2024-03-24T09:26:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:32 smithi143 ceph-mon[35965]: pgmap v136: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:32 smithi117 ceph-mon[35788]: pgmap v136: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:32 smithi116 ceph-mon[32207]: pgmap v136: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:33.438 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:26:33.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:26:33.438 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:26:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 -- 172.21.15.116:0/1896382035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4210107690 msgr2=0x7f4210107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 --2- 172.21.15.116:0/1896382035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4210107690 0x7f4210107a70 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4208009930 tx=0x7f420802bc30 comp rx=0 tx=0).stop 2024-03-24T09:26:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 -- 172.21.15.116:0/1896382035 shutdown_connections 2024-03-24T09:26:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 --2- 172.21.15.116:0/1896382035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42100692b0 0x7f421010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 --2- 172.21.15.116:0/1896382035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42100688f0 0x7f4210068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 --2- 172.21.15.116:0/1896382035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4210107690 0x7f4210107a70 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 -- 172.21.15.116:0/1896382035 >> 172.21.15.116:0/1896382035 conn(0x7f4210077710 msgr2=0x7f4210077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 -- 172.21.15.116:0/1896382035 shutdown_connections 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.554+0000 7f421808f640 1 -- 172.21.15.116:0/1896382035 wait complete. 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 Processor -- start 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 -- start start 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 0x7f42101005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42100692b0 0x7f4210100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4210107690 0x7f4210101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42101a6040 con 0x7f42100688f0 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42101a61b0 con 0x7f4210107690 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421808f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f42101a6370 con 0x7f42100692b0 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 0x7f42101005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421662c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4210107690 0x7f4210101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 0x7f42101005d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54562/0 (socket says 172.21.15.116:54562) 2024-03-24T09:26:33.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 -- 172.21.15.116:0/2254508087 learned_addr learned my addr 172.21.15.116:0/2254508087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421662c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4210107690 0x7f4210101050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48812/0 (socket says 172.21.15.116:48812) 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f421562a640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42100692b0 0x7f4210100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 -- 172.21.15.116:0/2254508087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42100692b0 msgr2=0x7f4210100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42100692b0 0x7f4210100b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 -- 172.21.15.116:0/2254508087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4210107690 msgr2=0x7f4210101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4210107690 0x7f4210101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 -- 172.21.15.116:0/2254508087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4208009560 con 0x7f42100688f0 2024-03-24T09:26:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.555+0000 7f4215e2b640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 0x7f42101005d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4208009230 tx=0x7f4208005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.556+0000 7f420effd640 1 -- 172.21.15.116:0/2254508087 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208004350 con 0x7f42100688f0 2024-03-24T09:26:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.556+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42101a65f0 con 0x7f42100688f0 2024-03-24T09:26:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.556+0000 7f420effd640 1 -- 172.21.15.116:0/2254508087 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f420802de90 con 0x7f42100688f0 2024-03-24T09:26:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.556+0000 7f420effd640 1 -- 172.21.15.116:0/2254508087 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208005220 con 0x7f42100688f0 2024-03-24T09:26:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.556+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42101a6b00 con 0x7f42100688f0 2024-03-24T09:26:33.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.558+0000 7f420effd640 1 -- 172.21.15.116:0/2254508087 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f4208003c30 con 0x7f42100688f0 2024-03-24T09:26:33.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.558+0000 7f421ac75740 1 Processor -- start 2024-03-24T09:26:33.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.558+0000 7f420effd640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41f80a2230 0x7f41f80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.558+0000 7f421662c640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41f80a2230 0x7f41f80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.558+0000 7f420effd640 1 -- 172.21.15.116:0/2254508087 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f42080f02b0 con 0x7f42100688f0 2024-03-24T09:26:33.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.558+0000 7f421ac75740 1 -- start start 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421ac75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc25eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421ac75740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc267720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421ac75740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc26a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421ac75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556cfc1cc530 con 0x556cfc25e760 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421ac75740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556cfc1c9a60 con 0x556cfc25f080 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421ac75740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556cfc1c5240 con 0x556cfc267c60 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc25eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421562a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc267720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc25eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54574/0 (socket says 172.21.15.116:54574) 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 -- 172.21.15.116:0/71755425 learned_addr learned my addr 172.21.15.116:0/71755425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421662c640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41f80a2230 0x7f41f80a46f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4200009b30 tx=0x7f4200005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421662c640 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc26a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 -- 172.21.15.116:0/71755425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 msgr2=0x556cfc26a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc26a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 -- 172.21.15.116:0/71755425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 msgr2=0x556cfc267720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc267720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 -- 172.21.15.116:0/71755425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cfc1c38c0 con 0x556cfc25e760 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f421562a640 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc267720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f4215e2b640 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc25eb40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f42080bc990 tx=0x7f42080bc610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0b8da7eed6a941 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f41f7fff640 1 -- 172.21.15.116:0/71755425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080bca00 con 0x556cfc25e760 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f41f7fff640 1 -- 172.21.15.116:0/71755425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42080c0b50 con 0x556cfc25e760 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.559+0000 7f41f7fff640 1 -- 172.21.15.116:0/71755425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080c4d10 con 0x556cfc25e760 2024-03-24T09:26:33.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 -- 172.21.15.116:0/71755425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 msgr2=0x556cfc25eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc25eb40 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f42080bc990 tx=0x7f42080bc610 comp rx=0 tx=0).stop 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 -- 172.21.15.116:0/71755425 shutdown_connections 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc26a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc267720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 --2- 172.21.15.116:0/71755425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc25eb40 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 -- 172.21.15.116:0/71755425 >> 172.21.15.116:0/71755425 conn(0x556cfc258e90 msgr2=0x556cfc259270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 -- 172.21.15.116:0/71755425 shutdown_connections 2024-03-24T09:26:33.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 -- 172.21.15.116:0/71755425 wait complete. 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.560+0000 7f421ac75740 1 -- start start 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 Processor -- start 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc264d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc265270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc2657b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556cfc169f80 con 0x556cfc25e760 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556cfbe984b0 con 0x556cfc25f080 2024-03-24T09:26:33.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556cfc14aad0 con 0x556cfc267c60 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421ac75740 10 client.0 _ll_register_callbacks cb 0x556cfc17db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc264d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc264d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54576/0 (socket says 172.21.15.116:54576) 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 -- 172.21.15.116:0/1009990177 learned_addr learned my addr 172.21.15.116:0/1009990177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421562a640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc265270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f421662c640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc2657b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 -- 172.21.15.116:0/1009990177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 msgr2=0x556cfc2657b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc2657b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 -- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 msgr2=0x556cfc265270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc265270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.561+0000 7f4215e2b640 1 -- 172.21.15.116:0/1009990177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42101a6b00 con 0x556cfc25e760 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f4215e2b640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc264d30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f42080ed810 tx=0x7f42080ed720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96f1ae3f4e37f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f41f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f41f57fa640 1 -- 172.21.15.116:0/1009990177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080bca00 con 0x556cfc25e760 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f41f57fa640 1 -- 172.21.15.116:0/1009990177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f42080c4f20 con 0x556cfc25e760 2024-03-24T09:26:33.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556cfc1e97a0 con 0x556cfc25e760 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f41f57fa640 1 -- 172.21.15.116:0/1009990177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f42080d5260 con 0x556cfc25e760 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 10 client.16983 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556cfc17b6d0 con 0x556cfc25e760 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f41f57fa640 1 -- 172.21.15.116:0/1009990177 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556cfc1e97a0 con 0x556cfc25e760 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f41f57fa640 1 -- 172.21.15.116:0/1009990177 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f420803c070 con 0x556cfc25e760 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 10 client.16983 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 10 client.16983 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:33.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 20 client.16983 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc277600 0x556cfc279a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 4 client.16983 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cfc1bb1f0 con 0x556cfc277600 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.562+0000 7f421562a640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc277600 0x556cfc279a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.563+0000 7f41f57fa640 10 client.16983 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.563+0000 7f421562a640 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc277600 0x556cfc279a20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f17ba7b51493e328 server_cookie=9841d23e319f294a in_seq=0 out_seq=0 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.563+0000 7f41f57fa640 1 -- 172.21.15.116:0/1009990177 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556cfc1bb1f0 con 0x556cfc277600 2024-03-24T09:26:33.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.563+0000 7f41f57fa640 10 client.16983 handle_command_reply: tid=0 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.563+0000 7f421ac75740 1 client.16983 shutdown 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 msgr2=0x556cfc264d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc264d30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f42080ed810 tx=0x7f42080ed720 comp rx=0 tx=0).stop 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 shutdown_connections 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc2657b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc277600 0x556cfc279a20 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc265270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 --2- 172.21.15.116:0/1009990177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc264d30 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 >> 172.21.15.116:0/1009990177 conn(0x556cfc258e90 msgr2=0x556cfc25abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f41f57fa640 0 client.16983 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 shutdown_connections 2024-03-24T09:26:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 1 -- 172.21.15.116:0/1009990177 wait complete. 2024-03-24T09:26:33.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.564+0000 7f421ac75740 20 client.16983 trim_cache size 0 max 16384 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 Processor -- start 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 -- start start 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc267460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc25f080 0x556cfc2732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc267c60 0x556cfc263eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556cfc257350 con 0x556cfc25e760 2024-03-24T09:26:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556cfc25df00 con 0x556cfc267c60 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.581+0000 7f421ac75740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556cfc2643f0 con 0x556cfc25f080 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc267460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc267460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54588/0 (socket says 172.21.15.116:54588) 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f421662c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc267c60 0x556cfc263eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 -- 172.21.15.116:0/607811853 learned_addr learned my addr 172.21.15.116:0/607811853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f421562a640 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc25f080 0x556cfc2732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 -- 172.21.15.116:0/607811853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc25f080 msgr2=0x556cfc2732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc25f080 0x556cfc2732b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 -- 172.21.15.116:0/607811853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc267c60 msgr2=0x556cfc263eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc267c60 0x556cfc263eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 -- 172.21.15.116:0/607811853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cfc15fc60 con 0x556cfc25e760 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f421662c640 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc267c60 0x556cfc263eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f4215e2b640 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc267460 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f42080c5360 tx=0x7f4208005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f0bd413672b0a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f41f77fe640 1 -- 172.21.15.116:0/607811853 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080c51a0 con 0x556cfc25e760 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f41f77fe640 1 -- 172.21.15.116:0/607811853 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42080bca00 con 0x556cfc25e760 2024-03-24T09:26:33.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.582+0000 7f41f77fe640 1 -- 172.21.15.116:0/607811853 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080055a0 con 0x556cfc25e760 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 -- 172.21.15.116:0/607811853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 msgr2=0x556cfc267460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc267460 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f42080c5360 tx=0x7f4208005fd0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 -- 172.21.15.116:0/607811853 shutdown_connections 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc267c60 0x556cfc263eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc25f080 0x556cfc2732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 --2- 172.21.15.116:0/607811853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc267460 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 -- 172.21.15.116:0/607811853 >> 172.21.15.116:0/607811853 conn(0x556cfc258e90 msgr2=0x556cfc25a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 -- 172.21.15.116:0/607811853 shutdown_connections 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 -- 172.21.15.116:0/607811853 wait complete. 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 -- start start 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.583+0000 7f421ac75740 1 Processor -- start 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc26fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc261600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc261b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556cfc262240 con 0x556cfc25e760 2024-03-24T09:26:33.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556cfc2623b0 con 0x556cfc25f080 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556cfc2dc960 con 0x556cfc267c60 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 10 client.0 _ll_register_callbacks cb 0x7f42080bc880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc26fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc26fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54602/0 (socket says 172.21.15.116:54602) 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 -- 172.21.15.116:0/3240154533 learned_addr learned my addr 172.21.15.116:0/3240154533 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421562a640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc261600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421662c640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc261b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 -- 172.21.15.116:0/3240154533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 msgr2=0x556cfc261b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc261b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 -- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 msgr2=0x556cfc261600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc261600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 -- 172.21.15.116:0/3240154533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4200017070 con 0x556cfc25e760 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421562a640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc261600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f4215e2b640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc26fbb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f42080b8e10 tx=0x7f42080b9070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448142e9f29e6fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f41f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f41f57fa640 1 -- 172.21.15.116:0/3240154533 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208005580 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f41f57fa640 1 -- 172.21.15.116:0/3240154533 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f42080c51a0 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42080c0b50 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f41f57fa640 1 -- 172.21.15.116:0/3240154533 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f42080b8820 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 10 client.16989 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.584+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f42101a6b00 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f41f57fa640 1 -- 172.21.15.116:0/3240154533 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f42080c0b50 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f41f57fa640 1 -- 172.21.15.116:0/3240154533 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f420803c070 con 0x556cfc25e760 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421ac75740 10 client.16989 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421ac75740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421ac75740 20 client.16989 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421ac75740 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc2ddcc0 0x556cfc2e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421ac75740 4 client.16989 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x556cfbec4860 con 0x556cfc2ddcc0 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421562a640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc2ddcc0 0x556cfc2e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f421562a640 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc2ddcc0 0x556cfc2e00b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283981eb0e1b7b7d server_cookie=f67a0eab34cdcf09 in_seq=0 out_seq=0 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.585+0000 7f41f57fa640 10 client.16989 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f41f57fa640 1 -- 172.21.15.116:0/3240154533 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x556cfbec4860 con 0x556cfc2ddcc0 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f41f57fa640 10 client.16989 handle_command_reply: tid=0 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 client.16989 shutdown 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 msgr2=0x556cfc26fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc26fbb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f42080b8e10 tx=0x7f42080b9070 comp rx=0 tx=0).stop 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 shutdown_connections 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556cfc267c60 0x556cfc261b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x556cfc2ddcc0 0x556cfc2e00b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556cfc25f080 0x556cfc261600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 --2- 172.21.15.116:0/3240154533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556cfc25e760 0x556cfc26fbb0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f41f57fa640 0 client.16989 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 >> 172.21.15.116:0/3240154533 conn(0x556cfc258e90 msgr2=0x556cfc25aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 shutdown_connections 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 1 -- 172.21.15.116:0/3240154533 wait complete. 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.587+0000 7f421ac75740 20 client.16989 trim_cache size 0 max 16384 2024-03-24T09:26:33.599 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:26:33.600 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:26:33.600 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "4e207943-af53-45c3-8fec-f85d311713d8", 2024-03-24T09:26:33.600 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:26:33.600 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:26:33.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41f80a2230 msgr2=0x7f41f80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41f80a2230 0x7f41f80a46f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4200009b30 tx=0x7f4200005f90 comp rx=0 tx=0).stop 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 msgr2=0x7f42101005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 0x7f42101005d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4208009230 tx=0x7f4208005dd0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 shutdown_connections 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4210107690 0x7f4210101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f41f80a2230 0x7f41f80a46f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42100692b0 0x7f4210100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 --2- 172.21.15.116:0/2254508087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42100688f0 0x7f42101005d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 >> 172.21.15.116:0/2254508087 conn(0x7f4210077710 msgr2=0x7f42100fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 shutdown_connections 2024-03-24T09:26:33.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.588+0000 7f421808f640 1 -- 172.21.15.116:0/2254508087 wait complete. 2024-03-24T09:26:33.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:26:33.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:26:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 -- 172.21.15.116:0/322141247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 msgr2=0x7fdd4010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 --2- 172.21.15.116:0/322141247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 0x7fdd4010d420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009c10 tx=0x7fdd3802bef0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 -- 172.21.15.116:0/322141247 shutdown_connections 2024-03-24T09:26:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 --2- 172.21.15.116:0/322141247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 0x7fdd4010d420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 --2- 172.21.15.116:0/322141247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd401005a0 0x7fdd40100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 --2- 172.21.15.116:0/322141247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd40106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 -- 172.21.15.116:0/322141247 >> 172.21.15.116:0/322141247 conn(0x7fdd40077710 msgr2=0x7fdd40077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.721+0000 7fdd4733a640 1 -- 172.21.15.116:0/322141247 shutdown_connections 2024-03-24T09:26:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 -- 172.21.15.116:0/322141247 wait complete. 2024-03-24T09:26:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 Processor -- start 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 -- start start 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd401005a0 0x7fdd4019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 0x7fdd4019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd401968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40196ef0 con 0x7fdd40106660 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40197060 con 0x7fdd401005a0 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.722+0000 7fdd4733a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40197220 con 0x7fdd40100f60 2024-03-24T09:26:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd401968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd401968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54624/0 (socket says 172.21.15.116:54624) 2024-03-24T09:26:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 -- 172.21.15.116:0/4147428485 learned_addr learned my addr 172.21.15.116:0/4147428485 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd450d6640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd401005a0 0x7fdd4019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd448d5640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 0x7fdd4019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 -- 172.21.15.116:0/4147428485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 msgr2=0x7fdd4019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 0x7fdd4019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 -- 172.21.15.116:0/4147428485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd401005a0 msgr2=0x7fdd4019c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd401005a0 0x7fdd4019c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 -- 172.21.15.116:0/4147428485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd30009490 con 0x7fdd40106660 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd458d7640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd401968a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdd38004af0 tx=0x7fdd38033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd367fc640 1 -- 172.21.15.116:0/4147428485 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38033e20 con 0x7fdd40106660 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd367fc640 1 -- 172.21.15.116:0/4147428485 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd38003b80 con 0x7fdd40106660 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd367fc640 1 -- 172.21.15.116:0/4147428485 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38034690 con 0x7fdd40106660 2024-03-24T09:26:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.723+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd38009840 con 0x7fdd40106660 2024-03-24T09:26:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.724+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd401a35b0 con 0x7fdd40106660 2024-03-24T09:26:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 Processor -- start 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd367fc640 1 -- 172.21.15.116:0/4147428485 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fdd38004040 con 0x7fdd40106660 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 -- start start 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabf3f70 0x55a3fabf6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fab06f30 con 0x55a3fabe1bd0 2024-03-24T09:26:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fab50810 con 0x55a3fabe1fb0 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.727+0000 7fdd49fee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a3fab4dd40 con 0x55a3fabf3f70 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd450d6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd450d6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48856/0 (socket says 172.21.15.116:48856) 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd450d6640 1 -- 172.21.15.116:0/989933417 learned_addr learned my addr 172.21.15.116:0/989933417 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabf3f70 0x55a3fabf6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd367fc640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdd2c0a2230 0x7fdd2c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 -- 172.21.15.116:0/989933417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 msgr2=0x55a3fabe2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 -- 172.21.15.116:0/989933417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 msgr2=0x55a3fabe9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd448d5640 1 -- 172.21.15.116:0/989933417 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3fab0aea0 con 0x55a3fabf3f70 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.728+0000 7fdd367fc640 1 -- 172.21.15.116:0/4147428485 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdd380f1060 con 0x7fdd40106660 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.729+0000 7fdd458d7640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdd2c0a2230 0x7fdd2c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.729+0000 7fdd448d5640 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabf3f70 0x55a3fabf6360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd400fea90 tx=0x7fdd30009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b976d5d7ebca23de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.729+0000 7fdd458d7640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdd2c0a2230 0x7fdd2c0a46f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd380c5a20 tx=0x7fdd380a8040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.729+0000 7fdd2b7fe640 1 -- 172.21.15.116:0/989933417 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30005020 con 0x55a3fabf3f70 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.729+0000 7fdd2b7fe640 1 -- 172.21.15.116:0/989933417 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd30005630 con 0x55a3fabf3f70 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.729+0000 7fdd2b7fe640 1 -- 172.21.15.116:0/989933417 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3000f9f0 con 0x55a3fabf3f70 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 -- 172.21.15.116:0/989933417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabf3f70 msgr2=0x55a3fabf6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabf3f70 0x55a3fabf6360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdd400fea90 tx=0x7fdd30009370 comp rx=0 tx=0).stop 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 -- 172.21.15.116:0/989933417 shutdown_connections 2024-03-24T09:26:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabf3f70 0x55a3fabf6360 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 --2- 172.21.15.116:0/989933417 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 -- 172.21.15.116:0/989933417 >> 172.21.15.116:0/989933417 conn(0x55a3fabdc900 msgr2=0x55a3fabdcce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 -- 172.21.15.116:0/989933417 shutdown_connections 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 -- 172.21.15.116:0/989933417 wait complete. 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.730+0000 7fdd49fee740 1 -- start start 2024-03-24T09:26:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 Processor -- start 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabe71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fab49520 con 0x55a3fabe1fb0 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fa81c430 con 0x55a3fabf3f70 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a3fa7d8e00 con 0x55a3fabe1bd0 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd450d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd49fee740 10 client.0 _ll_register_callbacks cb 0x55a3faae7d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd450d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54644/0 (socket says 172.21.15.116:54644) 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd450d6640 1 -- 172.21.15.116:0/4134852373 learned_addr learned my addr 172.21.15.116:0/4134852373 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd448d5640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.731+0000 7fdd448d5640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabe71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd450d6640 1 -- 172.21.15.116:0/4134852373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 msgr2=0x55a3fabe6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd450d6640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd450d6640 1 -- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 msgr2=0x55a3fabe71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd450d6640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabe71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd450d6640 1 -- 172.21.15.116:0/4134852373 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd30019040 con 0x55a3fabe1fb0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd450d6640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe6cb0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0126e0 tx=0x7fdd3c012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df63257e4ee584b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd28ff9640 1 -- 172.21.15.116:0/4134852373 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c017070 con 0x55a3fabe1fb0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd28ff9640 1 -- 172.21.15.116:0/4134852373 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdd3c0120d0 con 0x55a3fabe1fb0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a3fa8487e0 con 0x55a3fabe1fb0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd28ff9640 1 -- 172.21.15.116:0/4134852373 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdd3c012290 con 0x55a3fabe1fb0 2024-03-24T09:26:33.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd49fee740 10 client.17001 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.732+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3fab47ba0 con 0x55a3fabe1fb0 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd28ff9640 1 -- 172.21.15.116:0/4134852373 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a3fa8487e0 con 0x55a3fabe1fb0 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd28ff9640 1 -- 172.21.15.116:0/4134852373 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fdd3c01cb60 con 0x55a3fabe1fb0 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd49fee740 10 client.17001 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd49fee740 10 client.17001 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd49fee740 20 client.17001 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd49fee740 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac04540 0x55a3fac06960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd49fee740 4 client.17001 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:26:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3fa885720 con 0x55a3fac04540 2024-03-24T09:26:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.733+0000 7fdd448d5640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac04540 0x55a3fac06960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.734+0000 7fdd448d5640 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac04540 0x55a3fac06960 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=812d90552809879e server_cookie=ca199a5f8568dde6 in_seq=0 out_seq=0 2024-03-24T09:26:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.734+0000 7fdd28ff9640 10 client.17001 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd28ff9640 1 -- 172.21.15.116:0/4134852373 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3fa885720 con 0x55a3fac04540 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd28ff9640 10 client.17001 handle_command_reply: tid=0 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 client.17001 shutdown 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 msgr2=0x55a3fabe6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe6cb0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0126e0 tx=0x7fdd3c012ba0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 shutdown_connections 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac04540 0x55a3fac06960 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabe71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1fb0 0x55a3fabe6cb0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd28ff9640 0 client.17001 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 --2- 172.21.15.116:0/4134852373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 >> 172.21.15.116:0/4134852373 conn(0x55a3fabdc900 msgr2=0x55a3fabde650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 shutdown_connections 2024-03-24T09:26:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 1 -- 172.21.15.116:0/4134852373 wait complete. 2024-03-24T09:26:33.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.735+0000 7fdd49fee740 20 client.17001 trim_cache size 0 max 16384 2024-03-24T09:26:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.758+0000 7fdd49fee740 1 Processor -- start 2024-03-24T09:26:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 -- start start 2024-03-24T09:26:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabdbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 0x55a3fabe7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3faace430 con 0x55a3fabf3f70 2024-03-24T09:26:33.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fabd8240 con 0x55a3fabe1fb0 2024-03-24T09:26:33.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd49fee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a3fabf6bb0 con 0x55a3fabe1bd0 2024-03-24T09:26:33.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 0x55a3fabe7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 0x55a3fabe7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54646/0 (socket says 172.21.15.116:54646) 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 -- 172.21.15.116:0/944629861 learned_addr learned my addr 172.21.15.116:0/944629861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd450d6640 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabdbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 -- 172.21.15.116:0/944629861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 msgr2=0x55a3fabe8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 -- 172.21.15.116:0/944629861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 msgr2=0x55a3fabdbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.759+0000 7fdd448d5640 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabdbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.760+0000 7fdd448d5640 1 -- 172.21.15.116:0/944629861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3faae3b20 con 0x55a3fabf3f70 2024-03-24T09:26:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.760+0000 7fdd448d5640 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 0x55a3fabe7030 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdd300118c0 tx=0x7fdd30009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e004a2cb222e30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.760+0000 7fdd2affd640 1 -- 172.21.15.116:0/944629861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30011570 con 0x55a3fabf3f70 2024-03-24T09:26:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.760+0000 7fdd2affd640 1 -- 172.21.15.116:0/944629861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd30023040 con 0x55a3fabf3f70 2024-03-24T09:26:33.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.760+0000 7fdd2affd640 1 -- 172.21.15.116:0/944629861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3000fbd0 con 0x55a3fabf3f70 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 -- 172.21.15.116:0/944629861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 msgr2=0x55a3fabe7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 0x55a3fabe7030 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdd300118c0 tx=0x7fdd30009370 comp rx=0 tx=0).stop 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 -- 172.21.15.116:0/944629861 shutdown_connections 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabf3f70 0x55a3fabe7030 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabe1fb0 0x55a3fabdbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 --2- 172.21.15.116:0/944629861 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1bd0 0x55a3fabe8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 -- 172.21.15.116:0/944629861 >> 172.21.15.116:0/944629861 conn(0x55a3fabdc900 msgr2=0x55a3fabddac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 -- 172.21.15.116:0/944629861 shutdown_connections 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 -- 172.21.15.116:0/944629861 wait complete. 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.761+0000 7fdd49fee740 1 -- start start 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 Processor -- start 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabfbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1fb0 0x55a3fabf8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabf9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3fabfc350 con 0x55a3fabe1bd0 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a3fabfc4c0 con 0x55a3fabf3f70 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a3fabfc630 con 0x55a3fabe1fb0 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd49fee740 10 client.0 _ll_register_callbacks cb 0x7fdd3c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd448d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabfbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd448d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabfbe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54656/0 (socket says 172.21.15.116:54656) 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd448d5640 1 -- 172.21.15.116:0/1450639285 learned_addr learned my addr 172.21.15.116:0/1450639285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:26:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.762+0000 7fdd448d5640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabf9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd450d6640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1fb0 0x55a3fabf8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd448d5640 1 -- 172.21.15.116:0/1450639285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1fb0 msgr2=0x55a3fabf8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd448d5640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1fb0 0x55a3fabf8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd448d5640 1 -- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 msgr2=0x55a3fabf9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd448d5640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabf9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd448d5640 1 -- 172.21.15.116:0/1450639285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3fab0aea0 con 0x55a3fabe1bd0 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd450d6640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1fb0 0x55a3fabf8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:26:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd448d5640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabfbe10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009fd0 tx=0x7fdd30009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50826c7b0337a5bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd28ff9640 1 -- 172.21.15.116:0/1450639285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30004960 con 0x55a3fabe1bd0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd28ff9640 1 -- 172.21.15.116:0/1450639285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdd30000e40 con 0x55a3fabe1bd0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.763+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd30005630 con 0x55a3fabe1bd0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.764+0000 7fdd28ff9640 1 -- 172.21.15.116:0/1450639285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdd30004ac0 con 0x55a3fabe1bd0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.764+0000 7fdd49fee740 10 client.17007 fetch_fsmap learned FSMap version 32 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.764+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a3fab09700 con 0x55a3fabe1bd0 2024-03-24T09:26:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.764+0000 7fdd28ff9640 1 -- 172.21.15.116:0/1450639285 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdd30005630 con 0x55a3fabe1bd0 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd28ff9640 1 -- 172.21.15.116:0/1450639285 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fdd3003c040 con 0x55a3fabe1bd0 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd49fee740 10 client.17007 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd49fee740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd49fee740 20 client.17007 populate_metadata read hostname 'smithi116' 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd49fee740 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac6b5f0 0x55a3fac6da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd49fee740 4 client.17007 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:26:33.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a3fa8d4890 con 0x55a3fac6b5f0 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd450d6640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac6b5f0 0x55a3fac6da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd28ff9640 10 client.17007 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.765+0000 7fdd450d6640 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac6b5f0 0x55a3fac6da10 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86de4bf9bd24f881 server_cookie=29fc1a7fa46f95b8 in_seq=0 out_seq=0 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd28ff9640 1 -- 172.21.15.116:0/1450639285 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a3fa8d4890 con 0x55a3fac6b5f0 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd28ff9640 10 client.17007 handle_command_reply: tid=0 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 client.17007 shutdown 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 msgr2=0x55a3fabfbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabfbe10 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fdd30009fd0 tx=0x7fdd30009f40 comp rx=0 tx=0).stop 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 shutdown_connections 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a3fac6b5f0 0x55a3fac6da10 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a3fabf3f70 0x55a3fabf9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a3fabe1fb0 0x55a3fabf8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd28ff9640 0 client.17007 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 --2- 172.21.15.116:0/1450639285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3fabe1bd0 0x55a3fabfbe10 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 >> 172.21.15.116:0/1450639285 conn(0x55a3fabdc900 msgr2=0x55a3fabdb9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 shutdown_connections 2024-03-24T09:26:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.766+0000 7fdd49fee740 1 -- 172.21.15.116:0/1450639285 wait complete. 2024-03-24T09:26:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd49fee740 20 client.17007 trim_cache size 0 max 16384 2024-03-24T09:26:33.777 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:26:33.777 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:26:33.777 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:26:33.777 INFO:teuthology.orchestra.run.smithi116.stdout: "4e207943-af53-45c3-8fec-f85d311713d8": { 2024-03-24T09:26:33.778 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:26:33.778 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "4e207943-af53-45c3-8fec-f85d311713d8", 2024-03-24T09:26:33.778 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:26:33.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:26:33.778 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:26:33.778 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:26:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdd2c0a2230 msgr2=0x7fdd2c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdd2c0a2230 0x7fdd2c0a46f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd380c5a20 tx=0x7fdd380a8040 comp rx=0 tx=0).stop 2024-03-24T09:26:33.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 msgr2=0x7fdd401968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd401968a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdd38004af0 tx=0x7fdd38033cb0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 shutdown_connections 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd40106660 0x7fdd401968a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdd2c0a2230 0x7fdd2c0a46f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd40100f60 0x7fdd4019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 --2- 172.21.15.116:0/4147428485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd401005a0 0x7fdd4019c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.767+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 >> 172.21.15.116:0/4147428485 conn(0x7fdd40077710 msgr2=0x7fdd400ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:26:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.768+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 shutdown_connections 2024-03-24T09:26:33.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:26:33.768+0000 7fdd4733a640 1 -- 172.21.15.116:0/4147428485 wait complete. 2024-03-24T09:26:33.782 INFO:tasks.cephfs.filesystem:scrub status for tag:4e207943-af53-45c3-8fec-f85d311713d8 - {'path': '/', 'tag': '4e207943-af53-45c3-8fec-f85d311713d8', 'options': 'recursive,force'} 2024-03-24T09:26:34.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:33 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:34.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:33 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:33 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: pgmap v137: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:34 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: pgmap v137: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:26:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:34 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: pgmap v137: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:26:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:35.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:34 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:26:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:36 smithi143 ceph-mon[35965]: pgmap v138: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:36 smithi117 ceph-mon[35788]: pgmap v138: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:37.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:36 smithi116 ceph-mon[32207]: pgmap v138: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:38 smithi143 ceph-mon[35965]: pgmap v139: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:38 smithi117 ceph-mon[35788]: pgmap v139: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:39.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:38 smithi116 ceph-mon[32207]: pgmap v139: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:40 smithi143 ceph-mon[35965]: pgmap v140: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:40 smithi117 ceph-mon[35788]: pgmap v140: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:40 smithi116 ceph-mon[32207]: pgmap v140: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:42 smithi143 ceph-mon[35965]: pgmap v141: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:42 smithi117 ceph-mon[35788]: pgmap v141: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:42 smithi116 ceph-mon[32207]: pgmap v141: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:44 smithi143 ceph-mon[35965]: pgmap v142: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:44 smithi117 ceph-mon[35788]: pgmap v142: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:44 smithi116 ceph-mon[32207]: pgmap v142: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:46 smithi143 ceph-mon[35965]: pgmap v143: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:46 smithi117 ceph-mon[35788]: pgmap v143: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:46 smithi116 ceph-mon[32207]: pgmap v143: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:26:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:48 smithi143 ceph-mon[35965]: pgmap v144: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:48 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:48 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:48 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:48 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:49.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:48 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:26:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:48 smithi117 ceph-mon[35788]: pgmap v144: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:48 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:48 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:48 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:48 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:49.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:48 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:26:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:48 smithi116 ceph-mon[32207]: pgmap v144: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:26:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:48 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:48 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:48 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:49.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:48 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:49.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:48 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:26:51.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:50 smithi143 ceph-mon[35965]: pgmap v145: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:50 smithi117 ceph-mon[35788]: pgmap v145: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:50 smithi116 ceph-mon[32207]: pgmap v145: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:53.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:52 smithi143 ceph-mon[35965]: pgmap v146: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:52 smithi117 ceph-mon[35788]: pgmap v146: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:52 smithi116 ceph-mon[32207]: pgmap v146: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:54 smithi143 ceph-mon[35965]: pgmap v147: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:54 smithi117 ceph-mon[35788]: pgmap v147: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:54 smithi116 ceph-mon[32207]: pgmap v147: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:57.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:56 smithi143 ceph-mon[35965]: pgmap v148: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:56 smithi117 ceph-mon[35788]: pgmap v148: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:57.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:56 smithi116 ceph-mon[32207]: pgmap v148: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:26:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:26:58 smithi143 ceph-mon[35965]: pgmap v149: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:26:58 smithi117 ceph-mon[35788]: pgmap v149: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:26:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:26:58 smithi116 ceph-mon[32207]: pgmap v149: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:01.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:00 smithi143 ceph-mon[35965]: pgmap v150: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:00 smithi117 ceph-mon[35788]: pgmap v150: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:01.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:00 smithi116 ceph-mon[32207]: pgmap v150: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:02 smithi143 ceph-mon[35965]: pgmap v151: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:02 smithi117 ceph-mon[35788]: pgmap v151: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:02 smithi116 ceph-mon[32207]: pgmap v151: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:03.783 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:27:03.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:27:03.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/2109895994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c4102060 msgr2=0x7fd5c410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/2109895994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c4102060 0x7fd5c410e570 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009cd0 tx=0x7fd5bc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/2109895994 shutdown_connections 2024-03-24T09:27:03.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/2109895994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c4102060 0x7fd5c410e570 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/2109895994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/2109895994 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5c41076b0 0x7fd5c4107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/2109895994 >> 172.21.15.116:0/2109895994 conn(0x7fd5c4078b10 msgr2=0x7fd5c4078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:03.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/2109895994 shutdown_connections 2024-03-24T09:27:03.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.898+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/2109895994 wait complete. 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 Processor -- start 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 -- start start 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5c4102060 0x7fd5c4073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c41076b0 0x7fd5c4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c41a5fe0 con 0x7fd5c41076b0 2024-03-24T09:27:03.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c41a6150 con 0x7fd5c41016a0 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5cbd7a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd5c41a6310 con 0x7fd5c4102060 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4073a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44442/0 (socket says 172.21.15.116:44442) 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5ca317640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c41076b0 0x7fd5c4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5c9b16640 1 -- 172.21.15.116:0/1760433140 learned_addr learned my addr 172.21.15.116:0/1760433140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.899+0000 7fd5c9315640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5c4102060 0x7fd5c4073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.900+0000 7fd5c9b16640 1 -- 172.21.15.116:0/1760433140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5c4102060 msgr2=0x7fd5c4073f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.900+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5c4102060 0x7fd5c4073f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.900+0000 7fd5c9b16640 1 -- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c41076b0 msgr2=0x7fd5c4074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.900+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c41076b0 0x7fd5c4074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.900+0000 7fd5c9b16640 1 -- 172.21.15.116:0/1760433140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5bc009900 con 0x7fd5c41016a0 2024-03-24T09:27:03.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.900+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4073a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400e9f0 tx=0x7fd5b400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:03.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.903+0000 7fd5baffd640 1 -- 172.21.15.116:0/1760433140 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4009700 con 0x7fd5c41016a0 2024-03-24T09:27:03.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.903+0000 7fd5baffd640 1 -- 172.21.15.116:0/1760433140 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5b400cd80 con 0x7fd5c41016a0 2024-03-24T09:27:03.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.903+0000 7fd5baffd640 1 -- 172.21.15.116:0/1760433140 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4018840 con 0x7fd5c41016a0 2024-03-24T09:27:03.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.903+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5c41a65f0 con 0x7fd5c41016a0 2024-03-24T09:27:03.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.903+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c41a6b50 con 0x7fd5c41016a0 2024-03-24T09:27:03.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.906+0000 7fd5baffd640 1 -- 172.21.15.116:0/1760433140 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd5b40189a0 con 0x7fd5c41016a0 2024-03-24T09:27:03.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.906+0000 7fd5cea32740 1 Processor -- start 2024-03-24T09:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5baffd640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5b00a22b0 0x7fd5b00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 -- start start 2024-03-24T09:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5baffd640 1 -- 172.21.15.116:0/1760433140 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd5b4014070 con 0x7fd5c41016a0 2024-03-24T09:27:03.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 0x55f3f43165b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5ca317640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5b00a22b0 0x7fd5b00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f431f6d0 0x55f3f4321ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f423af30 con 0x55f3f431f6d0 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f4284810 con 0x55f3f43161d0 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5cea32740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f4281d40 con 0x55f3f4316af0 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 0x55f3f43165b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 0x55f3f43165b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44444/0 (socket says 172.21.15.116:44444) 2024-03-24T09:27:03.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9315640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2724240360 learned_addr learned my addr 172.21.15.116:0/2724240360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5ca317640 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f431f6d0 0x55f3f4321ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2724240360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 msgr2=0x55f3f431f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2724240360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f431f6d0 msgr2=0x55f3f4321ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f431f6d0 0x55f3f4321ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2724240360 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f423eea0 con 0x55f3f43161d0 2024-03-24T09:27:03.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.907+0000 7fd5ca317640 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f431f6d0 0x55f3f4321ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5ca317640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5b00a22b0 0x7fd5b00a4770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005cc0 tx=0x7fd5bc032f80 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 0x55f3f43165b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4097740 tx=0x7fd5b409a790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5afc0780c4b29237 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5affff640 1 -- 172.21.15.116:0/2724240360 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40a3460 con 0x55f3f43161d0 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5affff640 1 -- 172.21.15.116:0/2724240360 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5b4010b50 con 0x55f3f43161d0 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5affff640 1 -- 172.21.15.116:0/2724240360 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40ce4d0 con 0x55f3f43161d0 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5cea32740 1 -- 172.21.15.116:0/2724240360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 msgr2=0x55f3f43165b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 0x55f3f43165b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4097740 tx=0x7fd5b409a790 comp rx=0 tx=0).stop 2024-03-24T09:27:03.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5cea32740 1 -- 172.21.15.116:0/2724240360 shutdown_connections 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f431f6d0 0x55f3f4321ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.908+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2724240360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f43161d0 0x55f3f43165b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- 172.21.15.116:0/2724240360 >> 172.21.15.116:0/2724240360 conn(0x55f3f4310900 msgr2=0x55f3f4310ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- 172.21.15.116:0/2724240360 shutdown_connections 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- 172.21.15.116:0/2724240360 wait complete. 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- start start 2024-03-24T09:27:03.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 Processor -- start 2024-03-24T09:27:03.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 0x55f3f431c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f431f6d0 0x55f3f431d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f427d520 con 0x55f3f43161d0 2024-03-24T09:27:03.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3f50430 con 0x55f3f431f6d0 2024-03-24T09:27:03.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3f0ce00 con 0x55f3f4316af0 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.909+0000 7fd5cea32740 10 client.0 _ll_register_callbacks cb 0x55f3f421bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 0x55f3f431c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 0x55f3f431c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47428/0 (socket says 172.21.15.116:47428) 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2955020823 learned_addr learned my addr 172.21.15.116:0/2955020823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9315640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5ca317640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f431f6d0 0x55f3f431d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2955020823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 msgr2=0x55f3f431cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f431f6d0 msgr2=0x55f3f431d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f431f6d0 0x55f3f431d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 -- 172.21.15.116:0/2955020823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5c41a6b50 con 0x55f3f43161d0 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9315640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 0x55f3f431c6d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400c6a0 tx=0x7fd5b40ce4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8496d4ff829d347 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/2955020823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b40975a0 con 0x55f3f43161d0 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/2955020823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd5b40a3460 con 0x55f3f43161d0 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3f3f7c7e0 con 0x55f3f43161d0 2024-03-24T09:27:03.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/2955020823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5b409a1c0 con 0x55f3f43161d0 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5cea32740 10 client.17019 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f3f427bba0 con 0x55f3f43161d0 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.910+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/2955020823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5b4089080 con 0x55f3f43161d0 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/2955020823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fd5b4085070 con 0x55f3f43161d0 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5cea32740 10 client.17019 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5cea32740 10 client.17019 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5cea32740 20 client.17019 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f432f990 0x55f3f4331db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5cea32740 4 client.17019 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3f3fb9720 con 0x55f3f432f990 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5c9315640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f432f990 0x55f3f4331db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5ad7fa640 10 client.17019 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.911+0000 7fd5c9315640 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f432f990 0x55f3f4331db0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d7862707f2d5c15 server_cookie=7a159fe95b167b78 in_seq=0 out_seq=0 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/2955020823 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3f3fb9720 con 0x55f3f432f990 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5ad7fa640 10 client.17019 handle_command_reply: tid=0 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 client.17019 shutdown 2024-03-24T09:27:03.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 msgr2=0x55f3f431c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 0x55f3f431c6d0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400c6a0 tx=0x7fd5b40ce4c0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 shutdown_connections 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f432f990 0x55f3f4331db0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f431f6d0 0x55f3f431d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f4316af0 0x55f3f431cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 --2- 172.21.15.116:0/2955020823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f43161d0 0x55f3f431c6d0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5ad7fa640 0 client.17019 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 >> 172.21.15.116:0/2955020823 conn(0x55f3f4310900 msgr2=0x55f3f430fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:03.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 shutdown_connections 2024-03-24T09:27:03.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 1 -- 172.21.15.116:0/2955020823 wait complete. 2024-03-24T09:27:03.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.912+0000 7fd5cea32740 20 client.17019 trim_cache size 0 max 16384 2024-03-24T09:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.929+0000 7fd5cea32740 1 Processor -- start 2024-03-24T09:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.929+0000 7fd5cea32740 1 -- start start 2024-03-24T09:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 0x55f3f431ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f432af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4316af0 0x55f3f4337680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5cea32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f4202430 con 0x55f3f4310900 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5cea32740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3fb9720 con 0x55f3f4316af0 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5cea32740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f431db30 con 0x55f3f43161d0 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 0x55f3f431ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 0x55f3f431ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47434/0 (socket says 172.21.15.116:47434) 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 -- 172.21.15.116:0/3269147259 learned_addr learned my addr 172.21.15.116:0/3269147259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5ca317640 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4316af0 0x55f3f4337680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9315640 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f432af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 -- 172.21.15.116:0/3269147259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 msgr2=0x55f3f432af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f432af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 -- 172.21.15.116:0/3269147259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4316af0 msgr2=0x55f3f4337680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4316af0 0x55f3f4337680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 -- 172.21.15.116:0/3269147259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f4217b20 con 0x55f3f4310900 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 0x55f3f431ee00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40ce7a0 tx=0x7fd5b401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20277567fc7c8ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5af7fe640 1 -- 172.21.15.116:0/3269147259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4099fc0 con 0x55f3f4310900 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5af7fe640 1 -- 172.21.15.116:0/3269147259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f3f3f7c7e0 con 0x55f3f4310900 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.930+0000 7fd5af7fe640 1 -- 172.21.15.116:0/3269147259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5b409ffe0 con 0x55f3f4310900 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 -- 172.21.15.116:0/3269147259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 msgr2=0x55f3f431ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 0x55f3f431ee00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40ce7a0 tx=0x7fd5b401f000 comp rx=0 tx=0).stop 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 -- 172.21.15.116:0/3269147259 shutdown_connections 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4316af0 0x55f3f4337680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f432af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 --2- 172.21.15.116:0/3269147259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4310900 0x55f3f431ee00 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 -- 172.21.15.116:0/3269147259 >> 172.21.15.116:0/3269147259 conn(0x55f3f431b310 msgr2=0x55f3f430fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 -- 172.21.15.116:0/3269147259 shutdown_connections 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 -- 172.21.15.116:0/3269147259 wait complete. 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 -- start start 2024-03-24T09:27:03.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.931+0000 7fd5cea32740 1 Processor -- start 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4310900 0x55f3f4327400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f4327940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 0x55f3f4323980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f4324050 con 0x55f3f4316af0 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f43241c0 con 0x55f3f4310900 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f4324380 con 0x55f3f43161d0 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5cea32740 10 client.0 _ll_register_callbacks cb 0x7fd5c00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 0x55f3f4323980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4310900 0x55f3f4327400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5c9315640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f4327940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 0x55f3f4323980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47436/0 (socket says 172.21.15.116:47436) 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5c9b16640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4310900 0x55f3f4327400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44464/0 (socket says 172.21.15.116:44464) 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 -- 172.21.15.116:0/1445017640 learned_addr learned my addr 172.21.15.116:0/1445017640 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 -- 172.21.15.116:0/1445017640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 msgr2=0x55f3f4327940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f4327940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 -- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4310900 msgr2=0x55f3f4327400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4310900 0x55f3f4327400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 -- 172.21.15.116:0/1445017640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f427bba0 con 0x55f3f4316af0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ca317640 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 0x55f3f4323980 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009cd0 tx=0x7fd5bc02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a0ea7b6082aecb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.932+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/1445017640 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc0329e0 con 0x55f3f4316af0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/1445017640 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd5bc003970 con 0x55f3f4316af0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5b4010b50 con 0x55f3f4316af0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/1445017640 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd5bc003b30 con 0x55f3f4316af0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 10 client.17025 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f3f423d700 con 0x55f3f4316af0 2024-03-24T09:27:03.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/1445017640 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd5b4010b50 con 0x55f3f4316af0 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/1445017640 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fd5bc050480 con 0x55f3f4316af0 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 10 client.17025 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 20 client.17025 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f431f6d0 0x55f3f43991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 4 client.17025 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f3f4008890 con 0x55f3f431f6d0 2024-03-24T09:27:03.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.933+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f431f6d0 0x55f3f43991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5ad7fa640 10 client.17025 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5c9b16640 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f431f6d0 0x55f3f43991d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e4c111e4ec147b3 server_cookie=71ba86c0d2e173b1 in_seq=0 out_seq=0 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5ad7fa640 1 -- 172.21.15.116:0/1445017640 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f3f4008890 con 0x55f3f431f6d0 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5ad7fa640 10 client.17025 handle_command_reply: tid=0 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 client.17025 shutdown 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 msgr2=0x55f3f4323980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 0x55f3f4323980 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009cd0 tx=0x7fd5bc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 shutdown_connections 2024-03-24T09:27:03.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f4316af0 0x55f3f4323980 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f43161d0 0x55f3f4327940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f3f431f6d0 0x55f3f43991d0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 --2- 172.21.15.116:0/1445017640 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f4310900 0x55f3f4327400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5ad7fa640 0 client.17025 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 >> 172.21.15.116:0/1445017640 conn(0x55f3f431b310 msgr2=0x55f3f431c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 shutdown_connections 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 1 -- 172.21.15.116:0/1445017640 wait complete. 2024-03-24T09:27:03.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.934+0000 7fd5cea32740 20 client.17025 trim_cache size 0 max 16384 2024-03-24T09:27:03.945 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:27:03.945 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:27:03.945 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:27:03.945 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:27:03.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5b00a22b0 msgr2=0x7fd5b00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5b00a22b0 0x7fd5b00a4770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc005cc0 tx=0x7fd5bc032f80 comp rx=0 tx=0).stop 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 msgr2=0x7fd5c4073a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4073a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400e9f0 tx=0x7fd5b400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 shutdown_connections 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5c41076b0 0x7fd5c4074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5b00a22b0 0x7fd5b00a4770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5c4102060 0x7fd5c4073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 --2- 172.21.15.116:0/1760433140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5c41016a0 0x7fd5c4073a10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:03.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 >> 172.21.15.116:0/1760433140 conn(0x7fd5c4078b10 msgr2=0x7fd5c41040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:03.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 shutdown_connections 2024-03-24T09:27:03.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:03.935+0000 7fd5cbd7a640 1 -- 172.21.15.116:0/1760433140 wait complete. 2024-03-24T09:27:03.948 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:27:03.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:27:04.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 -- 172.21.15.116:0/3105174513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 msgr2=0x7f576406a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 --2- 172.21.15.116:0/3105174513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576406a220 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5754009890 tx=0x7f575402bc00 comp rx=0 tx=0).stop 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 -- 172.21.15.116:0/3105174513 shutdown_connections 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 --2- 172.21.15.116:0/3105174513 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f576406a760 0x7f576410e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 --2- 172.21.15.116:0/3105174513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576406a220 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 --2- 172.21.15.116:0/3105174513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5764107710 0x7f5764107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 -- 172.21.15.116:0/3105174513 >> 172.21.15.116:0/3105174513 conn(0x7f57640776b0 msgr2=0x7f5764077ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 -- 172.21.15.116:0/3105174513 shutdown_connections 2024-03-24T09:27:04.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.049+0000 7f576a3c7640 1 -- 172.21.15.116:0/3105174513 wait complete. 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 Processor -- start 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 -- start start 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576419d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f576406a760 0x7f576419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5764107710 0x7f5764197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57641980d0 con 0x7f5764069dc0 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5764198240 con 0x7f576406a760 2024-03-24T09:27:04.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f576a3c7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5764198400 con 0x7f5764107710 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576419d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576419d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47456/0 (socket says 172.21.15.116:47456) 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 -- 172.21.15.116:0/1387307873 learned_addr learned my addr 172.21.15.116:0/1387307873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5768964640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5764107710 0x7f5764197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f57637fe640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f576406a760 0x7f576419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 -- 172.21.15.116:0/1387307873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5764107710 msgr2=0x7f5764197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5764107710 0x7f5764197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 -- 172.21.15.116:0/1387307873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f576406a760 msgr2=0x7f576419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f576406a760 0x7f576419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.050+0000 7f5763fff640 1 -- 172.21.15.116:0/1387307873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5754009490 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.051+0000 7f5763fff640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576419d8d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f575800e9f0 tx=0x7f575800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.051+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758009700 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.051+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57641986e0 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.051+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f575800cd80 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.051+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758018840 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.051+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57641a4970 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.052+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5764102e30 con 0x7f5764069dc0 2024-03-24T09:27:04.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.053+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f57580189a0 con 0x7f5764069dc0 2024-03-24T09:27:04.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.053+0000 7f57617fa640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f574c0a2230 0x7f574c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.053+0000 7f57637fe640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f574c0a2230 0x7f574c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.053+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5758014070 con 0x7f5764069dc0 2024-03-24T09:27:04.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.054+0000 7f57637fe640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f574c0a2230 0x7f574c0a46f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5754000c00 tx=0x7f5754005890 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.058+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f5758053280 con 0x7f5764069dc0 2024-03-24T09:27:04.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.269+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5764109bb0 con 0x7f5764069dc0 2024-03-24T09:27:04.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.269+0000 7f57617fa640 1 -- 172.21.15.116:0/1387307873 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11960 (secure 0 0 0) 0x7f5758097070 con 0x7f5764069dc0 2024-03-24T09:27:04.270 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:27:04.271 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:23:22.966826+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24484,24515,24451,24443,24422]},"id":1}]} 2024-03-24T09:27:04.271 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-03-24T09:27:04.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.271+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f574c0a2230 msgr2=0x7f574c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.271+0000 7f576a3c7640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f574c0a2230 0x7f574c0a46f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5754000c00 tx=0x7f5754005890 comp rx=0 tx=0).stop 2024-03-24T09:27:04.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.271+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 msgr2=0x7f576419d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.271+0000 7f576a3c7640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576419d8d0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f575800e9f0 tx=0x7f575800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 shutdown_connections 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f574c0a2230 0x7f574c0a46f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5764107710 0x7f5764197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f576406a760 0x7f576419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 --2- 172.21.15.116:0/1387307873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5764069dc0 0x7f576419d8d0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 >> 172.21.15.116:0/1387307873 conn(0x7f57640776b0 msgr2=0x7f57640ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 shutdown_connections 2024-03-24T09:27:04.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.272+0000 7f576a3c7640 1 -- 172.21.15.116:0/1387307873 wait complete. 2024-03-24T09:27:04.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:27:04.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 -- 172.21.15.116:0/2135764749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01008f0 msgr2=0x7ff8c0100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2135764749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01008f0 0x7ff8c0100d70 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009930 tx=0x7ff8a802bc30 comp rx=0 tx=0).stop 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 -- 172.21.15.116:0/2135764749 shutdown_connections 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2135764749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8c01012b0 0x7ff8c010d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2135764749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01008f0 0x7ff8c0100d70 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2135764749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c01066e0 0x7ff8c0106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 -- 172.21.15.116:0/2135764749 >> 172.21.15.116:0/2135764749 conn(0x7ff8c00775f0 msgr2=0x7ff8c0077a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 -- 172.21.15.116:0/2135764749 shutdown_connections 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.386+0000 7ff8c5781640 1 -- 172.21.15.116:0/2135764749 wait complete. 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 Processor -- start 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 -- start start 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8c01008f0 0x7ff8c019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 0x7ff8c019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c01066e0 0x7ff8c0196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0196f90 con 0x7ff8c01012b0 2024-03-24T09:27:04.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0197100 con 0x7ff8c01008f0 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8c5781640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01972c0 con 0x7ff8c01066e0 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 0x7ff8c019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 0x7ff8c019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47464/0 (socket says 172.21.15.116:47464) 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 -- 172.21.15.116:0/2124464183 learned_addr learned my addr 172.21.15.116:0/2124464183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8beffd640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8c01008f0 0x7ff8c019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c01066e0 0x7ff8c0196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 -- 172.21.15.116:0/2124464183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c01066e0 msgr2=0x7ff8c0196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c01066e0 0x7ff8c0196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 -- 172.21.15.116:0/2124464183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8c01008f0 msgr2=0x7ff8c019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8c01008f0 0x7ff8c019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.387+0000 7ff8b7fff640 1 -- 172.21.15.116:0/2124464183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b8009840 con 0x7ff8c01012b0 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.388+0000 7ff8b7fff640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 0x7ff8c019ccd0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009230 tx=0x7ff8a8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.388+0000 7ff8bcff9640 1 -- 172.21.15.116:0/2124464183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8004350 con 0x7ff8c01012b0 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.388+0000 7ff8bcff9640 1 -- 172.21.15.116:0/2124464183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8a80044b0 con 0x7ff8c01012b0 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.388+0000 7ff8bcff9640 1 -- 172.21.15.116:0/2124464183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8005320 con 0x7ff8c01012b0 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.388+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b0009490 con 0x7ff8c01012b0 2024-03-24T09:27:04.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.388+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a8009560 con 0x7ff8c01012b0 2024-03-24T09:27:04.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.390+0000 7ff8bcff9640 1 -- 172.21.15.116:0/2124464183 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7ff8a8005480 con 0x7ff8c01012b0 2024-03-24T09:27:04.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.390+0000 7ff8c8439740 1 Processor -- start 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.390+0000 7ff8c8439740 1 -- start start 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51200 0x562f1be515e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8bcff9640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff8ac0a22b0 0x7ff8ac0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be5cb00 0x562f1be5eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8c8439740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562f1bdbf840 con 0x562f1be51bb0 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8c8439740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562f1bdbcd70 con 0x562f1be5cb00 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8c8439740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562f1bdb8550 con 0x562f1be51200 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8b7fff640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff8ac0a22b0 0x7ff8ac0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47474/0 (socket says 172.21.15.116:47474) 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 -- 172.21.15.116:0/317769698 learned_addr learned my addr 172.21.15.116:0/317769698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be5cb00 0x562f1be5eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51200 0x562f1be515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8bcff9640 1 -- 172.21.15.116:0/2124464183 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8a80f0310 con 0x7ff8c01012b0 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 -- 172.21.15.116:0/317769698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51200 msgr2=0x562f1be515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51200 0x562f1be515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 -- 172.21.15.116:0/317769698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be5cb00 msgr2=0x562f1be5eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be5cb00 0x562f1be5eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 -- 172.21.15.116:0/317769698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f1bd4dde0 con 0x562f1be51bb0 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be5cb00 0x562f1be5eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8b7fff640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff8ac0a22b0 0x7ff8ac0a4770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80bc9f0 tx=0x7ff8a8009260 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8beffd640 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5c5c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009e20 tx=0x7ff8b0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e760bce962b6c45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8b57fa640 1 -- 172.21.15.116:0/317769698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0004230 con 0x562f1be51bb0 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8b57fa640 1 -- 172.21.15.116:0/317769698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8b0004390 con 0x562f1be51bb0 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.391+0000 7ff8b57fa640 1 -- 172.21.15.116:0/317769698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0019660 con 0x562f1be51bb0 2024-03-24T09:27:04.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 -- 172.21.15.116:0/317769698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 msgr2=0x562f1be5c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5c5c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009e20 tx=0x7ff8b0011810 comp rx=0 tx=0).stop 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 -- 172.21.15.116:0/317769698 shutdown_connections 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be5cb00 0x562f1be5eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5c5c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 --2- 172.21.15.116:0/317769698 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51200 0x562f1be515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 -- 172.21.15.116:0/317769698 >> 172.21.15.116:0/317769698 conn(0x562f1be4bb20 msgr2=0x562f1be4bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 -- 172.21.15.116:0/317769698 shutdown_connections 2024-03-24T09:27:04.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.392+0000 7ff8c8439740 1 -- 172.21.15.116:0/317769698 wait complete. 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 -- start start 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 Processor -- start 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be59b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be5a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562f1ba8b430 con 0x562f1be51bb0 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562f1ba47e00 con 0x562f1be51200 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562f1bd3d590 con 0x562f1be5cb00 2024-03-24T09:27:04.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8c8439740 10 client.0 _ll_register_callbacks cb 0x562f1bdacfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be59b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47484/0 (socket says 172.21.15.116:47484) 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 -- 172.21.15.116:0/1456032558 learned_addr learned my addr 172.21.15.116:0/1456032558 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be5a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 -- 172.21.15.116:0/1456032558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 msgr2=0x562f1be5a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be5a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 -- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 msgr2=0x562f1be59b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be59b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 -- 172.21.15.116:0/1456032558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b0009a80 con 0x562f1be51bb0 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be59b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8beffd640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5a080 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005360 tx=0x7ff8b0005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3859ce1c3e6ee6d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.393+0000 7ff8a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8a67fc640 1 -- 172.21.15.116:0/1456032558 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00046c0 con 0x562f1be51bb0 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8a67fc640 1 -- 172.21.15.116:0/1456032558 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8b0004820 con 0x562f1be51bb0 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562f1bab77e0 con 0x562f1be51bb0 2024-03-24T09:27:04.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8a67fc640 1 -- 172.21.15.116:0/1456032558 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8b00049e0 con 0x562f1be51bb0 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 10 client.17043 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562f1bdb6bd0 con 0x562f1be51bb0 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8a67fc640 1 -- 172.21.15.116:0/1456032558 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562f1bab77e0 con 0x562f1be51bb0 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8a67fc640 1 -- 172.21.15.116:0/1456032558 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff8b001a440 con 0x562f1be51bb0 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 10 client.17043 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 10 client.17043 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:27:04.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 20 client.17043 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1be6cb50 0x562f1be6ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 4 client.17043 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f1baf4720 con 0x562f1be6cb50 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.394+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1be6cb50 0x562f1be6ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.395+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1be6cb50 0x562f1be6ef90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=344ddd66f6ce6db9 server_cookie=7631a465917b376 in_seq=0 out_seq=0 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.395+0000 7ff8a67fc640 10 client.17043 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.395+0000 7ff8a67fc640 1 -- 172.21.15.116:0/1456032558 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562f1baf4720 con 0x562f1be6cb50 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.395+0000 7ff8a67fc640 10 client.17043 handle_command_reply: tid=0 2024-03-24T09:27:04.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.395+0000 7ff8c8439740 1 client.17043 shutdown 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 msgr2=0x562f1be5a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5a080 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0005360 tx=0x7ff8b0005390 comp rx=0 tx=0).stop 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 shutdown_connections 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be5a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1be6cb50 0x562f1be6ef90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be5a080 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8a67fc640 0 client.17043 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:04.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 --2- 172.21.15.116:0/1456032558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be59b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 >> 172.21.15.116:0/1456032558 conn(0x562f1be4bb20 msgr2=0x562f1be4d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 shutdown_connections 2024-03-24T09:27:04.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 1 -- 172.21.15.116:0/1456032558 wait complete. 2024-03-24T09:27:04.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.396+0000 7ff8c8439740 20 client.17043 trim_cache size 0 max 16384 2024-03-24T09:27:04.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 Processor -- start 2024-03-24T09:27:04.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 -- start start 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be683a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be4ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be748a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562f1be50750 con 0x562f1be51bb0 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562f1be4de70 con 0x562f1be51200 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8c8439740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562f1be4cb20 con 0x562f1be5cb00 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be4ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be683a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be683a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44528/0 (socket says 172.21.15.116:44528) 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be4ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47496/0 (socket says 172.21.15.116:47496) 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2256690752 learned_addr learned my addr 172.21.15.116:0/2256690752 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2256690752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 msgr2=0x562f1be748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2256690752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 msgr2=0x562f1be4ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be4ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2256690752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f1bd52c80 con 0x562f1be51200 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.409+0000 7ff8beffd640 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be4ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be683a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x562f1be4ccb0 tx=0x7ff8b8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43943d53643c7ee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8b4ff9640 1 -- 172.21.15.116:0/2256690752 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b802d070 con 0x562f1be51200 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8b4ff9640 1 -- 172.21.15.116:0/2256690752 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8b8004b20 con 0x562f1be51200 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8b4ff9640 1 -- 172.21.15.116:0/2256690752 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8016070 con 0x562f1be51200 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 -- 172.21.15.116:0/2256690752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 msgr2=0x562f1be683a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be683a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x562f1be4ccb0 tx=0x7ff8b8004a40 comp rx=0 tx=0).stop 2024-03-24T09:27:04.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 -- 172.21.15.116:0/2256690752 shutdown_connections 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be5cb00 0x562f1be748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be51bb0 0x562f1be4ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2256690752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be683a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 -- 172.21.15.116:0/2256690752 >> 172.21.15.116:0/2256690752 conn(0x562f1be4bb20 msgr2=0x562f1be4a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 -- 172.21.15.116:0/2256690752 shutdown_connections 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.410+0000 7ff8c8439740 1 -- 172.21.15.116:0/2256690752 wait complete. 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 -- start start 2024-03-24T09:27:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 Processor -- start 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be60b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51bb0 0x562f1be61070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 0x562f1becbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562f1becc3e0 con 0x562f1be5cb00 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562f1becc550 con 0x562f1be51200 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562f1becc710 con 0x562f1be51bb0 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8c8439740 10 client.0 _ll_register_callbacks cb 0x7ff8b8011720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 0x562f1becbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 0x562f1becbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47498/0 (socket says 172.21.15.116:47498) 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2536258332 learned_addr learned my addr 172.21.15.116:0/2536258332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be60b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8beffd640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51bb0 0x562f1be61070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2536258332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51bb0 msgr2=0x562f1be61070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51bb0 0x562f1be61070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 msgr2=0x562f1be60b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be60b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.411+0000 7ff8bf7fe640 1 -- 172.21.15.116:0/2536258332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f1bdb6bd0 con 0x562f1be5cb00 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8bf7fe640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 0x562f1becbea0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009b50 tx=0x7ff8b802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a78ff73f1e60b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8a67fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8a67fc640 1 -- 172.21.15.116:0/2536258332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b8016070 con 0x562f1be5cb00 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8a67fc640 1 -- 172.21.15.116:0/2536258332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ff8b8011050 con 0x562f1be5cb00 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8b0004390 con 0x562f1be5cb00 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8a67fc640 1 -- 172.21.15.116:0/2536258332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8b802d040 con 0x562f1be5cb00 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 10 client.17049 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7ff8b8012b70 con 0x562f1be5cb00 2024-03-24T09:27:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8a67fc640 1 -- 172.21.15.116:0/2536258332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff8b0004390 con 0x562f1be5cb00 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8a67fc640 1 -- 172.21.15.116:0/2536258332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff8b80130d0 con 0x562f1be5cb00 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 10 client.17049 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 10 client.17049 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 20 client.17049 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1bed3f20 0x562f1bed6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 4 client.17049 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f1bb43890 con 0x562f1bed3f20 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.412+0000 7ff8beffd640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1bed3f20 0x562f1bed6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8a67fc640 10 client.17049 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8beffd640 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1bed3f20 0x562f1bed6310 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=79593904bbf9cf2a server_cookie=16a3111ffce6ac1f in_seq=0 out_seq=0 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8a67fc640 1 -- 172.21.15.116:0/2536258332 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f1bb43890 con 0x562f1bed3f20 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8a67fc640 10 client.17049 handle_command_reply: tid=0 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 client.17049 shutdown 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 msgr2=0x562f1becbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 0x562f1becbea0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009b50 tx=0x7ff8b802a040 comp rx=0 tx=0).stop 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 shutdown_connections 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x562f1bed3f20 0x562f1bed6310 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562f1be5cb00 0x562f1becbea0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562f1be51bb0 0x562f1be61070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8a67fc640 0 client.17049 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 --2- 172.21.15.116:0/2536258332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562f1be51200 0x562f1be60b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 >> 172.21.15.116:0/2536258332 conn(0x562f1be4bb20 msgr2=0x562f1be4acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 shutdown_connections 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 1 -- 172.21.15.116:0/2536258332 wait complete. 2024-03-24T09:27:04.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.413+0000 7ff8c8439740 20 client.17049 trim_cache size 0 max 16384 2024-03-24T09:27:04.426 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:04.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff8ac0a22b0 msgr2=0x7ff8ac0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff8ac0a22b0 0x7ff8ac0a4770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80bc9f0 tx=0x7ff8a8009260 comp rx=0 tx=0).stop 2024-03-24T09:27:04.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 msgr2=0x7ff8c019ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 0x7ff8c019ccd0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8009230 tx=0x7ff8a8005dd0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 shutdown_connections 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff8ac0a22b0 0x7ff8ac0a4770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff8c01066e0 0x7ff8c0196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c01012b0 0x7ff8c019ccd0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 --2- 172.21.15.116:0/2124464183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8c01008f0 0x7ff8c019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 >> 172.21.15.116:0/2124464183 conn(0x7ff8c00775f0 msgr2=0x7ff8c00ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 shutdown_connections 2024-03-24T09:27:04.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.414+0000 7ff8c5781640 1 -- 172.21.15.116:0/2124464183 wait complete. 2024-03-24T09:27:04.429 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:27:04.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2920245094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd41076b0 msgr2=0x7fdcd4107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2920245094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd41076b0 0x7fdcd4107a90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009900 tx=0x7fdcc802bc30 comp rx=0 tx=0).stop 2024-03-24T09:27:04.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2920245094 shutdown_connections 2024-03-24T09:27:04.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2920245094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd4102060 0x7fdcd410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2920245094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41016a0 0x7fdcd4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2920245094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd41076b0 0x7fdcd4107a90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2920245094 >> 172.21.15.116:0/2920245094 conn(0x7fdcd4078b10 msgr2=0x7fdcd4078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2920245094 shutdown_connections 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2920245094 wait complete. 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 Processor -- start 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.532+0000 7fdcdaf3e640 1 -- start start 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 0x7fdcd419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4102060 0x7fdcd419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41076b0 0x7fdcd4197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd4198080 con 0x7fdcd41016a0 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd41981f0 con 0x7fdcd4102060 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdcd41983b0 con 0x7fdcd41076b0 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 0x7fdcd419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 0x7fdcd419d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47522/0 (socket says 172.21.15.116:47522) 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2644166451 learned_addr learned my addr 172.21.15.116:0/2644166451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4102060 0x7fdcd419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd94db640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41076b0 0x7fdcd4197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2644166451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41076b0 msgr2=0x7fdcd4197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41076b0 0x7fdcd4197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2644166451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4102060 msgr2=0x7fdcd419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4102060 0x7fdcd419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2644166451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc0009490 con 0x7fdcd41016a0 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4102060 0x7fdcd419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 0x7fdcd419d880 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005940 tx=0x7fdcc8005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd1ffb640 1 -- 172.21.15.116:0/2644166451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8004350 con 0x7fdcd41016a0 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc8009560 con 0x7fdcd41016a0 2024-03-24T09:27:04.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd1ffb640 1 -- 172.21.15.116:0/2644166451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc80044b0 con 0x7fdcd41016a0 2024-03-24T09:27:04.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.533+0000 7fdcd1ffb640 1 -- 172.21.15.116:0/2644166451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80053c0 con 0x7fdcd41016a0 2024-03-24T09:27:04.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.534+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41a4740 con 0x7fdcd41016a0 2024-03-24T09:27:04.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcd1ffb640 1 -- 172.21.15.116:0/2644166451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fdcc8005520 con 0x7fdcd41016a0 2024-03-24T09:27:04.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 Processor -- start 2024-03-24T09:27:04.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 -- start start 2024-03-24T09:27:04.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00a3f0 0x55efcb00a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb0133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb015ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55efcaf78840 con 0x55efcb0138f0 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55efcaf75d70 con 0x55efcb00a3f0 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcddc03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55efcaf71550 con 0x55efcb00ad10 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcd1ffb640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcc40a2230 0x7fdcc40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb0133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.536+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb015ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb015ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47532/0 (socket says 172.21.15.116:47532) 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd3fff640 1 -- 172.21.15.116:0/2342067474 learned_addr learned my addr 172.21.15.116:0/2342067474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd94db640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00a3f0 0x55efcb00a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd94db640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcc40a2230 0x7fdcc40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd1ffb640 1 -- 172.21.15.116:0/2644166451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdcc80f04d0 con 0x7fdcd41016a0 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2342067474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00a3f0 msgr2=0x55efcb00a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00a3f0 0x55efcb00a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2342067474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 msgr2=0x55efcb015ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb015ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2342067474 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efcaf06de0 con 0x55efcb00ad10 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd94db640 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00a3f0 0x55efcb00a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb015ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd94db640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcc40a2230 0x7fdcc40a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdccc005fd0 tx=0x7fdccc009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb0133b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80ed920 tx=0x7fdcc8038000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb8cfeeafd375236 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcbeffd640 1 -- 172.21.15.116:0/2342067474 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80c30c0 con 0x55efcb00ad10 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcbeffd640 1 -- 172.21.15.116:0/2342067474 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc80bcc50 con 0x55efcb00ad10 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.537+0000 7fdcbeffd640 1 -- 172.21.15.116:0/2342067474 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80eda90 con 0x55efcb00ad10 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 -- 172.21.15.116:0/2342067474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 msgr2=0x55efcb0133b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb0133b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80ed920 tx=0x7fdcc8038000 comp rx=0 tx=0).stop 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 -- 172.21.15.116:0/2342067474 shutdown_connections 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb015ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb0133b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2342067474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00a3f0 0x55efcb00a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 -- 172.21.15.116:0/2342067474 >> 172.21.15.116:0/2342067474 conn(0x55efcb004b20 msgr2=0x55efcb004f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 -- 172.21.15.116:0/2342067474 shutdown_connections 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 -- 172.21.15.116:0/2342067474 wait complete. 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 -- start start 2024-03-24T09:27:04.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.538+0000 7fdcddc03740 1 Processor -- start 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 0x55efcb010980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00ad10 0x55efcb010ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb011400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55efcac44430 con 0x55efcb0138f0 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55efcac00e00 con 0x55efcb00ad10 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55efcaef6590 con 0x55efcb00a3f0 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcddc03740 10 client.0 _ll_register_callbacks cb 0x55efcaf65fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd94db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb011400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 0x55efcb010980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00ad10 0x55efcb010ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 0x55efcb010980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:35570/0 (socket says 172.21.15.116:35570) 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd94db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb011400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47544/0 (socket says 172.21.15.116:47544) 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2158307720 learned_addr learned my addr 172.21.15.116:0/2158307720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00ad10 msgr2=0x55efcb010ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00ad10 0x55efcb010ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2158307720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 msgr2=0x55efcb011400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb011400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 -- 172.21.15.116:0/2158307720 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcd41a4740 con 0x55efcb00a3f0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd94db640 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb011400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 0x55efcb010980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80bcbe0 tx=0x7fdcc8005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd7149c53b82331a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.539+0000 7fdcb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2158307720 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80c5360 con 0x55efcb00a3f0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2158307720 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdcc80a7070 con 0x55efcb00a3f0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55efcac707e0 con 0x55efcb00a3f0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2158307720 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdcc80c3050 con 0x55efcb00a3f0 2024-03-24T09:27:04.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 10 client.26758 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55efcaf6fbd0 con 0x55efcb00a3f0 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2158307720 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55efcac707e0 con 0x55efcb00a3f0 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2158307720 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fdcc803c070 con 0x55efcb00a3f0 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 10 client.26758 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 10 client.26758 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 20 client.26758 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb0233a0 0x55efcb025790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 4 client.26758 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efcacad720 con 0x55efcb0233a0 2024-03-24T09:27:04.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.540+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb0233a0 0x55efcb025790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcb3fff640 10 client.26758 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb0233a0 0x55efcb025790 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4798185d066ff1eb server_cookie=8f9f75e31d6d9b7b in_seq=0 out_seq=0 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2158307720 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55efcacad720 con 0x55efcb0233a0 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcb3fff640 10 client.26758 handle_command_reply: tid=0 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 client.26758 shutdown 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 msgr2=0x55efcb010980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 0x55efcb010980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80bcbe0 tx=0x7fdcc8005c00 comp rx=0 tx=0).stop 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 shutdown_connections 2024-03-24T09:27:04.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb0233a0 0x55efcb025790 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb0138f0 0x55efcb011400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcb3fff640 0 client.26758 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb00ad10 0x55efcb010ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 --2- 172.21.15.116:0/2158307720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00a3f0 0x55efcb010980 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.541+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 >> 172.21.15.116:0/2158307720 conn(0x55efcb004b20 msgr2=0x55efcb006870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.542+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 shutdown_connections 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.542+0000 7fdcddc03740 1 -- 172.21.15.116:0/2158307720 wait complete. 2024-03-24T09:27:04.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.542+0000 7fdcddc03740 20 client.26758 trim_cache size 0 max 16384 2024-03-24T09:27:04.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 Processor -- start 2024-03-24T09:27:04.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 -- start start 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb0130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb01ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55efcac707e0 con 0x55efcb00a3f0 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55efcb011950 con 0x55efcb0138f0 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcddc03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55efcb005b20 con 0x55efcb00ad10 2024-03-24T09:27:04.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb0130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb0130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47556/0 (socket says 172.21.15.116:47556) 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 -- 172.21.15.116:0/1932330684 learned_addr learned my addr 172.21.15.116:0/1932330684 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd94db640 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb01ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 -- 172.21.15.116:0/1932330684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 msgr2=0x55efcb01ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb01ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 -- 172.21.15.116:0/1932330684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 msgr2=0x55efcb003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 -- 172.21.15.116:0/1932330684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efcaf0bc80 con 0x55efcb00a3f0 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd94db640 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb003b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb0130b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80b9620 tx=0x7fdcc80c0ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d9e47426e61bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcbe7fc640 1 -- 172.21.15.116:0/1932330684 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80c30b0 con 0x55efcb00a3f0 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcbe7fc640 1 -- 172.21.15.116:0/1932330684 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc80bc5b0 con 0x55efcb00a3f0 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.555+0000 7fdcbe7fc640 1 -- 172.21.15.116:0/1932330684 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80c5130 con 0x55efcb00a3f0 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 -- 172.21.15.116:0/1932330684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 msgr2=0x55efcb0130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb0130b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80b9620 tx=0x7fdcc80c0ee0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 -- 172.21.15.116:0/1932330684 shutdown_connections 2024-03-24T09:27:04.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb003b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb01ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 --2- 172.21.15.116:0/1932330684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb0130b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 -- 172.21.15.116:0/1932330684 >> 172.21.15.116:0/1932330684 conn(0x55efcb004b20 msgr2=0x55efcb003430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 -- 172.21.15.116:0/1932330684 shutdown_connections 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 -- 172.21.15.116:0/1932330684 wait complete. 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 -- start start 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.556+0000 7fdcddc03740 1 Processor -- start 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb00d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb00da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb00dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55efcb0886b0 con 0x55efcb00a3f0 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55efcb0887f0 con 0x55efcb0138f0 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55efcb088930 con 0x55efcb00ad10 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 10 client.0 _ll_register_callbacks cb 0x7fdcc0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb00d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb00d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47558/0 (socket says 172.21.15.116:47558) 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 -- 172.21.15.116:0/3349984936 learned_addr learned my addr 172.21.15.116:0/3349984936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd94db640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb00dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb00da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 -- 172.21.15.116:0/3349984936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 msgr2=0x55efcb00da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb00da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 -- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 msgr2=0x55efcb00dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb00dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 -- 172.21.15.116:0/3349984936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efcaf6fbd0 con 0x55efcb00a3f0 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcd8cda640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb00d540 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80bceb0 tx=0x7fdcc8005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b12bae2e94ba19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcb3fff640 1 -- 172.21.15.116:0/3349984936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcc80c30b0 con 0x55efcb00a3f0 2024-03-24T09:27:04.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcb3fff640 1 -- 172.21.15.116:0/3349984936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fdcc80a7040 con 0x55efcb00a3f0 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdcc80bcc50 con 0x55efcb00a3f0 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcb3fff640 1 -- 172.21.15.116:0/3349984936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdcc80c5360 con 0x55efcb00a3f0 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 10 client.17067 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.557+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55efcaf31920 con 0x55efcb00a3f0 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcb3fff640 1 -- 172.21.15.116:0/3349984936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdcc80bcc50 con 0x55efcb00a3f0 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcb3fff640 1 -- 172.21.15.116:0/3349984936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fdcc803c070 con 0x55efcb00a3f0 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcddc03740 10 client.17067 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcddc03740 10 client.17067 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:27:04.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcddc03740 20 client.17067 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcddc03740 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb089ba0 0x55efcb08bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcddc03740 4 client.17067 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55efcacfc890 con 0x55efcb089ba0 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb089ba0 0x55efcb08bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcd3fff640 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb089ba0 0x55efcb08bf90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b1e7f47d9f2ba874 server_cookie=85acfeb82a5c1dd9 in_seq=0 out_seq=0 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.558+0000 7fdcb3fff640 10 client.17067 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcb3fff640 1 -- 172.21.15.116:0/3349984936 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55efcacfc890 con 0x55efcb089ba0 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcb3fff640 10 client.17067 handle_command_reply: tid=0 2024-03-24T09:27:04.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 client.17067 shutdown 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 msgr2=0x55efcb00d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb00d540 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdcc80bceb0 tx=0x7fdcc8005c00 comp rx=0 tx=0).stop 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 shutdown_connections 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55efcb089ba0 0x55efcb08bf90 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55efcb0138f0 0x55efcb00dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcb3fff640 0 client.17067 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55efcb00ad10 0x55efcb00da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 --2- 172.21.15.116:0/3349984936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55efcb00a3f0 0x55efcb00d540 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 >> 172.21.15.116:0/3349984936 conn(0x55efcb004b20 msgr2=0x55efcb0111b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 shutdown_connections 2024-03-24T09:27:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 1 -- 172.21.15.116:0/3349984936 wait complete. 2024-03-24T09:27:04.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcddc03740 20 client.17067 trim_cache size 0 max 16384 2024-03-24T09:27:04.571 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcc40a2230 msgr2=0x7fdcc40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcc40a2230 0x7fdcc40a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdccc005fd0 tx=0x7fdccc009500 comp rx=0 tx=0).stop 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 msgr2=0x7fdcd419d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 0x7fdcd419d880 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8005940 tx=0x7fdcc8005df0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 shutdown_connections 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdcc40a2230 0x7fdcc40a46f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcd41076b0 0x7fdcd4197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd4102060 0x7fdcd419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 --2- 172.21.15.116:0/2644166451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcd41016a0 0x7fdcd419d880 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.559+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 >> 172.21.15.116:0/2644166451 conn(0x7fdcd4078b10 msgr2=0x7fdcd41041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.560+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 shutdown_connections 2024-03-24T09:27:04.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.560+0000 7fdcdaf3e640 1 -- 172.21.15.116:0/2644166451 wait complete. 2024-03-24T09:27:04.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:27:04.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 -- 172.21.15.116:0/2890599420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 msgr2=0x7f53d010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 --2- 172.21.15.116:0/2890599420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 0x7f53d010d510 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009cd0 tx=0x7f53c802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 -- 172.21.15.116:0/2890599420 shutdown_connections 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 --2- 172.21.15.116:0/2890599420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 0x7f53d010d510 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 --2- 172.21.15.116:0/2890599420 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d0101d00 0x7f53d01052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 --2- 172.21.15.116:0/2890599420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d0101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 -- 172.21.15.116:0/2890599420 >> 172.21.15.116:0/2890599420 conn(0x7f53d0075950 msgr2=0x7f53d0075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 -- 172.21.15.116:0/2890599420 shutdown_connections 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.678+0000 7f53d631e640 1 -- 172.21.15.116:0/2890599420 wait complete. 2024-03-24T09:27:04.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 Processor -- start 2024-03-24T09:27:04.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 -- start start 2024-03-24T09:27:04.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 0x7f53d019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d0196a40 0x7f53d0196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53d0197430 con 0x7f53d0101350 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f53d01975a0 con 0x7f53d01059d0 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d631e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f53d0197760 con 0x7f53d0196a40 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d019c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47572/0 (socket says 172.21.15.116:47572) 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53cffff640 1 -- 172.21.15.116:0/1662619704 learned_addr learned my addr 172.21.15.116:0/1662619704 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 0x7f53d019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.679+0000 7f53d48bb640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d0196a40 0x7f53d0196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cffff640 1 -- 172.21.15.116:0/1662619704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d0196a40 msgr2=0x7f53d0196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cffff640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d0196a40 0x7f53d0196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cffff640 1 -- 172.21.15.116:0/1662619704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 msgr2=0x7f53d019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cffff640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 0x7f53d019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cffff640 1 -- 172.21.15.116:0/1662619704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c0009490 con 0x7f53d0101350 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cffff640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d019c890 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f53c000ea60 tx=0x7f53c000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cd7fa640 1 -- 172.21.15.116:0/1662619704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c000cde0 con 0x7f53d0101350 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c8009900 con 0x7f53d0101350 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cd7fa640 1 -- 172.21.15.116:0/1662619704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53c00044d0 con 0x7f53d0101350 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53cd7fa640 1 -- 172.21.15.116:0/1662619704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c0010690 con 0x7f53d0101350 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.680+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d01a3b20 con 0x7f53d0101350 2024-03-24T09:27:04.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.682+0000 7f53cd7fa640 1 -- 172.21.15.116:0/1662619704 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f53c0003e40 con 0x7f53d0101350 2024-03-24T09:27:04.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 Processor -- start 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 -- start start 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f42138c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138cd10 0x55f4213953b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f421397ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f4212fa840 con 0x55f42138cd10 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f4212f7d70 con 0x55f4213958f0 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f4212f3550 con 0x55f42138c3f0 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138cd10 0x55f4213953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f421397ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cd7fa640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f53b80a2230 0x7f53b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138cd10 0x55f4213953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47578/0 (socket says 172.21.15.116:47578) 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cffff640 1 -- 172.21.15.116:0/311325131 learned_addr learned my addr 172.21.15.116:0/311325131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d48bb640 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f42138c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53d48bb640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f53b80a2230 0x7f53b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cf7fe640 1 -- 172.21.15.116:0/311325131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 msgr2=0x55f42138c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f42138c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cf7fe640 1 -- 172.21.15.116:0/311325131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138cd10 msgr2=0x55f4213953b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138cd10 0x55f4213953b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.683+0000 7f53cf7fe640 1 -- 172.21.15.116:0/311325131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f421288de0 con 0x55f4213958f0 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53cd7fa640 1 -- 172.21.15.116:0/1662619704 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f53c0014070 con 0x7f53d0101350 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f421397ce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f53d0198230 tx=0x7f53c4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a01df8ff9f1769c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53d48bb640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f53b80a2230 0x7f53b80a46f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f53c8005cc0 tx=0x7f53c8005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53b67fc640 1 -- 172.21.15.116:0/311325131 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4004050 con 0x55f4213958f0 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53b67fc640 1 -- 172.21.15.116:0/311325131 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53c4005020 con 0x55f4213958f0 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53b67fc640 1 -- 172.21.15.116:0/311325131 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4004ae0 con 0x55f4213958f0 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53d8fdc740 1 -- 172.21.15.116:0/311325131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 msgr2=0x55f421397ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.684+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f421397ce0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f53d0198230 tx=0x7f53c4011810 comp rx=0 tx=0).stop 2024-03-24T09:27:04.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- 172.21.15.116:0/311325131 shutdown_connections 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f421397ce0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138cd10 0x55f4213953b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/311325131 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f42138c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- 172.21.15.116:0/311325131 >> 172.21.15.116:0/311325131 conn(0x55f421386b20 msgr2=0x55f421386f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- 172.21.15.116:0/311325131 shutdown_connections 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- 172.21.15.116:0/311325131 wait complete. 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- start start 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 Processor -- start 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 0x55f421392920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f421392e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f4213933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f420fc6430 con 0x55f42138c3f0 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f420f82e00 con 0x55f4213958f0 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f421278590 con 0x55f42138cd10 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53d8fdc740 10 client.0 _ll_register_callbacks cb 0x55f4212e7fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.685+0000 7f53cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 0x55f421392920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 0x55f421392920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47590/0 (socket says 172.21.15.116:47590) 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cf7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f421392e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 -- 172.21.15.116:0/2921893538 learned_addr learned my addr 172.21.15.116:0/2921893538 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53d48bb640 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f4213933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 -- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 msgr2=0x55f421392e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f421392e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 -- 172.21.15.116:0/2921893538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 msgr2=0x55f4213933a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f4213933a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 -- 172.21.15.116:0/2921893538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c4009850 con 0x55f42138c3f0 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53cffff640 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 0x55f421392920 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f53c000c760 tx=0x7f53c009aa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60dee9d6121983e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53af7fe640 1 -- 172.21.15.116:0/2921893538 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c00a3680 con 0x55f42138c3f0 2024-03-24T09:27:04.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53af7fe640 1 -- 172.21.15.116:0/2921893538 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f53c009aee0 con 0x55f42138c3f0 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f420ff27e0 con 0x55f42138c3f0 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53af7fe640 1 -- 172.21.15.116:0/2921893538 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f53c00ce5a0 con 0x55f42138c3f0 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53d8fdc740 10 client.17079 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4212f1bd0 con 0x55f42138c3f0 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.686+0000 7f53af7fe640 1 -- 172.21.15.116:0/2921893538 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f420ff27e0 con 0x55f42138c3f0 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53af7fe640 1 -- 172.21.15.116:0/2921893538 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f53c009e070 con 0x55f42138c3f0 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53d8fdc740 10 client.17079 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53d8fdc740 10 client.17079 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:27:04.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53d8fdc740 20 client.17079 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f4213a5970 0x55f4213a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53d8fdc740 4 client.17079 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f42102f720 con 0x55f4213a5970 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f4213a5970 0x55f4213a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53af7fe640 10 client.17079 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.687+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f4213a5970 0x55f4213a7d60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c55256612c03f5ee server_cookie=c8858381c8e6a61b in_seq=0 out_seq=0 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53af7fe640 1 -- 172.21.15.116:0/2921893538 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f42102f720 con 0x55f4213a5970 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53af7fe640 10 client.17079 handle_command_reply: tid=0 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 client.17079 shutdown 2024-03-24T09:27:04.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 msgr2=0x55f421392920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 0x55f421392920 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f53c000c760 tx=0x7f53c009aa20 comp rx=0 tx=0).stop 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 shutdown_connections 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f4213a5970 0x55f4213a7d60 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4213958f0 0x55f4213933a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f421392e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/2921893538 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f42138c3f0 0x55f421392920 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53af7fe640 0 client.17079 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 >> 172.21.15.116:0/2921893538 conn(0x55f421386b20 msgr2=0x55f421388870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 shutdown_connections 2024-03-24T09:27:04.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 1 -- 172.21.15.116:0/2921893538 wait complete. 2024-03-24T09:27:04.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.688+0000 7f53d8fdc740 20 client.17079 trim_cache size 0 max 16384 2024-03-24T09:27:04.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 Processor -- start 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 -- start start 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 0x55f4213911f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f4213a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f4213931c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f4213a2470 con 0x55f4213958f0 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f421391730 con 0x55f42138c3f0 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f421377e20 con 0x55f42138cd10 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d48bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f4213931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 0x55f4213911f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d48bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f4213931c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47596/0 (socket says 172.21.15.116:47596) 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 0x55f4213911f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44644/0 (socket says 172.21.15.116:44644) 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53d48bb640 1 -- 172.21.15.116:0/381509652 learned_addr learned my addr 172.21.15.116:0/381509652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.701+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f4213a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d48bb640 1 -- 172.21.15.116:0/381509652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 msgr2=0x55f4213a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d48bb640 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f4213a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d48bb640 1 -- 172.21.15.116:0/381509652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 msgr2=0x55f4213911f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d48bb640 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 0x55f4213911f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d48bb640 1 -- 172.21.15.116:0/381509652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f42128dc80 con 0x55f4213958f0 2024-03-24T09:27:04.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53cffff640 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 0x55f4213911f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d48bb640 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f4213931c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009890 tx=0x7f53c802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef863f486f342c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53b5ffb640 1 -- 172.21.15.116:0/381509652 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8047040 con 0x55f4213958f0 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53b5ffb640 1 -- 172.21.15.116:0/381509652 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53c8034780 con 0x55f4213958f0 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53b5ffb640 1 -- 172.21.15.116:0/381509652 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8003ea0 con 0x55f4213958f0 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d8fdc740 1 -- 172.21.15.116:0/381509652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 msgr2=0x55f4213931c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.702+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f4213931c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009890 tx=0x7f53c802e040 comp rx=0 tx=0).stop 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- 172.21.15.116:0/381509652 shutdown_connections 2024-03-24T09:27:04.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f4213931c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138cd10 0x55f4213a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/381509652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138c3f0 0x55f4213911f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- 172.21.15.116:0/381509652 >> 172.21.15.116:0/381509652 conn(0x55f421386b20 msgr2=0x55f421385cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- 172.21.15.116:0/381509652 shutdown_connections 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- 172.21.15.116:0/381509652 wait complete. 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- start start 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 Processor -- start 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f421399780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138cd10 0x55f421399cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f42140aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f42139a400 con 0x55f4213958f0 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f42140b1e0 con 0x55f42138cd10 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f42140b370 con 0x55f42138c3f0 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d8fdc740 10 client.0 _ll_register_callbacks cb 0x7f53c000ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d48bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f42140aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53cffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f421399780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d48bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f42140aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47612/0 (socket says 172.21.15.116:47612) 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53cf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138cd10 0x55f421399cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.703+0000 7f53d48bb640 1 -- 172.21.15.116:0/3604082885 learned_addr learned my addr 172.21.15.116:0/3604082885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d48bb640 1 -- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 msgr2=0x55f421399780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d48bb640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f421399780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d48bb640 1 -- 172.21.15.116:0/3604082885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138cd10 msgr2=0x55f421399cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d48bb640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138cd10 0x55f421399cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d48bb640 1 -- 172.21.15.116:0/3604082885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4212f1bd0 con 0x55f4213958f0 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53cffff640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f421399780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53cf7fe640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138cd10 0x55f421399cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d48bb640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f42140aca0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f53c802c630 tx=0x7f53c8032800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1012742c1a74c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53af7fe640 1 -- 172.21.15.116:0/3604082885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8057650 con 0x55f4213958f0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53af7fe640 1 -- 172.21.15.116:0/3604082885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f53c8003970 con 0x55f4213958f0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53c4005020 con 0x55f4213958f0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53af7fe640 1 -- 172.21.15.116:0/3604082885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f53c8034e90 con 0x55f4213958f0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d8fdc740 10 client.17085 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f53d010dd20 con 0x55f4213958f0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.704+0000 7f53af7fe640 1 -- 172.21.15.116:0/3604082885 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f53c4005020 con 0x55f4213958f0 2024-03-24T09:27:04.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53af7fe640 1 -- 172.21.15.116:0/3604082885 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f53c80617b0 con 0x55f4213958f0 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 10 client.17085 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 10 client.17085 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 20 client.17085 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f42140c810 0x55f42140ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 4 client.17085 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f42107e890 con 0x55f42140c810 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53cffff640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f42140c810 0x55f42140ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53af7fe640 10 client.17085 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:04.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53cffff640 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f42140c810 0x55f42140ec30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9111bab501acf7ab server_cookie=41ab0ba182664b24 in_seq=0 out_seq=0 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53af7fe640 1 -- 172.21.15.116:0/3604082885 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f42107e890 con 0x55f42140c810 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53af7fe640 10 client.17085 handle_command_reply: tid=0 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 client.17085 shutdown 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 msgr2=0x55f42140aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f42140aca0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f53c802c630 tx=0x7f53c8032800 comp rx=0 tx=0).stop 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 shutdown_connections 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f4213958f0 0x55f42140aca0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55f42140c810 0x55f42140ec30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f42138cd10 0x55f421399cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53d8fdc740 1 --2- 172.21.15.116:0/3604082885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f42138c3f0 0x55f421399780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.705+0000 7f53af7fe640 0 client.17085 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 >> 172.21.15.116:0/3604082885 conn(0x55f421386b20 msgr2=0x55f421385cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 shutdown_connections 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d8fdc740 1 -- 172.21.15.116:0/3604082885 wait complete. 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d8fdc740 20 client.17085 trim_cache size 0 max 16384 2024-03-24T09:27:04.717 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f53b80a2230 msgr2=0x7f53b80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f53b80a2230 0x7f53b80a46f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f53c8005cc0 tx=0x7f53c8005c50 comp rx=0 tx=0).stop 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 msgr2=0x7f53d019c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d019c890 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f53c000ea60 tx=0x7f53c000ef20 comp rx=0 tx=0).stop 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 shutdown_connections 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f53b80a2230 0x7f53b80a46f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53d0196a40 0x7f53d0196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f53d01059d0 0x7f53d019cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 --2- 172.21.15.116:0/1662619704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53d0101350 0x7f53d019c890 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 >> 172.21.15.116:0/1662619704 conn(0x7f53d0075950 msgr2=0x7f53d00ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 shutdown_connections 2024-03-24T09:27:04.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.706+0000 7f53d631e640 1 -- 172.21.15.116:0/1662619704 wait complete. 2024-03-24T09:27:04.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:27:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 -- 172.21.15.116:0/3486834983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310106660 msgr2=0x7f7310106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 --2- 172.21.15.116:0/3486834983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310106660 0x7f7310106a40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f7300009860 tx=0x7f730002bb70 comp rx=0 tx=0).stop 2024-03-24T09:27:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 -- 172.21.15.116:0/3486834983 shutdown_connections 2024-03-24T09:27:04.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 --2- 172.21.15.116:0/3486834983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7310100f60 0x7f731010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 --2- 172.21.15.116:0/3486834983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73101005a0 0x7f7310100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 --2- 172.21.15.116:0/3486834983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310106660 0x7f7310106a40 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 -- 172.21.15.116:0/3486834983 >> 172.21.15.116:0/3486834983 conn(0x7f7310077710 msgr2=0x7f7310077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 -- 172.21.15.116:0/3486834983 shutdown_connections 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.823+0000 7f73165d7640 1 -- 172.21.15.116:0/3486834983 wait complete. 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 Processor -- start 2024-03-24T09:27:04.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 -- start start 2024-03-24T09:27:04.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73101005a0 0x7f7310198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 0x7f7310198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7310106660 0x7f73101a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73101997f0 con 0x7f7310100f60 2024-03-24T09:27:04.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f73101a3ff0 con 0x7f7310106660 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f73165d7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f73101a4130 con 0x7f73101005a0 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 0x7f7310198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 0x7f7310198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47628/0 (socket says 172.21.15.116:47628) 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f730f7fe640 1 -- 172.21.15.116:0/1081928558 learned_addr learned my addr 172.21.15.116:0/1081928558 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f7314b74640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7310106660 0x7f73101a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.824+0000 7f730ffff640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73101005a0 0x7f7310198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730f7fe640 1 -- 172.21.15.116:0/1081928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73101005a0 msgr2=0x7f7310198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730f7fe640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73101005a0 0x7f7310198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730f7fe640 1 -- 172.21.15.116:0/1081928558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7310106660 msgr2=0x7f73101a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730f7fe640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7310106660 0x7f73101a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730f7fe640 1 -- 172.21.15.116:0/1081928558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7308009840 con 0x7f7310100f60 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f7314b74640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7310106660 0x7f73101a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730f7fe640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 0x7f7310198f70 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f730400e9f0 tx=0x7f730400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730d7fa640 1 -- 172.21.15.116:0/1081928558 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730400cc60 con 0x7f7310100f60 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7300009490 con 0x7f7310100f60 2024-03-24T09:27:04.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730d7fa640 1 -- 172.21.15.116:0/1081928558 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f730400cdc0 con 0x7f7310100f60 2024-03-24T09:27:04.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f730d7fa640 1 -- 172.21.15.116:0/1081928558 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7304004dd0 con 0x7f7310100f60 2024-03-24T09:27:04.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.825+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73101a4760 con 0x7f7310100f60 2024-03-24T09:27:04.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.827+0000 7f730d7fa640 1 -- 172.21.15.116:0/1081928558 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7304005000 con 0x7f7310100f60 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.827+0000 7f73191bb740 1 Processor -- start 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.827+0000 7f73191bb740 1 -- start start 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f73191bb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c513f0 0x556ca8c517d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f73191bb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 0x556ca8c5a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f73191bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f73191bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ca8bbf840 con 0x556ca8c5a8f0 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f73191bb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ca8bbcd70 con 0x556ca8c51d10 2024-03-24T09:27:04.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f73191bb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ca8bb8550 con 0x556ca8c513f0 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730d7fa640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72f80a2230 0x7f72f80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f7314b74640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72f80a2230 0x7f72f80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 0x556ca8c5a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 0x556ca8c5a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44698/0 (socket says 172.21.15.116:44698) 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 -- 172.21.15.116:0/302494549 learned_addr learned my addr 172.21.15.116:0/302494549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f7314b74640 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c513f0 0x556ca8c517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730d7fa640 1 -- 172.21.15.116:0/1081928558 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f7304014070 con 0x7f7310100f60 2024-03-24T09:27:04.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 -- 172.21.15.116:0/302494549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c513f0 msgr2=0x556ca8c517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c513f0 0x556ca8c517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 -- 172.21.15.116:0/302494549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c5a8f0 msgr2=0x556ca8c5cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 -- 172.21.15.116:0/302494549 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca8b4dde0 con 0x556ca8c51d10 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730f7fe640 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f7314b74640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72f80a2230 0x7f72f80a46f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7308005fd0 tx=0x7f7308011860 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f730ffff640 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 0x556ca8c5a3b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f73000057c0 tx=0x7f7300005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c6e352a5d18909 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f72f67fc640 1 -- 172.21.15.116:0/302494549 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300004030 con 0x556ca8c51d10 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f72f67fc640 1 -- 172.21.15.116:0/302494549 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f730002ce90 con 0x556ca8c51d10 2024-03-24T09:27:04.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.828+0000 7f72f67fc640 1 -- 172.21.15.116:0/302494549 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730002c800 con 0x556ca8c51d10 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 -- 172.21.15.116:0/302494549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 msgr2=0x556ca8c5a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 0x556ca8c5a3b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f73000057c0 tx=0x7f7300005b20 comp rx=0 tx=0).stop 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 -- 172.21.15.116:0/302494549 shutdown_connections 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c51d10 0x556ca8c5a3b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 --2- 172.21.15.116:0/302494549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c513f0 0x556ca8c517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 -- 172.21.15.116:0/302494549 >> 172.21.15.116:0/302494549 conn(0x556ca8c4bb20 msgr2=0x556ca8c4bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 -- 172.21.15.116:0/302494549 shutdown_connections 2024-03-24T09:27:04.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 -- 172.21.15.116:0/302494549 wait complete. 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.829+0000 7f73191bb740 1 -- start start 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 Processor -- start 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c57930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c57e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ca888b430 con 0x556ca8c51d10 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ca8847e00 con 0x556ca8c513f0 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ca8b3d590 con 0x556ca8c5a8f0 2024-03-24T09:27:04.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f73191bb740 10 client.0 _ll_register_callbacks cb 0x556ca8bacfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c57e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c57930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c57e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47634/0 (socket says 172.21.15.116:47634) 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730f7fe640 1 -- 172.21.15.116:0/2966447286 learned_addr learned my addr 172.21.15.116:0/2966447286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f7314b74640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730ffff640 1 -- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 msgr2=0x556ca8c583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730ffff640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730ffff640 1 -- 172.21.15.116:0/2966447286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 msgr2=0x556ca8c57e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730ffff640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c57e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730ffff640 1 -- 172.21.15.116:0/2966447286 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f731010dd20 con 0x556ca8c513f0 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.830+0000 7f730f7fe640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c57e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f730ffff640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c57930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7300009830 tx=0x7f73000346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2928c293c19dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f72ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f72ef7fe640 1 -- 172.21.15.116:0/2966447286 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7300004450 con 0x556ca8c513f0 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556ca88b77e0 con 0x556ca8c513f0 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f72ef7fe640 1 -- 172.21.15.116:0/2966447286 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f730004e070 con 0x556ca8c513f0 2024-03-24T09:27:04.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f72ef7fe640 1 -- 172.21.15.116:0/2966447286 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f730002c580 con 0x556ca8c513f0 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 10 client.26924 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556ca8bb6bd0 con 0x556ca8c513f0 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f72ef7fe640 1 -- 172.21.15.116:0/2966447286 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x556ca88b77e0 con 0x556ca8c513f0 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f72ef7fe640 1 -- 172.21.15.116:0/2966447286 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f7300003ae0 con 0x556ca8c513f0 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 10 client.26924 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 10 client.26924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 20 client.26924 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8c6a7c0 0x556ca8c6cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 4 client.26924 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.831+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ca88f4720 con 0x556ca8c6a7c0 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.832+0000 7f730f7fe640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8c6a7c0 0x556ca8c6cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.832+0000 7f72ef7fe640 10 client.26924 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.832+0000 7f730f7fe640 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8c6a7c0 0x556ca8c6cbf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d8acd5c113718ca server_cookie=c4a12cd4508acb76 in_seq=0 out_seq=0 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.832+0000 7f72ef7fe640 1 -- 172.21.15.116:0/2966447286 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ca88f4720 con 0x556ca8c6a7c0 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.832+0000 7f72ef7fe640 10 client.26924 handle_command_reply: tid=0 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.832+0000 7f73191bb740 1 client.26924 shutdown 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 msgr2=0x556ca8c57930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c57930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7300009830 tx=0x7f73000346f0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 shutdown_connections 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8c6a7c0 0x556ca8c6cbf0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c57e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f72ef7fe640 0 client.26924 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 --2- 172.21.15.116:0/2966447286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c57930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 >> 172.21.15.116:0/2966447286 conn(0x556ca8c4bb20 msgr2=0x556ca8c4d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 shutdown_connections 2024-03-24T09:27:04.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 1 -- 172.21.15.116:0/2966447286 wait complete. 2024-03-24T09:27:04.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.833+0000 7f73191bb740 20 client.26924 trim_cache size 0 max 16384 2024-03-24T09:27:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 Processor -- start 2024-03-24T09:27:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 -- start start 2024-03-24T09:27:04.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c5a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c66120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c724d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ca8c50740 con 0x556ca8c51d10 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ca8c4cb20 con 0x556ca8c513f0 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f73191bb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ca8c3ce20 con 0x556ca8c5a8f0 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c5a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c66120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c5a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44716/0 (socket says 172.21.15.116:44716) 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730ffff640 1 -- 172.21.15.116:0/2626215157 learned_addr learned my addr 172.21.15.116:0/2626215157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f7314b74640 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 -- 172.21.15.116:0/2626215157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 msgr2=0x556ca8c724d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c724d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 -- 172.21.15.116:0/2626215157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 msgr2=0x556ca8c5a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c5a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 -- 172.21.15.116:0/2626215157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ca8b52c80 con 0x556ca8c51d10 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f7314b74640 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:27:04.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730ffff640 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c5a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.846+0000 7f730f7fe640 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c66120 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f73040976b0 tx=0x7f730406cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42054a22ca785fca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f72f5ffb640 1 -- 172.21.15.116:0/2626215157 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730409b3e0 con 0x556ca8c51d10 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f72f5ffb640 1 -- 172.21.15.116:0/2626215157 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f730409ace0 con 0x556ca8c51d10 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f72f5ffb640 1 -- 172.21.15.116:0/2626215157 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73040052b0 con 0x556ca8c51d10 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 -- 172.21.15.116:0/2626215157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 msgr2=0x556ca8c66120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c66120 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f73040976b0 tx=0x7f730406cca0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 -- 172.21.15.116:0/2626215157 shutdown_connections 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c724d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c66120 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 --2- 172.21.15.116:0/2626215157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c5a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 -- 172.21.15.116:0/2626215157 >> 172.21.15.116:0/2626215157 conn(0x556ca8c4bb20 msgr2=0x556ca8c520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 -- 172.21.15.116:0/2626215157 shutdown_connections 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 -- 172.21.15.116:0/2626215157 wait complete. 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.847+0000 7f73191bb740 1 -- start start 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 Processor -- start 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c62b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c5de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556ca8c5eb90 con 0x556ca8c51d10 2024-03-24T09:27:04.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556ca8c5ed00 con 0x556ca8c513f0 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556ca8ccfb30 con 0x556ca8c5a8f0 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f73191bb740 10 client.0 _ll_register_callbacks cb 0x7f7304004f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c5de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c5de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47650/0 (socket says 172.21.15.116:47650) 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 -- 172.21.15.116:0/3273591734 learned_addr learned my addr 172.21.15.116:0/3273591734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730ffff640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c62b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f7314b74640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 -- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 msgr2=0x556ca8c5e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 -- 172.21.15.116:0/3273591734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 msgr2=0x556ca8c62b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c62b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 -- 172.21.15.116:0/3273591734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ca8bb6bd0 con 0x556ca8c51d10 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730ffff640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c62b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f730f7fe640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c5de80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f73040cbaf0 tx=0x7f7304005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e94b2d0ee7e0b07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.848+0000 7f72ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f72ef7fe640 1 -- 172.21.15.116:0/3273591734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73040a3480 con 0x556ca8c51d10 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f72ef7fe640 1 -- 172.21.15.116:0/3273591734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f73040cd040 con 0x556ca8c51d10 2024-03-24T09:27:04.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f730002ce90 con 0x556ca8c51d10 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f72ef7fe640 1 -- 172.21.15.116:0/3273591734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f730409b3e0 con 0x556ca8c51d10 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 10 client.17103 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f7300009490 con 0x556ca8c51d10 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f72ef7fe640 1 -- 172.21.15.116:0/3273591734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f730002ce90 con 0x556ca8c51d10 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f72ef7fe640 1 -- 172.21.15.116:0/3273591734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f730401f060 con 0x556ca8c51d10 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 10 client.17103 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 10 client.17103 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 20 client.17103 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8cd1050 0x556ca8cd3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 4 client.17103 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ca8943890 con 0x556ca8cd1050 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.849+0000 7f730ffff640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8cd1050 0x556ca8cd3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f72ef7fe640 10 client.17103 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f730ffff640 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8cd1050 0x556ca8cd3470 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4f46b92c1b6ccd5 server_cookie=3a72d9872a83f6ec in_seq=0 out_seq=0 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f72ef7fe640 1 -- 172.21.15.116:0/3273591734 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ca8943890 con 0x556ca8cd1050 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f72ef7fe640 10 client.17103 handle_command_reply: tid=0 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 client.17103 shutdown 2024-03-24T09:27:04.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 msgr2=0x556ca8c5de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c5de80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f73040cbaf0 tx=0x7f7304005750 comp rx=0 tx=0).stop 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 shutdown_connections 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x556ca8cd1050 0x556ca8cd3470 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x556ca8c5a8f0 0x556ca8c5e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f72ef7fe640 0 client.17103 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556ca8c51d10 0x556ca8c5de80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 --2- 172.21.15.116:0/3273591734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556ca8c513f0 0x556ca8c62b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 >> 172.21.15.116:0/3273591734 conn(0x556ca8c4bb20 msgr2=0x556ca8c4a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 shutdown_connections 2024-03-24T09:27:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 1 -- 172.21.15.116:0/3273591734 wait complete. 2024-03-24T09:27:04.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73191bb740 20 client.17103 trim_cache size 0 max 16384 2024-03-24T09:27:04.863 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72f80a2230 msgr2=0x7f72f80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73165d7640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72f80a2230 0x7f72f80a46f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7308005fd0 tx=0x7f7308011860 comp rx=0 tx=0).stop 2024-03-24T09:27:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 msgr2=0x7f7310198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.850+0000 7f73165d7640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 0x7f7310198f70 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f730400e9f0 tx=0x7f730400edf0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 shutdown_connections 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7310106660 0x7f73101a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7310100f60 0x7f7310198f70 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72f80a2230 0x7f72f80a46f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 --2- 172.21.15.116:0/1081928558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f73101005a0 0x7f7310198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 >> 172.21.15.116:0/1081928558 conn(0x7f7310077710 msgr2=0x7f73100ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 shutdown_connections 2024-03-24T09:27:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.851+0000 7f73165d7640 1 -- 172.21.15.116:0/1081928558 wait complete. 2024-03-24T09:27:04.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:27:04.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 -- 172.21.15.116:0/1369475619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 msgr2=0x7f286010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 --2- 172.21.15.116:0/1369475619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f286010d420 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ca0 tx=0x7f285802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 -- 172.21.15.116:0/1369475619 shutdown_connections 2024-03-24T09:27:04.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 --2- 172.21.15.116:0/1369475619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f286010d420 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 --2- 172.21.15.116:0/1369475619 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28601005a0 0x7f2860100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 --2- 172.21.15.116:0/1369475619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2860106660 0x7f2860106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 -- 172.21.15.116:0/1369475619 >> 172.21.15.116:0/1369475619 conn(0x7f2860077710 msgr2=0x7f2860077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 -- 172.21.15.116:0/1369475619 shutdown_connections 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.969+0000 7f286744e640 1 -- 172.21.15.116:0/1369475619 wait complete. 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 Processor -- start 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 -- start start 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28601005a0 0x7f28601989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f2860198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2860106660 0x7f28601a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2860199770 con 0x7f2860100f60 2024-03-24T09:27:04.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f28601a3f70 con 0x7f2860106660 2024-03-24T09:27:04.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f286744e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f28601a40b0 con 0x7f28601005a0 2024-03-24T09:27:04.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f2860198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f2860198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47668/0 (socket says 172.21.15.116:47668) 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 -- 172.21.15.116:0/370260388 learned_addr learned my addr 172.21.15.116:0/370260388 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28659eb640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2860106660 0x7f28601a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28651ea640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28601005a0 0x7f28601989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 -- 172.21.15.116:0/370260388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28601005a0 msgr2=0x7f28601989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28601005a0 0x7f28601989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 -- 172.21.15.116:0/370260388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2860106660 msgr2=0x7f28601a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.970+0000 7f28649e9640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2860106660 0x7f28601a3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f28649e9640 1 -- 172.21.15.116:0/370260388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c009490 con 0x7f2860100f60 2024-03-24T09:27:04.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f28649e9640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f2860198ef0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f285000ea20 tx=0x7f285000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f28567fc640 1 -- 172.21.15.116:0/370260388 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850009720 con 0x7f2860100f60 2024-03-24T09:27:04.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2858009900 con 0x7f2860100f60 2024-03-24T09:27:04.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f28567fc640 1 -- 172.21.15.116:0/370260388 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f285000cd20 con 0x7f2860100f60 2024-03-24T09:27:04.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f28567fc640 1 -- 172.21.15.116:0/370260388 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850010430 con 0x7f2860100f60 2024-03-24T09:27:04.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.971+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28601a4690 con 0x7f2860100f60 2024-03-24T09:27:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.973+0000 7f28567fc640 1 -- 172.21.15.116:0/370260388 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2850004f70 con 0x7f2860100f60 2024-03-24T09:27:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 Processor -- start 2024-03-24T09:27:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 -- start start 2024-03-24T09:27:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x5568779397d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x5568779423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877944ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5568778a7840 con 0x5568779428f0 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5568778a4d70 con 0x556877939d10 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f286a02f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5568778a0550 con 0x5568779393f0 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28567fc640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f284c0a22b0 0x7f284c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28649e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877944ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28659eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x5568779397d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x5568779423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x5568779423b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:44764/0 (socket says 172.21.15.116:44764) 2024-03-24T09:27:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 -- 172.21.15.116:0/3078886812 learned_addr learned my addr 172.21.15.116:0/3078886812 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28659eb640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f284c0a22b0 0x7f284c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28567fc640 1 -- 172.21.15.116:0/370260388 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2850014070 con 0x7f2860100f60 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 -- 172.21.15.116:0/3078886812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 msgr2=0x5568779397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x5568779397d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 -- 172.21.15.116:0/3078886812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 msgr2=0x556877944ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877944ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28651ea640 1 -- 172.21.15.116:0/3078886812 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556877835de0 con 0x556877939d10 2024-03-24T09:27:04.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.974+0000 7f28659eb640 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x5568779397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f28659eb640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f284c0a22b0 0x7f284c0a4770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2858004af0 tx=0x7f2858005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f28651ea640 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x5568779423b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f285c009e00 tx=0x7f285c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c762b82726e2322 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f284b7fe640 1 -- 172.21.15.116:0/3078886812 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c004230 con 0x556877939d10 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f284b7fe640 1 -- 172.21.15.116:0/3078886812 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f285c004390 con 0x556877939d10 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f284b7fe640 1 -- 172.21.15.116:0/3078886812 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c019740 con 0x556877939d10 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 -- 172.21.15.116:0/3078886812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 msgr2=0x5568779423b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x5568779423b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f285c009e00 tx=0x7f285c011810 comp rx=0 tx=0).stop 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 -- 172.21.15.116:0/3078886812 shutdown_connections 2024-03-24T09:27:04.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877944ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x5568779423b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 --2- 172.21.15.116:0/3078886812 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x5568779397d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 -- 172.21.15.116:0/3078886812 >> 172.21.15.116:0/3078886812 conn(0x556877933b20 msgr2=0x556877933f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.975+0000 7f286a02f740 1 -- 172.21.15.116:0/3078886812 shutdown_connections 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 -- 172.21.15.116:0/3078886812 wait complete. 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 -- start start 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 Processor -- start 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 0x55687793f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779428f0 0x556877940390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556877573430 con 0x5568779393f0 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55687752fe00 con 0x556877939d10 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556877825590 con 0x5568779428f0 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f286a02f740 10 client.0 _ll_register_callbacks cb 0x556877894fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f28651ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 0x55687793f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f28651ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 0x55687793f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47684/0 (socket says 172.21.15.116:47684) 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f28651ea640 1 -- 172.21.15.116:0/535193440 learned_addr learned my addr 172.21.15.116:0/535193440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f28659eb640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779428f0 0x556877940390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.976+0000 7f28649e9640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28651ea640 1 -- 172.21.15.116:0/535193440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779428f0 msgr2=0x556877940390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28651ea640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779428f0 0x556877940390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28651ea640 1 -- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 msgr2=0x55687793fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28651ea640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28651ea640 1 -- 172.21.15.116:0/535193440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c009a60 con 0x5568779393f0 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28659eb640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779428f0 0x556877940390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:04.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f28651ea640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 0x55687793f910 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f285c0093f0 tx=0x7f285c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bfc5b07399c8c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f2848ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f2848ff9640 1 -- 172.21.15.116:0/535193440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c003800 con 0x5568779393f0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f2848ff9640 1 -- 172.21.15.116:0/535193440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f285c003960 con 0x5568779393f0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55687759f7e0 con 0x5568779393f0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f2848ff9640 1 -- 172.21.15.116:0/535193440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f285c0194c0 con 0x5568779393f0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f286a02f740 10 client.17115 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55687789ebd0 con 0x5568779393f0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.977+0000 7f2848ff9640 1 -- 172.21.15.116:0/535193440 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55687759f7e0 con 0x5568779393f0 2024-03-24T09:27:04.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f2848ff9640 1 -- 172.21.15.116:0/535193440 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f285c01a630 con 0x5568779393f0 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f286a02f740 10 client.17115 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f286a02f740 10 client.17115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f286a02f740 20 client.17115 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f286a02f740 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779529a0 0x556877954de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f286a02f740 4 client.17115 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568775dc720 con 0x5568779529a0 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f28649e9640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779529a0 0x556877954de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:04.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f2848ff9640 10 client.17115 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.978+0000 7f28649e9640 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779529a0 0x556877954de0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab4bc0da479809e9 server_cookie=35572d09873f9b42 in_seq=0 out_seq=0 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f2848ff9640 1 -- 172.21.15.116:0/535193440 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568775dc720 con 0x5568779529a0 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f2848ff9640 10 client.17115 handle_command_reply: tid=0 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 client.17115 shutdown 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 msgr2=0x55687793f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 0x55687793f910 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f285c0093f0 tx=0x7f285c005780 comp rx=0 tx=0).stop 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 shutdown_connections 2024-03-24T09:27:04.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779428f0 0x556877940390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779529a0 0x556877954de0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 --2- 172.21.15.116:0/535193440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779393f0 0x55687793f910 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 >> 172.21.15.116:0/535193440 conn(0x556877933b20 msgr2=0x556877935870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f2848ff9640 0 client.17115 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 shutdown_connections 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 1 -- 172.21.15.116:0/535193440 wait complete. 2024-03-24T09:27:04.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.979+0000 7f286a02f740 20 client.17115 trim_cache size 0 max 16384 2024-03-24T09:27:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.992+0000 7f286a02f740 1 Processor -- start 2024-03-24T09:27:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.992+0000 7f286a02f740 1 -- start start 2024-03-24T09:27:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f286a02f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x556877942040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f286a02f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687794e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f286a02f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877935680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f286a02f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556877939050 con 0x5568779428f0 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f286a02f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5568779384e0 con 0x556877939d10 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f286a02f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x556877934b20 con 0x5568779393f0 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877935680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877935680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47686/0 (socket says 172.21.15.116:47686) 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 -- 172.21.15.116:0/1535473168 learned_addr learned my addr 172.21.15.116:0/1535473168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28651ea640 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x556877942040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28649e9640 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687794e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 -- 172.21.15.116:0/1535473168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 msgr2=0x556877942040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x556877942040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 -- 172.21.15.116:0/1535473168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 msgr2=0x55687794e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687794e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 -- 172.21.15.116:0/1535473168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55687783ac80 con 0x5568779428f0 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28649e9640 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687794e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f28659eb640 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877935680 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ca0 tx=0x7f2858009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4be5230e3228f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f284affd640 1 -- 172.21.15.116:0/1535473168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858032820 con 0x5568779428f0 2024-03-24T09:27:05.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f284affd640 1 -- 172.21.15.116:0/1535473168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2858049050 con 0x5568779428f0 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.993+0000 7f284affd640 1 -- 172.21.15.116:0/1535473168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858004680 con 0x5568779428f0 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 -- 172.21.15.116:0/1535473168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 msgr2=0x556877935680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877935680 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f2858009ca0 tx=0x7f2858009360 comp rx=0 tx=0).stop 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 -- 172.21.15.116:0/1535473168 shutdown_connections 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x556877935680 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687794e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 --2- 172.21.15.116:0/1535473168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x556877942040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 -- 172.21.15.116:0/1535473168 >> 172.21.15.116:0/1535473168 conn(0x556877933b20 msgr2=0x556877934f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 -- 172.21.15.116:0/1535473168 shutdown_connections 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 -- 172.21.15.116:0/1535473168 wait complete. 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 -- start start 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.994+0000 7f286a02f740 1 Processor -- start 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x55687793c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x5568779b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55687793d4a0 con 0x5568779428f0 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55687793d610 con 0x556877939d10 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5568779b8220 con 0x5568779393f0 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f286a02f740 10 client.0 _ll_register_callbacks cb 0x7f285c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x5568779b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28649e9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x5568779b7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47702/0 (socket says 172.21.15.116:47702) 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28651ea640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x55687793c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 -- 172.21.15.116:0/4101473670 learned_addr learned my addr 172.21.15.116:0/4101473670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 -- 172.21.15.116:0/4101473670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 msgr2=0x55687793c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x55687793c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 -- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 msgr2=0x55687793ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 -- 172.21.15.116:0/4101473670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28601a4690 con 0x5568779428f0 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f28659eb640 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x5568779b7ce0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f285802c630 tx=0x7f2858005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb2fbf496acabf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.995+0000 7f2848ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:05.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f2848ff9640 1 -- 172.21.15.116:0/4101473670 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858032820 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f2848ff9640 1 -- 172.21.15.116:0/4101473670 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2858032980 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f285c004390 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f2848ff9640 1 -- 172.21.15.116:0/4101473670 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2858003970 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 10 client.17121 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f2858030070 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f2848ff9640 1 -- 172.21.15.116:0/4101473670 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f285c004390 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f2848ff9640 1 -- 172.21.15.116:0/4101473670 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f28580507c0 con 0x5568779428f0 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 10 client.17121 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:05.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 10 client.17121 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 20 client.17121 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779b9850 0x5568779bbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 4 client.17121 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55687762b890 con 0x5568779b9850 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.996+0000 7f28651ea640 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779b9850 0x5568779bbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f2848ff9640 10 client.17121 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f28651ea640 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779b9850 0x5568779bbc70 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d404f95155083007 server_cookie=53f7a85600bc0796 in_seq=0 out_seq=0 2024-03-24T09:27:05.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f2848ff9640 1 -- 172.21.15.116:0/4101473670 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55687762b890 con 0x5568779b9850 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f2848ff9640 10 client.17121 handle_command_reply: tid=0 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 client.17121 shutdown 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 msgr2=0x5568779b7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x5568779b7ce0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f285802c630 tx=0x7f2858005ef0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 shutdown_connections 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5568779428f0 0x5568779b7ce0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5568779b9850 0x5568779bbc70 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556877939d10 0x55687793ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 --2- 172.21.15.116:0/4101473670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5568779393f0 0x55687793c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f2848ff9640 0 client.17121 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 >> 172.21.15.116:0/4101473670 conn(0x556877933b20 msgr2=0x556877932cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 shutdown_connections 2024-03-24T09:27:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 1 -- 172.21.15.116:0/4101473670 wait complete. 2024-03-24T09:27:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.997+0000 7f286a02f740 20 client.17121 trim_cache size 0 max 16384 2024-03-24T09:27:05.010 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:05.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f284c0a22b0 msgr2=0x7f284c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f284c0a22b0 0x7f284c0a4770 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2858004af0 tx=0x7f2858005c50 comp rx=0 tx=0).stop 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 msgr2=0x7f2860198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f2860198ef0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f285000ea20 tx=0x7f285000ee20 comp rx=0 tx=0).stop 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 shutdown_connections 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2860106660 0x7f28601a3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2860100f60 0x7f2860198ef0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f284c0a22b0 0x7f284c0a4770 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 --2- 172.21.15.116:0/370260388 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f28601005a0 0x7f28601989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:05.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 >> 172.21.15.116:0/370260388 conn(0x7f2860077710 msgr2=0x7f28600ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:05.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 shutdown_connections 2024-03-24T09:27:05.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:04.998+0000 7f286744e640 1 -- 172.21.15.116:0/370260388 wait complete. 2024-03-24T09:27:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:04 smithi143 ceph-mon[35965]: pgmap v152: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:04 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1387307873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:27:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:04 smithi117 ceph-mon[35788]: pgmap v152: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:04 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1387307873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:27:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:04 smithi116 ceph-mon[32207]: pgmap v152: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:04 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1387307873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:27:06.014 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:27:06.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:27:06.017 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:27:06.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 -- 172.21.15.116:0/3562994605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 msgr2=0x7f5ccc10d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 --2- 172.21.15.116:0/3562994605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc10d510 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009cd0 tx=0x7f5cc402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 -- 172.21.15.116:0/3562994605 shutdown_connections 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 --2- 172.21.15.116:0/3562994605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc10d510 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 --2- 172.21.15.116:0/3562994605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc101d00 0x7f5ccc1052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 --2- 172.21.15.116:0/3562994605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 0x7f5ccc101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 -- 172.21.15.116:0/3562994605 >> 172.21.15.116:0/3562994605 conn(0x7f5ccc075950 msgr2=0x7f5ccc075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 -- 172.21.15.116:0/3562994605 shutdown_connections 2024-03-24T09:27:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.137+0000 7f5cd291e640 1 -- 172.21.15.116:0/3562994605 wait complete. 2024-03-24T09:27:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 Processor -- start 2024-03-24T09:27:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 -- start start 2024-03-24T09:27:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 0x7f5ccc19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc197b20 0x7f5ccc197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc1985f0 con 0x7f5ccc1059d0 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc198760 con 0x7f5ccc197b20 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd291e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc198920 con 0x7f5ccc101350 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5ccb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5ccb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc19dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47730/0 (socket says 172.21.15.116:47730) 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 0x7f5ccc19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/383393702 learned_addr learned my addr 172.21.15.116:0/383393702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.138+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc197b20 0x7f5ccc197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/383393702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 msgr2=0x7f5ccc19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 0x7f5ccc19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/383393702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc197b20 msgr2=0x7f5ccc197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc197b20 0x7f5ccc197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/383393702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cbc009490 con 0x7f5ccc1059d0 2024-03-24T09:27:06.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 0x7f5ccc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc19dfa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00e970 tx=0x7f5cbc00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5cc97fa640 1 -- 172.21.15.116:0/383393702 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc00cd80 con 0x7f5ccc1059d0 2024-03-24T09:27:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc4009900 con 0x7f5ccc1059d0 2024-03-24T09:27:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5cc97fa640 1 -- 172.21.15.116:0/383393702 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cbc0044d0 con 0x7f5ccc1059d0 2024-03-24T09:27:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5cc97fa640 1 -- 172.21.15.116:0/383393702 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc010430 con 0x7f5ccc1059d0 2024-03-24T09:27:06.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.139+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc1a4c10 con 0x7f5ccc1059d0 2024-03-24T09:27:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.141+0000 7f5cc97fa640 1 -- 172.21.15.116:0/383393702 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f5cbc0036f0 con 0x7f5ccc1059d0 2024-03-24T09:27:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.141+0000 7f5cd55dc740 1 Processor -- start 2024-03-24T09:27:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.141+0000 7f5cd55dc740 1 -- start start 2024-03-24T09:27:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4713b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e471f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571e4681530 con 0x5571e4713760 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571e467ea60 con 0x5571e4714080 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5571e467a240 con 0x5571e471cc60 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd0ebb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4713b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd0ebb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4713b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47738/0 (socket says 172.21.15.116:47738) 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd0ebb640 1 -- 172.21.15.116:0/3745943390 learned_addr learned my addr 172.21.15.116:0/3745943390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cc97fa640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5cb40a2230 0x7f5cb40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e471f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5cb40a2230 0x7f5cb40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3745943390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 msgr2=0x5571e471f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e471f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3745943390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 msgr2=0x5571e4713b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4713b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3745943390 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571e46788c0 con 0x5571e4714080 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cc97fa640 1 -- 172.21.15.116:0/383393702 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5cbc014070 con 0x7f5ccc1059d0 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471c720 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009fd0 tx=0x7f5cc00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9a6ec81754aa57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cb27fc640 1 -- 172.21.15.116:0/3745943390 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0004310 con 0x5571e4714080 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.142+0000 7f5cb27fc640 1 -- 172.21.15.116:0/3745943390 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cc0004470 con 0x5571e4714080 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cb27fc640 1 -- 172.21.15.116:0/3745943390 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0004e20 con 0x5571e4714080 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3745943390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 msgr2=0x5571e471c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5cb40a2230 0x7f5cb40a46f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4032570 tx=0x7f5cc402c0f0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471c720 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009fd0 tx=0x7f5cc00118d0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3745943390 shutdown_connections 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e471f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471c720 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3745943390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4713b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3745943390 >> 172.21.15.116:0/3745943390 conn(0x5571e470de90 msgr2=0x5571e470e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3745943390 shutdown_connections 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3745943390 wait complete. 2024-03-24T09:27:06.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.143+0000 7f5cd55dc740 1 -- start start 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 Processor -- start 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e4713760 0x5571e4719c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 0x5571e471a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571e461ef80 con 0x5571e471cc60 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571e434d4b0 con 0x5571e4714080 2024-03-24T09:27:06.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5571e45ffad0 con 0x5571e4713760 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd55dc740 10 client.0 _ll_register_callbacks cb 0x5571e4632b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 0x5571e471a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 0x5571e471a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47748/0 (socket says 172.21.15.116:47748) 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 -- 172.21.15.116:0/1747292486 learned_addr learned my addr 172.21.15.116:0/1747292486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5ccb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e4713760 0x5571e4719c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 -- 172.21.15.116:0/1747292486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e4713760 msgr2=0x5571e4719c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e4713760 0x5571e4719c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 -- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 msgr2=0x5571e471a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5cd0ebb640 1 -- 172.21.15.116:0/1747292486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc0009c00 con 0x5571e471cc60 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.144+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 0x5571e471a6e0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4036000 tx=0x7f5cc40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51de4dac6c03d8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:06.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cab7fe640 1 -- 172.21.15.116:0/1747292486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40327d0 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cab7fe640 1 -- 172.21.15.116:0/1747292486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5cc4032930 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5571e469e7a0 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cab7fe640 1 -- 172.21.15.116:0/1747292486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5cc4003970 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 10 client.17133 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5571e46306d0 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cab7fe640 1 -- 172.21.15.116:0/1747292486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5571e469e7a0 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cab7fe640 1 -- 172.21.15.116:0/1747292486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f5cc404d430 con 0x5571e471cc60 2024-03-24T09:27:06.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 10 client.17133 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 20 client.17133 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e472ce40 0x5571e472f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 4 client.17133 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.145+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571e46701f0 con 0x5571e472ce40 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.146+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e472ce40 0x5571e472f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.146+0000 7f5cab7fe640 10 client.17133 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.146+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e472ce40 0x5571e472f260 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1402a33c224b3985 server_cookie=e8630e495cf53c9b in_seq=0 out_seq=0 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cab7fe640 1 -- 172.21.15.116:0/1747292486 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5571e46701f0 con 0x5571e472ce40 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cab7fe640 10 client.17133 handle_command_reply: tid=0 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 client.17133 shutdown 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 msgr2=0x5571e471a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 0x5571e471a6e0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4036000 tx=0x7f5cc40095c0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 shutdown_connections 2024-03-24T09:27:06.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e471cc60 0x5571e471a6e0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e472ce40 0x5571e472f260 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e471a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/1747292486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e4713760 0x5571e4719c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cab7fe640 0 client.17133 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 >> 172.21.15.116:0/1747292486 conn(0x5571e470de90 msgr2=0x5571e470fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 shutdown_connections 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 1 -- 172.21.15.116:0/1747292486 wait complete. 2024-03-24T09:27:06.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.147+0000 7f5cd55dc740 20 client.17133 trim_cache size 0 max 16384 2024-03-24T09:27:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 Processor -- start 2024-03-24T09:27:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 -- start start 2024-03-24T09:27:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4713760 0x5571e4728520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 0x5571e470cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4734b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571e4714ac0 con 0x5571e4714080 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571e472b550 con 0x5571e4713760 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.164+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5571e471c710 con 0x5571e471cc60 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 0x5571e470cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 0x5571e470cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47760/0 (socket says 172.21.15.116:47760) 2024-03-24T09:27:06.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/2785151783 learned_addr learned my addr 172.21.15.116:0/2785151783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4713760 0x5571e4728520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4734b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/2785151783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 msgr2=0x5571e4734b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4734b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/2785151783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4713760 msgr2=0x5571e4728520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4713760 0x5571e4728520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 -- 172.21.15.116:0/2785151783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571e4614c60 con 0x5571e4714080 2024-03-24T09:27:06.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4713760 0x5571e4728520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 0x5571e470cd70 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc09ab30 tx=0x7f5cbc097200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ef7add23195581 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5cb1ffb640 1 -- 172.21.15.116:0/2785151783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc09ef80 con 0x5571e4714080 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5cb1ffb640 1 -- 172.21.15.116:0/2785151783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cbc0a35d0 con 0x5571e4714080 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.165+0000 7f5cb1ffb640 1 -- 172.21.15.116:0/2785151783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc004210 con 0x5571e4714080 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- 172.21.15.116:0/2785151783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 msgr2=0x5571e470cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 0x5571e470cd70 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc09ab30 tx=0x7f5cbc097200 comp rx=0 tx=0).stop 2024-03-24T09:27:06.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- 172.21.15.116:0/2785151783 shutdown_connections 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4734b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4714080 0x5571e470cd70 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/2785151783 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4713760 0x5571e4728520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- 172.21.15.116:0/2785151783 >> 172.21.15.116:0/2785151783 conn(0x5571e470de90 msgr2=0x5571e470c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- 172.21.15.116:0/2785151783 shutdown_connections 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- 172.21.15.116:0/2785151783 wait complete. 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- start start 2024-03-24T09:27:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 Processor -- start 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4717040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e4717580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4792180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5571e47926c0 con 0x5571e4713760 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5571e4792800 con 0x5571e4714080 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.166+0000 7f5cd55dc740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5571e4792990 con 0x5571e471cc60 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cd55dc740 10 client.0 _ll_register_callbacks cb 0x7f5cc4032cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4717040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4717040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47766/0 (socket says 172.21.15.116:47766) 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3913046649 learned_addr learned my addr 172.21.15.116:0/3913046649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e4717580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cd0ebb640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4792180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3913046649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 msgr2=0x5571e4792180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4792180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 msgr2=0x5571e4717580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e4717580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 -- 172.21.15.116:0/3913046649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571e46788c0 con 0x5571e4713760 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5ccbfff640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4717040 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x5571e4713230 tx=0x7f5cc00045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bae1e64f9e3cc6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cab7fe640 1 -- 172.21.15.116:0/3913046649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc002c2b0 con 0x5571e4713760 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cab7fe640 1 -- 172.21.15.116:0/3913046649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5cc002b070 con 0x5571e4713760 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cc0004470 con 0x5571e4713760 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cab7fe640 1 -- 172.21.15.116:0/3913046649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5cc002c410 con 0x5571e4713760 2024-03-24T09:27:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cd55dc740 10 client.17139 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x7f5ccc1a4c10 con 0x5571e4713760 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.167+0000 7f5cab7fe640 1 -- 172.21.15.116:0/3913046649 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5cc0004470 con 0x5571e4713760 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cab7fe640 1 -- 172.21.15.116:0/3913046649 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f5cc0021040 con 0x5571e4713760 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cd55dc740 10 client.17139 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cd55dc740 10 client.17139 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cd55dc740 20 client.17139 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e4794250 0x5571e4796670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cd55dc740 4 client.17139 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5571e4379860 con 0x5571e4794250 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e4794250 0x5571e4796670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5ccb7fe640 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e4794250 0x5571e4796670 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6bc4fb44f3b825f server_cookie=e34920ad1edfb448 in_seq=0 out_seq=0 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.168+0000 7f5cab7fe640 10 client.17139 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.169+0000 7f5cab7fe640 1 -- 172.21.15.116:0/3913046649 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5571e4379860 con 0x5571e4794250 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.169+0000 7f5cab7fe640 10 client.17139 handle_command_reply: tid=0 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.169+0000 7f5cd55dc740 1 client.17139 shutdown 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 msgr2=0x5571e4717040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4717040 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x5571e4713230 tx=0x7f5cc00045e0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 shutdown_connections 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5571e471cc60 0x5571e4792180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5571e4794250 0x5571e4796670 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5571e4714080 0x5571e4717580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 --2- 172.21.15.116:0/3913046649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5571e4713760 0x5571e4717040 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cab7fe640 0 client.17139 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 >> 172.21.15.116:0/3913046649 conn(0x5571e470de90 msgr2=0x5571e470cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 shutdown_connections 2024-03-24T09:27:06.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 1 -- 172.21.15.116:0/3913046649 wait complete. 2024-03-24T09:27:06.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd55dc740 20 client.17139 trim_cache size 0 max 16384 2024-03-24T09:27:06.182 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:27:06.182 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:27:06.183 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "164acb15-db9c-4ebc-97d6-ff5acf32e238", 2024-03-24T09:27:06.183 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:27:06.183 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:27:06.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5cb40a2230 msgr2=0x7f5cb40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5cb40a2230 0x7f5cb40a46f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4032570 tx=0x7f5cc402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 msgr2=0x7f5ccc19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc19dfa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc00e970 tx=0x7f5cbc00ee30 comp rx=0 tx=0).stop 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 shutdown_connections 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc197b20 0x7f5ccc197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ccc1059d0 0x7f5ccc19dfa0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5cb40a2230 0x7f5cb40a46f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 --2- 172.21.15.116:0/383393702 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ccc101350 0x7f5ccc19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 >> 172.21.15.116:0/383393702 conn(0x7f5ccc075950 msgr2=0x7f5ccc104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.170+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 shutdown_connections 2024-03-24T09:27:06.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.171+0000 7f5cd291e640 1 -- 172.21.15.116:0/383393702 wait complete. 2024-03-24T09:27:06.186 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:27:06.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:27:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 -- 172.21.15.116:0/1918538324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8694100f80 msgr2=0x7f869410d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 --2- 172.21.15.116:0/1918538324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8694100f80 0x7f869410d440 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f868c009be0 tx=0x7f868c02bef0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 -- 172.21.15.116:0/1918538324 shutdown_connections 2024-03-24T09:27:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 --2- 172.21.15.116:0/1918538324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8694100f80 0x7f869410d440 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 --2- 172.21.15.116:0/1918538324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86941005c0 0x7f8694100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 --2- 172.21.15.116:0/1918538324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8694106680 0x7f8694106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 -- 172.21.15.116:0/1918538324 >> 172.21.15.116:0/1918538324 conn(0x7f86940773c0 msgr2=0x7f86940fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 -- 172.21.15.116:0/1918538324 shutdown_connections 2024-03-24T09:27:06.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.308+0000 7f869891e640 1 -- 172.21.15.116:0/1918538324 wait complete. 2024-03-24T09:27:06.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 Processor -- start 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 -- start start 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 0x7f869410dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8694100f80 0x7f869410e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8694106680 0x7f869410e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86941a6020 con 0x7f86941005c0 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f86941a6190 con 0x7f8694100f80 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869891e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f86941a6350 con 0x7f8694106680 2024-03-24T09:27:06.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 0x7f869410dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 0x7f869410dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47792/0 (socket says 172.21.15.116:47792) 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.309+0000 7f869259c640 1 -- 172.21.15.116:0/1393912631 learned_addr learned my addr 172.21.15.116:0/1393912631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f868bfff640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8694100f80 0x7f869410e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869259c640 1 -- 172.21.15.116:0/1393912631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8694106680 msgr2=0x7f869410e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869259c640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8694106680 0x7f869410e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869259c640 1 -- 172.21.15.116:0/1393912631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8694100f80 msgr2=0x7f869410e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869259c640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8694100f80 0x7f869410e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869259c640 1 -- 172.21.15.116:0/1393912631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8684009560 con 0x7f86941005c0 2024-03-24T09:27:06.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869259c640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 0x7f869410dcd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f868400e9c0 tx=0x7f868400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f868b7fe640 1 -- 172.21.15.116:0/1393912631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868400cde0 con 0x7f86941005c0 2024-03-24T09:27:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f868b7fe640 1 -- 172.21.15.116:0/1393912631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86840044d0 con 0x7f86941005c0 2024-03-24T09:27:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868c009840 con 0x7f86941005c0 2024-03-24T09:27:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f868b7fe640 1 -- 172.21.15.116:0/1393912631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684010430 con 0x7f86941005c0 2024-03-24T09:27:06.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.310+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86941a69b0 con 0x7f86941005c0 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.313+0000 7f869b5d9740 1 Processor -- start 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 -- start start 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f868b7fe640 1 -- 172.21.15.116:0/1393912631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f8684004000 con 0x7f86941005c0 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309c1d0 0x5654a309c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30a7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5654a2fc0f30 con 0x5654a309caf0 2024-03-24T09:27:06.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5654a300a810 con 0x5654a30a56d0 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869b5d9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5654a3007d40 con 0x5654a309c1d0 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30a5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47800/0 (socket says 172.21.15.116:47800) 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 -- 172.21.15.116:0/757437153 learned_addr learned my addr 172.21.15.116:0/757437153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f868bfff640 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30a7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 -- 172.21.15.116:0/757437153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309c1d0 msgr2=0x5654a309c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309c1d0 0x5654a309c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 -- 172.21.15.116:0/757437153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 msgr2=0x5654a30a7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30a7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.314+0000 7f869259c640 1 -- 172.21.15.116:0/757437153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a2fc4ea0 con 0x5654a309caf0 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f868b7fe640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86800a2230 0x7f86800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f8692d9d640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86800a2230 0x7f86800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f868b7fe640 1 -- 172.21.15.116:0/1393912631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8684014070 con 0x7f86941005c0 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f869259c640 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30a5190 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f86840cbb90 tx=0x7f86840d4ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f377dbec363363bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f8688ff9640 1 -- 172.21.15.116:0/757437153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684009400 con 0x5654a309caf0 2024-03-24T09:27:06.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f8688ff9640 1 -- 172.21.15.116:0/757437153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86840d3070 con 0x5654a309caf0 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.315+0000 7f8688ff9640 1 -- 172.21.15.116:0/757437153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868409acd0 con 0x5654a309caf0 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 -- 172.21.15.116:0/757437153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 msgr2=0x5654a30a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30a5190 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f86840cbb90 tx=0x7f86840d4ac0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f8692d9d640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86800a2230 0x7f86800a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f868c036040 tx=0x7f868c009330 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 -- 172.21.15.116:0/757437153 shutdown_connections 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30a7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30a5190 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 --2- 172.21.15.116:0/757437153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309c1d0 0x5654a309c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 -- 172.21.15.116:0/757437153 >> 172.21.15.116:0/757437153 conn(0x5654a3096900 msgr2=0x5654a3096ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 -- 172.21.15.116:0/757437153 shutdown_connections 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.316+0000 7f869b5d9740 1 -- 172.21.15.116:0/757437153 wait complete. 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 -- start start 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 Processor -- start 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309caf0 0x5654a30a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30a3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5654a3003520 con 0x5654a309c1d0 2024-03-24T09:27:06.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5654a2cd6430 con 0x5654a309caf0 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5654a2c92e00 con 0x5654a30a56d0 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869b5d9740 10 client.0 _ll_register_callbacks cb 0x5654a2fa1d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30a27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47816/0 (socket says 172.21.15.116:47816) 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f869259c640 1 -- 172.21.15.116:0/980496297 learned_addr learned my addr 172.21.15.116:0/980496297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f868bfff640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309caf0 0x5654a30a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.317+0000 7f8692d9d640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869259c640 1 -- 172.21.15.116:0/980496297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 msgr2=0x5654a30a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869259c640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869259c640 1 -- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309caf0 msgr2=0x5654a30a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869259c640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309caf0 0x5654a30a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869259c640 1 -- 172.21.15.116:0/980496297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86941a69b0 con 0x5654a309c1d0 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f868bfff640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309caf0 0x5654a30a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869259c640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30a27e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8684000c00 tx=0x7f8684097760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f869eb4d820a7dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f8679ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:06.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f8679ffb640 1 -- 172.21.15.116:0/980496297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684009400 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f8679ffb640 1 -- 172.21.15.116:0/980496297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8684003760 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5654a2d027e0 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f8679ffb640 1 -- 172.21.15.116:0/980496297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f868409a820 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869b5d9740 10 client.17151 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5654a3001ba0 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.318+0000 7f8679ffb640 1 -- 172.21.15.116:0/980496297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5654a2d027e0 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f8679ffb640 1 -- 172.21.15.116:0/980496297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f868401f060 con 0x5654a309c1d0 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f869b5d9740 10 client.17151 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:06.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f869b5d9740 10 client.17151 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f869b5d9740 20 client.17151 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f869b5d9740 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a30c0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f869b5d9740 4 client.17151 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654a2d3f720 con 0x5654a30be0c0 2024-03-24T09:27:06.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.319+0000 7f868bfff640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a30c0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.320+0000 7f868bfff640 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a30c0540 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d18e95f28a1c0cb server_cookie=864b2349c4c9250a in_seq=0 out_seq=0 2024-03-24T09:27:06.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.320+0000 7f8679ffb640 10 client.17151 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f8679ffb640 1 -- 172.21.15.116:0/980496297 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5654a2d3f720 con 0x5654a30be0c0 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f8679ffb640 10 client.17151 handle_command_reply: tid=0 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 client.17151 shutdown 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 msgr2=0x5654a30a27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30a27e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8684000c00 tx=0x7f8684097760 comp rx=0 tx=0).stop 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 shutdown_connections 2024-03-24T09:27:06.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a30c0540 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309caf0 0x5654a30a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 --2- 172.21.15.116:0/980496297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30a27e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f8679ffb640 0 client.17151 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.321+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 >> 172.21.15.116:0/980496297 conn(0x5654a3096900 msgr2=0x5654a3098650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.322+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 shutdown_connections 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.322+0000 7f869b5d9740 1 -- 172.21.15.116:0/980496297 wait complete. 2024-03-24T09:27:06.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.322+0000 7f869b5d9740 20 client.17151 trim_cache size 0 max 16384 2024-03-24T09:27:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 Processor -- start 2024-03-24T09:27:06.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 -- start start 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309c1d0 0x5654a30a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30b97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30959e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5654a2f88430 con 0x5654a309caf0 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5654a309b5f0 con 0x5654a309c1d0 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869b5d9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5654a30baed0 con 0x5654a30a56d0 2024-03-24T09:27:06.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30b97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30b97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47822/0 (socket says 172.21.15.116:47822) 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 -- 172.21.15.116:0/3543598577 learned_addr learned my addr 172.21.15.116:0/3543598577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f8692d9d640 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f869259c640 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309c1d0 0x5654a30a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 -- 172.21.15.116:0/3543598577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 msgr2=0x5654a30959e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30959e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 -- 172.21.15.116:0/3543598577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309c1d0 msgr2=0x5654a30a4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309c1d0 0x5654a30a4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 -- 172.21.15.116:0/3543598577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a2f9db20 con 0x5654a309caf0 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.345+0000 7f868bfff640 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30b97f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x5654a3097a90 tx=0x7f867c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ec800f89e7a53d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.346+0000 7f867bfff640 1 -- 172.21.15.116:0/3543598577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c017040 con 0x5654a309caf0 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.346+0000 7f867bfff640 1 -- 172.21.15.116:0/3543598577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f867c018040 con 0x5654a309caf0 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.346+0000 7f867bfff640 1 -- 172.21.15.116:0/3543598577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867c00fdc0 con 0x5654a309caf0 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 -- 172.21.15.116:0/3543598577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 msgr2=0x5654a30b97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30b97f0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x5654a3097a90 tx=0x7f867c005f90 comp rx=0 tx=0).stop 2024-03-24T09:27:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 -- 172.21.15.116:0/3543598577 shutdown_connections 2024-03-24T09:27:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a30a56d0 0x5654a30959e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309caf0 0x5654a30b97f0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 --2- 172.21.15.116:0/3543598577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a309c1d0 0x5654a30a4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 -- 172.21.15.116:0/3543598577 >> 172.21.15.116:0/3543598577 conn(0x5654a3096900 msgr2=0x5654a30952d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 -- 172.21.15.116:0/3543598577 shutdown_connections 2024-03-24T09:27:06.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 -- 172.21.15.116:0/3543598577 wait complete. 2024-03-24T09:27:06.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.347+0000 7f869b5d9740 1 -- start start 2024-03-24T09:27:06.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 Processor -- start 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309caf0 0x5654a30b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30b2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5654a3123440 con 0x5654a309c1d0 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5654a3123580 con 0x5654a30a56d0 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5654a31236c0 con 0x5654a309caf0 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869b5d9740 10 client.0 _ll_register_callbacks cb 0x7f867c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f868bfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309caf0 0x5654a30b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f8692d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f868bfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309caf0 0x5654a30b1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:35866/0 (socket says 172.21.15.116:35866) 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f868bfff640 1 -- 172.21.15.116:0/2951663567 learned_addr learned my addr 172.21.15.116:0/2951663567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.348+0000 7f869259c640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869259c640 1 -- 172.21.15.116:0/2951663567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309caf0 msgr2=0x5654a30b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869259c640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309caf0 0x5654a30b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869259c640 1 -- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 msgr2=0x5654a30b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869259c640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869259c640 1 -- 172.21.15.116:0/2951663567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f867c012690 con 0x5654a309c1d0 2024-03-24T09:27:06.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f8692d9d640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869259c640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30b16d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f868409ae50 tx=0x7f86840d63c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe401252c7e99bed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f8679ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f8679ffb640 1 -- 172.21.15.116:0/2951663567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684009400 con 0x5654a309c1d0 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f8679ffb640 1 -- 172.21.15.116:0/2951663567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8684003760 con 0x5654a309c1d0 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86840d3070 con 0x5654a309c1d0 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f8679ffb640 1 -- 172.21.15.116:0/2951663567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f868409aa00 con 0x5654a309c1d0 2024-03-24T09:27:06.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869b5d9740 10 client.17157 fetch_fsmap learned FSMap version 32 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.349+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5654a2fc3700 con 0x5654a309c1d0 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f8679ffb640 1 -- 172.21.15.116:0/2951663567 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86840d3070 con 0x5654a309c1d0 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f8679ffb640 1 -- 172.21.15.116:0/2951663567 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f868401f050 con 0x5654a309c1d0 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f869b5d9740 10 client.17157 fetch_fsmap finished waiting for FSMap version 32 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f869b5d9740 10 client.17157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f869b5d9740 20 client.17157 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f869b5d9740 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a3126b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f869b5d9740 4 client.17157 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:27:06.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5654a2d8e890 con 0x5654a30be0c0 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.350+0000 7f868bfff640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a3126b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f868bfff640 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a3126b00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63860461d8fcd703 server_cookie=52ecb7821a55ed3c in_seq=0 out_seq=0 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f8679ffb640 10 client.17157 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f8679ffb640 1 -- 172.21.15.116:0/2951663567 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5654a2d8e890 con 0x5654a30be0c0 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f8679ffb640 10 client.17157 handle_command_reply: tid=0 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f869b5d9740 1 client.17157 shutdown 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 msgr2=0x5654a30b16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.351+0000 7f869b5d9740 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30b16d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f868409ae50 tx=0x7f86840d63c0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 shutdown_connections 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5654a30be0c0 0x5654a3126b00 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5654a30a56d0 0x5654a30b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5654a309caf0 0x5654a30b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 --2- 172.21.15.116:0/2951663567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5654a309c1d0 0x5654a30b16d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 >> 172.21.15.116:0/2951663567 conn(0x5654a3096900 msgr2=0x5654a30952d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f8679ffb640 0 client.17157 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 shutdown_connections 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 1 -- 172.21.15.116:0/2951663567 wait complete. 2024-03-24T09:27:06.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869b5d9740 20 client.17157 trim_cache size 0 max 16384 2024-03-24T09:27:06.362 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:27:06.362 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:27:06.362 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:27:06.362 INFO:teuthology.orchestra.run.smithi116.stdout: "164acb15-db9c-4ebc-97d6-ff5acf32e238": { 2024-03-24T09:27:06.362 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:27:06.363 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "164acb15-db9c-4ebc-97d6-ff5acf32e238", 2024-03-24T09:27:06.363 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:27:06.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:27:06.363 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:27:06.363 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:27:06.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86800a2230 msgr2=0x7f86800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86800a2230 0x7f86800a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f868c036040 tx=0x7f868c009330 comp rx=0 tx=0).stop 2024-03-24T09:27:06.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 msgr2=0x7f869410dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:06.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 0x7f869410dcd0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f868400e9c0 tx=0x7f868400ee80 comp rx=0 tx=0).stop 2024-03-24T09:27:06.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 shutdown_connections 2024-03-24T09:27:06.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86800a2230 0x7f86800a46f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8694106680 0x7f869410e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8694100f80 0x7f869410e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 --2- 172.21.15.116:0/1393912631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86941005c0 0x7f869410dcd0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:06.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 >> 172.21.15.116:0/1393912631 conn(0x7f86940773c0 msgr2=0x7f86940fed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:06.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 shutdown_connections 2024-03-24T09:27:06.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:06.352+0000 7f869891e640 1 -- 172.21.15.116:0/1393912631 wait complete. 2024-03-24T09:27:06.367 INFO:tasks.cephfs.filesystem:scrub status for tag:164acb15-db9c-4ebc-97d6-ff5acf32e238 - {'path': '/', 'tag': '164acb15-db9c-4ebc-97d6-ff5acf32e238', 'options': 'recursive,force'} 2024-03-24T09:27:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:06 smithi143 ceph-mon[35965]: pgmap v153: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:06 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:06 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:07.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:06 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:07.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:06 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:07.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:06 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:06 smithi117 ceph-mon[35788]: pgmap v153: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:06 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:06 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:06 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:06 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:07.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:06 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:06 smithi116 ceph-mon[32207]: pgmap v153: 129 pgs: 129 active+clean; 152 MiB data, 902 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:06 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:06 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:06 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:07.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:06 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:07.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:06 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:27:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:08.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:08.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:07 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:27:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:08.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:08.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:07 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:27:08.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:08.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:08.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:08.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:08.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:07 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:08 smithi143 ceph-mon[35965]: pgmap v154: 129 pgs: 129 active+clean; 152 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:08 smithi117 ceph-mon[35788]: pgmap v154: 129 pgs: 129 active+clean; 152 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:09.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:08 smithi116 ceph-mon[32207]: pgmap v154: 129 pgs: 129 active+clean; 152 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:10.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:09 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:09 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:10.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:09 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:11.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:10 smithi143 ceph-mon[35965]: pgmap v155: 129 pgs: 129 active+clean; 152 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:10 smithi117 ceph-mon[35788]: pgmap v155: 129 pgs: 129 active+clean; 152 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:11.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:10 smithi116 ceph-mon[32207]: pgmap v155: 129 pgs: 129 active+clean; 152 MiB data, 906 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:27:13.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:12 smithi143 ceph-mon[35965]: pgmap v156: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:12 smithi117 ceph-mon[35788]: pgmap v156: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:12 smithi116 ceph-mon[32207]: pgmap v156: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:27:15.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:14 smithi143 ceph-mon[35965]: pgmap v157: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:27:15.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:14 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:15.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:14 smithi117 ceph-mon[35788]: pgmap v157: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:27:15.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:14 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:14 smithi116 ceph-mon[32207]: pgmap v157: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:27:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:14 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:16.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:27:16.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:27:16.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:27:16.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:15 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:27:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:27:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:27:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:27:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:15 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:27:16.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:27:16.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:27:16.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:27:16.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:15 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:27:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:16 smithi143 ceph-mon[35965]: pgmap v158: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:27:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:16 smithi117 ceph-mon[35788]: pgmap v158: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:27:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:16 smithi116 ceph-mon[32207]: pgmap v158: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:27:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:18 smithi143 ceph-mon[35965]: pgmap v159: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:27:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:18 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:18 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:18 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:19.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:18 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:19.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:18 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:18 smithi117 ceph-mon[35788]: pgmap v159: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:27:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:18 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:18 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:18 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:18 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:18 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:18 smithi116 ceph-mon[32207]: pgmap v159: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:27:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:18 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:18 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:18 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:18 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:18 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:20.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:19 smithi143 ceph-mon[35965]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:19 smithi117 ceph-mon[35788]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:19 smithi116 ceph-mon[32207]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T09:27:21.218 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:20 smithi117 ceph-mon[35788]: pgmap v160: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:27:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:20 smithi143 ceph-mon[35965]: pgmap v160: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:27:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:20 smithi116 ceph-mon[32207]: pgmap v160: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:27:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:22 smithi143 ceph-mon[35965]: pgmap v161: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:27:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:22 smithi117 ceph-mon[35788]: pgmap v161: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:27:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:22 smithi116 ceph-mon[32207]: pgmap v161: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:27:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:24 smithi143 ceph-mon[35965]: pgmap v162: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:24 smithi117 ceph-mon[35788]: pgmap v162: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:25.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:24 smithi116 ceph-mon[32207]: pgmap v162: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:26 smithi143 ceph-mon[35965]: pgmap v163: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:26 smithi117 ceph-mon[35788]: pgmap v163: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:27.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:26 smithi116 ceph-mon[32207]: pgmap v163: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:28 smithi143 ceph-mon[35965]: pgmap v164: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:28 smithi117 ceph-mon[35788]: pgmap v164: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:29.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:28 smithi116 ceph-mon[32207]: pgmap v164: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:30 smithi143 ceph-mon[35965]: pgmap v165: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:30 smithi117 ceph-mon[35788]: pgmap v165: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:30 smithi116 ceph-mon[32207]: pgmap v165: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:32 smithi143 ceph-mon[35965]: pgmap v166: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:32 smithi117 ceph-mon[35788]: pgmap v166: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:33.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:32 smithi116 ceph-mon[32207]: pgmap v166: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:35.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:34 smithi143 ceph-mon[35965]: pgmap v167: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:34 smithi117 ceph-mon[35788]: pgmap v167: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:34 smithi116 ceph-mon[32207]: pgmap v167: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:36.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:27:36.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:27:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 -- 172.21.15.116:0/3389479720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81016a0 msgr2=0x7f3db8101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/3389479720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81016a0 0x7f3db8101b20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009930 tx=0x7f3dac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:27:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 -- 172.21.15.116:0/3389479720 shutdown_connections 2024-03-24T09:27:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/3389479720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3db8102060 0x7f3db810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/3389479720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81016a0 0x7f3db8101b20 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/3389479720 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db81076b0 0x7f3db8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 -- 172.21.15.116:0/3389479720 >> 172.21.15.116:0/3389479720 conn(0x7f3db8078b10 msgr2=0x7f3db8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 -- 172.21.15.116:0/3389479720 shutdown_connections 2024-03-24T09:27:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.483+0000 7f3dbd91e640 1 -- 172.21.15.116:0/3389479720 wait complete. 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 Processor -- start 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 -- start start 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3db81016a0 0x7f3db819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db8102060 0x7f3db819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 0x7f3db8197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8198100 con 0x7f3db81076b0 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8198270 con 0x7f3db81016a0 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3dbd91e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8198430 con 0x7f3db8102060 2024-03-24T09:27:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 0x7f3db8197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3db67fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db8102060 0x7f3db819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 0x7f3db8197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52652/0 (socket says 172.21.15.116:52652) 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3db77fe640 1 -- 172.21.15.116:0/1320474286 learned_addr learned my addr 172.21.15.116:0/1320474286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.484+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3db81016a0 0x7f3db819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3db77fe640 1 -- 172.21.15.116:0/1320474286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db8102060 msgr2=0x7f3db819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3db77fe640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db8102060 0x7f3db819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3db77fe640 1 -- 172.21.15.116:0/1320474286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3db81016a0 msgr2=0x7f3db819d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3db77fe640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3db81016a0 0x7f3db819d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3db77fe640 1 -- 172.21.15.116:0/1320474286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac009560 con 0x7f3db81076b0 2024-03-24T09:27:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3db77fe640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 0x7f3db8197ab0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3db000e970 tx=0x7f3db000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3dbc91c640 1 -- 172.21.15.116:0/1320474286 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0009720 con 0x7f3db81076b0 2024-03-24T09:27:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db8198710 con 0x7f3db81076b0 2024-03-24T09:27:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3dbc91c640 1 -- 172.21.15.116:0/1320474286 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3db00054d0 con 0x7f3db81076b0 2024-03-24T09:27:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3dbc91c640 1 -- 172.21.15.116:0/1320474286 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0010430 con 0x7f3db81076b0 2024-03-24T09:27:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.485+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db81a49a0 con 0x7f3db81076b0 2024-03-24T09:27:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.487+0000 7f3dbc91c640 1 -- 172.21.15.116:0/1320474286 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f3db0004f30 con 0x7f3db81076b0 2024-03-24T09:27:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 Processor -- start 2024-03-24T09:27:36.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 -- start start 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dbc91c640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3da00a2230 0x7f3da00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 0x55da2b7d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d0af0 0x55da2b7d9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7dbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da2b6f4f30 con 0x55da2b7d96d0 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da2b73e810 con 0x55da2b7d01d0 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dc05db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55da2b73bd40 con 0x55da2b7d0af0 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3db67fc640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3da00a2230 0x7f3da00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3db6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 0x55da2b7d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7dbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3db6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 0x55da2b7d05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:47672/0 (socket says 172.21.15.116:47672) 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3db6ffd640 1 -- 172.21.15.116:0/1704545565 learned_addr learned my addr 172.21.15.116:0/1704545565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3dbc91c640 1 -- 172.21.15.116:0/1320474286 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3db0014070 con 0x7f3db81076b0 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.488+0000 7f3db67fc640 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d0af0 0x55da2b7d9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db6ffd640 1 -- 172.21.15.116:0/1704545565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d0af0 msgr2=0x55da2b7d9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d0af0 0x55da2b7d9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db6ffd640 1 -- 172.21.15.116:0/1704545565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 msgr2=0x55da2b7dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7dbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db6ffd640 1 -- 172.21.15.116:0/1704545565 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2b6f8ea0 con 0x55da2b7d01d0 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db77fe640 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7dbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db67fc640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3da00a2230 0x7f3da00a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005ab0 tx=0x7f3dac005dc0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 0x55da2b7d05b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009fb0 tx=0x7f3da80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8de2302f420c57aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3d9dffb640 1 -- 172.21.15.116:0/1704545565 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8004d20 con 0x55da2b7d01d0 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3d9dffb640 1 -- 172.21.15.116:0/1704545565 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3da8004e80 con 0x55da2b7d01d0 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.489+0000 7f3d9dffb640 1 -- 172.21.15.116:0/1704545565 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8005020 con 0x55da2b7d01d0 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 -- 172.21.15.116:0/1704545565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 msgr2=0x55da2b7d05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 0x55da2b7d05b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009fb0 tx=0x7f3da80118d0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 -- 172.21.15.116:0/1704545565 shutdown_connections 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7dbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d0af0 0x55da2b7d9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 --2- 172.21.15.116:0/1704545565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d01d0 0x55da2b7d05b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 -- 172.21.15.116:0/1704545565 >> 172.21.15.116:0/1704545565 conn(0x55da2b7ca900 msgr2=0x55da2b7cace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 -- 172.21.15.116:0/1704545565 shutdown_connections 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 -- 172.21.15.116:0/1704545565 wait complete. 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 -- start start 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.490+0000 7f3dc05db740 1 Processor -- start 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da2b737520 con 0x55da2b7d0af0 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da2b40a430 con 0x55da2b7d96d0 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55da2b3c6e00 con 0x55da2b7d01d0 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 10 client.0 _ll_register_callbacks cb 0x55da2b6d5d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52656/0 (socket says 172.21.15.116:52656) 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 -- 172.21.15.116:0/2179847869 learned_addr learned my addr 172.21.15.116:0/2179847869 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 -- 172.21.15.116:0/2179847869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 msgr2=0x55da2b7d66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 -- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 msgr2=0x55da2b7d7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 -- 172.21.15.116:0/2179847869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da8009be0 con 0x55da2b7d0af0 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db77fe640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:36.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3db67fc640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6c20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009900 tx=0x7f3dac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798ebbc82d5ba664 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3d96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2179847869 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac02cb60 con 0x55da2b7d0af0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2179847869 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3dac003800 con 0x55da2b7d0af0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55da2b473720 con 0x55da2b7d0af0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2179847869 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3dac0039c0 con 0x55da2b7d0af0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 10 client.17169 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.491+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55da2b735ba0 con 0x55da2b7d0af0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2179847869 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55da2b473720 con 0x55da2b7d0af0 2024-03-24T09:27:36.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2179847869 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3dac03c070 con 0x55da2b7d0af0 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3dc05db740 10 client.17169 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3dc05db740 10 client.17169 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3dc05db740 20 client.17169 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b7e9bf0 0x55da2b7ec010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3dc05db740 4 client.17169 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da2b4c2890 con 0x55da2b7e9bf0 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b7e9bf0 0x55da2b7ec010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3d96ffd640 10 client.17169 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:36.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.492+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b7e9bf0 0x55da2b7ec010 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc00bc14b878f1b3 server_cookie=83f8c6ebf3b69722 in_seq=0 out_seq=0 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2179847869 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da2b4c2890 con 0x55da2b7e9bf0 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3d96ffd640 10 client.17169 handle_command_reply: tid=0 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 client.17169 shutdown 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 msgr2=0x55da2b7d6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6c20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009900 tx=0x7f3dac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 shutdown_connections 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b7e9bf0 0x55da2b7ec010 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6c20 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2179847869 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3d96ffd640 0 client.17169 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 >> 172.21.15.116:0/2179847869 conn(0x55da2b7ca900 msgr2=0x55da2b7cc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 shutdown_connections 2024-03-24T09:27:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 1 -- 172.21.15.116:0/2179847869 wait complete. 2024-03-24T09:27:36.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.493+0000 7f3dc05db740 20 client.17169 trim_cache size 0 max 16384 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.510+0000 7f3dc05db740 1 Processor -- start 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 -- start start 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7e4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da2b6bc430 con 0x55da2b7d96d0 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da2b7d2300 con 0x55da2b7d0af0 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3dc05db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55da2b7cb900 con 0x55da2b7d01d0 2024-03-24T09:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52668/0 (socket says 172.21.15.116:52668) 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db6ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7e4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 -- 172.21.15.116:0/268619316 learned_addr learned my addr 172.21.15.116:0/268619316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db67fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 -- 172.21.15.116:0/268619316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 msgr2=0x55da2b7e4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7e4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 -- 172.21.15.116:0/268619316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d0af0 msgr2=0x55da2b7d6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 -- 172.21.15.116:0/268619316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2b6d1b20 con 0x55da2b7d96d0 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3db77fe640 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7f18e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3db00cbc60 tx=0x7f3db009ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=740cfcc4311b5d96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3d9d7fa640 1 -- 172.21.15.116:0/268619316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db001c1c0 con 0x55da2b7d96d0 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3d9d7fa640 1 -- 172.21.15.116:0/268619316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3db000ce30 con 0x55da2b7d96d0 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.511+0000 7f3d9d7fa640 1 -- 172.21.15.116:0/268619316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db00991e0 con 0x55da2b7d96d0 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 -- 172.21.15.116:0/268619316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 msgr2=0x55da2b7f18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7f18e0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3db00cbc60 tx=0x7f3db009ac20 comp rx=0 tx=0).stop 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 -- 172.21.15.116:0/268619316 shutdown_connections 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d96d0 0x55da2b7f18e0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d0af0 0x55da2b7d6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 --2- 172.21.15.116:0/268619316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7e4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 -- 172.21.15.116:0/268619316 >> 172.21.15.116:0/268619316 conn(0x55da2b7ca900 msgr2=0x55da2b7c9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 -- 172.21.15.116:0/268619316 shutdown_connections 2024-03-24T09:27:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 -- 172.21.15.116:0/268619316 wait complete. 2024-03-24T09:27:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.512+0000 7f3dc05db740 1 -- start start 2024-03-24T09:27:36.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 Processor -- start 2024-03-24T09:27:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da2b84f080 con 0x55da2b7d0af0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55da2b84f1f0 con 0x55da2b7d96d0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55da2b84f3b0 con 0x55da2b7d01d0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 10 client.0 _ll_register_callbacks cb 0x7f3da80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52682/0 (socket says 172.21.15.116:52682) 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 -- 172.21.15.116:0/2937970809 learned_addr learned my addr 172.21.15.116:0/2937970809 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db77fe640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 -- 172.21.15.116:0/2937970809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 msgr2=0x55da2b7d3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 -- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 msgr2=0x55da2b7d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 -- 172.21.15.116:0/2937970809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db8198710 con 0x55da2b7d0af0 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3db67fc640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d3a50 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x55da2b7d7cb0 tx=0x7f3dac0097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b19432422ab2ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3d96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2937970809 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac042470 con 0x55da2b7d0af0 2024-03-24T09:27:36.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55da2b4367e0 con 0x55da2b7d0af0 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.513+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2937970809 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3dac02cb60 con 0x55da2b7d0af0 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2937970809 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3dac0056e0 con 0x55da2b7d0af0 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 10 client.17175 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55da2b6f7700 con 0x55da2b7d0af0 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2937970809 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55da2b4367e0 con 0x55da2b7d0af0 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2937970809 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3dac03c040 con 0x55da2b7d0af0 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 10 client.17175 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 10 client.17175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:36.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 20 client.17175 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b8509b0 0x55da2b852dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 4 client.17175 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55da2b3e2700 con 0x55da2b8509b0 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b8509b0 0x55da2b852dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3d96ffd640 10 client.17175 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.514+0000 7f3db6ffd640 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b8509b0 0x55da2b852dd0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b610dd3e4219b783 server_cookie=ec7600133c29832d in_seq=0 out_seq=0 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3d96ffd640 1 -- 172.21.15.116:0/2937970809 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55da2b3e2700 con 0x55da2b8509b0 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3d96ffd640 10 client.17175 handle_command_reply: tid=0 2024-03-24T09:27:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 client.17175 shutdown 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 msgr2=0x55da2b7d3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d3a50 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x55da2b7d7cb0 tx=0x7f3dac0097a0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 shutdown_connections 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55da2b8509b0 0x55da2b852dd0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55da2b7d96d0 0x55da2b7d3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da2b7d0af0 0x55da2b7d3a50 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3d96ffd640 0 client.17175 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 --2- 172.21.15.116:0/2937970809 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55da2b7d01d0 0x55da2b7d3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 >> 172.21.15.116:0/2937970809 conn(0x55da2b7ca900 msgr2=0x55da2b7cba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 shutdown_connections 2024-03-24T09:27:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 1 -- 172.21.15.116:0/2937970809 wait complete. 2024-03-24T09:27:36.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dc05db740 20 client.17175 trim_cache size 0 max 16384 2024-03-24T09:27:36.525 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:27:36.525 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:27:36.525 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:27:36.525 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:27:36.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3da00a2230 msgr2=0x7f3da00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3da00a2230 0x7f3da00a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3dac005ab0 tx=0x7f3dac005dc0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 msgr2=0x7f3db8197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.515+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 0x7f3db8197ab0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3db000e970 tx=0x7f3db000ed70 comp rx=0 tx=0).stop 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 shutdown_connections 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db81076b0 0x7f3db8197ab0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3da00a2230 0x7f3da00a46f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3db8102060 0x7f3db819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 --2- 172.21.15.116:0/1320474286 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3db81016a0 0x7f3db819d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 >> 172.21.15.116:0/1320474286 conn(0x7f3db8078b10 msgr2=0x7f3db81041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 shutdown_connections 2024-03-24T09:27:36.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.516+0000 7f3dbd91e640 1 -- 172.21.15.116:0/1320474286 wait complete. 2024-03-24T09:27:36.528 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:27:36.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:27:36.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.629+0000 7ffb55b13640 1 -- 172.21.15.116:0/259409371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50106fd0 msgr2=0x7ffb50100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.629+0000 7ffb55b13640 1 --2- 172.21.15.116:0/259409371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50106fd0 0x7ffb50100610 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009900 tx=0x7ffb4002bcd0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.629+0000 7ffb55b13640 1 -- 172.21.15.116:0/259409371 shutdown_connections 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.629+0000 7ffb55b13640 1 --2- 172.21.15.116:0/259409371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb50100d40 0x7ffb5010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.629+0000 7ffb55b13640 1 --2- 172.21.15.116:0/259409371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50106fd0 0x7ffb50100610 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.629+0000 7ffb55b13640 1 --2- 172.21.15.116:0/259409371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb501066b0 0x7ffb50106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- 172.21.15.116:0/259409371 >> 172.21.15.116:0/259409371 conn(0x7ffb50077710 msgr2=0x7ffb50077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- 172.21.15.116:0/259409371 shutdown_connections 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- 172.21.15.116:0/259409371 wait complete. 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 Processor -- start 2024-03-24T09:27:36.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- start start 2024-03-24T09:27:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 0x7ffb50196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb501066b0 0x7ffb50196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb50106fd0 0x7ffb501973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501a6070 con 0x7ffb50100d40 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501a61e0 con 0x7ffb50106fd0 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.630+0000 7ffb55b13640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ffb501a63a0 con 0x7ffb501066b0 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 0x7ffb50196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 0x7ffb50196930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52700/0 (socket says 172.21.15.116:52700) 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 -- 172.21.15.116:0/1763586540 learned_addr learned my addr 172.21.15.116:0/1763586540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4ffff640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb50106fd0 0x7ffb501973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb46ffd640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb501066b0 0x7ffb50196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 -- 172.21.15.116:0/1763586540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb501066b0 msgr2=0x7ffb50196e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb501066b0 0x7ffb50196e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 -- 172.21.15.116:0/1763586540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb50106fd0 msgr2=0x7ffb501973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb50106fd0 0x7ffb501973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 -- 172.21.15.116:0/1763586540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb40009560 con 0x7ffb50100d40 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4f7fe640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 0x7ffb50196930 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800e9c0 tx=0x7ffb3800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb380096e0 con 0x7ffb50100d40 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb501a6680 con 0x7ffb50100d40 2024-03-24T09:27:36.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ffb380044e0 con 0x7ffb50100d40 2024-03-24T09:27:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38010430 con 0x7ffb50100d40 2024-03-24T09:27:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.631+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb501a6b60 con 0x7ffb50100d40 2024-03-24T09:27:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.632+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb50101d50 con 0x7ffb50100d40 2024-03-24T09:27:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.633+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7ffb38003f80 con 0x7ffb50100d40 2024-03-24T09:27:36.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.634+0000 7ffb4d7fa640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ffb300a2230 0x7ffb300a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.634+0000 7ffb46ffd640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ffb300a2230 0x7ffb300a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.634+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7ffb38014070 con 0x7ffb50100d40 2024-03-24T09:27:36.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.634+0000 7ffb46ffd640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ffb300a2230 0x7ffb300a46f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb40000c00 tx=0x7ffb40005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.638+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7ffb380970b0 con 0x7ffb50100d40 2024-03-24T09:27:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.852+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb50108ba0 con 0x7ffb50100d40 2024-03-24T09:27:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.853+0000 7ffb4d7fa640 1 -- 172.21.15.116:0/1763586540 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7ffb380aee60 con 0x7ffb50100d40 2024-03-24T09:27:36.854 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:27:36.854 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:27:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:27:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ffb300a2230 msgr2=0x7ffb300a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ffb300a2230 0x7ffb300a46f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ffb40000c00 tx=0x7ffb40005fd0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 msgr2=0x7ffb50196930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 0x7ffb50196930 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800e9c0 tx=0x7ffb3800ee80 comp rx=0 tx=0).stop 2024-03-24T09:27:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 shutdown_connections 2024-03-24T09:27:36.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb50106fd0 0x7ffb501973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ffb300a2230 0x7ffb300a46f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ffb501066b0 0x7ffb50196e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 --2- 172.21.15.116:0/1763586540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb50100d40 0x7ffb50196930 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 >> 172.21.15.116:0/1763586540 conn(0x7ffb50077710 msgr2=0x7ffb500ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 shutdown_connections 2024-03-24T09:27:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.855+0000 7ffb55b13640 1 -- 172.21.15.116:0/1763586540 wait complete. 2024-03-24T09:27:36.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:27:36.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 -- 172.21.15.116:0/1582862897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 msgr2=0x7f9e8010e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1582862897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 0x7f9e8010e590 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009ca0 tx=0x7f9e7802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 -- 172.21.15.116:0/1582862897 shutdown_connections 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1582862897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 0x7f9e8010e590 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1582862897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e80100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1582862897 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80107910 0x7f9e80107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 -- 172.21.15.116:0/1582862897 >> 172.21.15.116:0/1582862897 conn(0x7f9e80075950 msgr2=0x7f9e80075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 -- 172.21.15.116:0/1582862897 shutdown_connections 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.970+0000 7f9e871e9640 1 -- 172.21.15.116:0/1582862897 wait complete. 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 Processor -- start 2024-03-24T09:27:36.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 -- start start 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e8019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 0x7f9e8019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80107910 0x7f9e80197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80198100 con 0x7f9e800fffd0 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80198270 con 0x7f9e80100990 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e871e9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9e80198430 con 0x7f9e80107910 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e8019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e8019d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52722/0 (socket says 172.21.15.116:52722) 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 -- 172.21.15.116:0/1502753620 learned_addr learned my addr 172.21.15.116:0/1502753620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 0x7f9e8019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e85786640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80107910 0x7f9e80197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 -- 172.21.15.116:0/1502753620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80107910 msgr2=0x7f9e80197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80107910 0x7f9e80197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 -- 172.21.15.116:0/1502753620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 msgr2=0x7f9e8019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 0x7f9e8019de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 -- 172.21.15.116:0/1502753620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e78009900 con 0x7f9e800fffd0 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.971+0000 7f9e84f85640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e8019d900 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400e9f0 tx=0x7f9e7400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.972+0000 7f9e7dffb640 1 -- 172.21.15.116:0/1502753620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74009720 con 0x7f9e800fffd0 2024-03-24T09:27:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.972+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e80198710 con 0x7f9e800fffd0 2024-03-24T09:27:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.972+0000 7f9e7dffb640 1 -- 172.21.15.116:0/1502753620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e7400cd80 con 0x7f9e800fffd0 2024-03-24T09:27:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.972+0000 7f9e7dffb640 1 -- 172.21.15.116:0/1502753620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e74010430 con 0x7f9e800fffd0 2024-03-24T09:27:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.972+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e801a49a0 con 0x7f9e800fffd0 2024-03-24T09:27:36.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.974+0000 7f9e7dffb640 1 -- 172.21.15.116:0/1502753620 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9e74005020 con 0x7f9e800fffd0 2024-03-24T09:27:36.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 Processor -- start 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 -- start start 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e7dffb640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e700a2230 0x7f9e700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7d3f0 0x55b6fda7d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda863b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda88ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd9eb840 con 0x55b6fda7dd10 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd9e8d70 con 0x55b6fda7d3f0 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd9e4550 con 0x55b6fda868f0 2024-03-24T09:27:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e85786640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e700a2230 0x7f9e700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e7dffb640 1 -- 172.21.15.116:0/1502753620 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9e74014070 con 0x7f9e800fffd0 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda863b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda863b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52736/0 (socket says 172.21.15.116:52736) 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e7ffff640 1 -- 172.21.15.116:0/3743959398 learned_addr learned my addr 172.21.15.116:0/3743959398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e84f85640 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7d3f0 0x55b6fda7d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.975+0000 7f9e85786640 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda88ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e7ffff640 1 -- 172.21.15.116:0/3743959398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 msgr2=0x55b6fda88ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda88ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e7ffff640 1 -- 172.21.15.116:0/3743959398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7d3f0 msgr2=0x55b6fda7d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7d3f0 0x55b6fda7d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e7ffff640 1 -- 172.21.15.116:0/3743959398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6fd979de0 con 0x55b6fda7dd10 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e84f85640 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7d3f0 0x55b6fda7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e85786640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e700a2230 0x7f9e700a46f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9e78004af0 tx=0x7f9e78032fa0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda863b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009c60 tx=0x7f9e6c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28965c20ac9f455 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e6affd640 1 -- 172.21.15.116:0/3743959398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c004230 con 0x55b6fda7dd10 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e6affd640 1 -- 172.21.15.116:0/3743959398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e6c004390 con 0x55b6fda7dd10 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.976+0000 7f9e6affd640 1 -- 172.21.15.116:0/3743959398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c004c30 con 0x55b6fda7dd10 2024-03-24T09:27:36.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 -- 172.21.15.116:0/3743959398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 msgr2=0x55b6fda863b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda863b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009c60 tx=0x7f9e6c011810 comp rx=0 tx=0).stop 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 -- 172.21.15.116:0/3743959398 shutdown_connections 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda88ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda863b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/3743959398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7d3f0 0x55b6fda7d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 -- 172.21.15.116:0/3743959398 >> 172.21.15.116:0/3743959398 conn(0x55b6fda77b20 msgr2=0x55b6fda77f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 -- 172.21.15.116:0/3743959398 shutdown_connections 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.977+0000 7f9e89dc8740 1 -- 172.21.15.116:0/3743959398 wait complete. 2024-03-24T09:27:36.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 -- start start 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 Processor -- start 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda83970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7dd10 0x55b6fda83eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd6b7430 con 0x55b6fda7d3f0 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd673e00 con 0x55b6fda868f0 2024-03-24T09:27:36.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd969590 con 0x55b6fda7dd10 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e89dc8740 10 client.0 _ll_register_callbacks cb 0x55b6fd9d8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda83970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda83970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52744/0 (socket says 172.21.15.116:52744) 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 -- 172.21.15.116:0/4126766474 learned_addr learned my addr 172.21.15.116:0/4126766474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e85786640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda843f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7dd10 0x55b6fda83eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 -- 172.21.15.116:0/4126766474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7dd10 msgr2=0x55b6fda83eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7dd10 0x55b6fda83eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 -- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 msgr2=0x55b6fda843f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda843f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.978+0000 7f9e84f85640 1 -- 172.21.15.116:0/4126766474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6c009890 con 0x55b6fda7d3f0 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e84f85640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda83970 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9e7409ad00 tx=0x7f9e7409a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a002e79fbdfa63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4126766474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7400ba30 con 0x55b6fda7d3f0 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4126766474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9e7400be30 con 0x55b6fda7d3f0 2024-03-24T09:27:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b6fd6e37e0 con 0x55b6fda7d3f0 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4126766474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9e740b92c0 con 0x55b6fda7d3f0 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 10 client.17193 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b6fd9e2bd0 con 0x55b6fda7d3f0 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4126766474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b6fd6e37e0 con 0x55b6fda7d3f0 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4126766474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9e74085040 con 0x55b6fda7d3f0 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 10 client.17193 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 10 client.17193 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:27:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 20 client.17193 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fda96a10 0x55b6fda98e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 4 client.17193 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6fd720720 con 0x55b6fda96a10 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.979+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fda96a10 0x55b6fda98e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.980+0000 7f9e5ffff640 10 client.17193 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.980+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fda96a10 0x55b6fda98e20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b89c27d6c58953b4 server_cookie=bf11a313418eef51 in_seq=0 out_seq=0 2024-03-24T09:27:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.980+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4126766474 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6fd720720 con 0x55b6fda96a10 2024-03-24T09:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.980+0000 7f9e5ffff640 10 client.17193 handle_command_reply: tid=0 2024-03-24T09:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.980+0000 7f9e89dc8740 1 client.17193 shutdown 2024-03-24T09:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 msgr2=0x55b6fda83970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda83970 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f9e7409ad00 tx=0x7f9e7409a800 comp rx=0 tx=0).stop 2024-03-24T09:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 shutdown_connections 2024-03-24T09:27:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7dd10 0x55b6fda83eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fda96a10 0x55b6fda98e20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4126766474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda83970 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e5ffff640 0 client.17193 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 >> 172.21.15.116:0/4126766474 conn(0x55b6fda77b20 msgr2=0x55b6fda79870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 shutdown_connections 2024-03-24T09:27:36.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4126766474 wait complete. 2024-03-24T09:27:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.981+0000 7f9e89dc8740 20 client.17193 trim_cache size 0 max 16384 2024-03-24T09:27:36.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 Processor -- start 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 -- start start 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda860a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7dd10 0x55b6fda841e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda9e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6fd6e37e0 con 0x55b6fda7d3f0 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6fda7f5c0 con 0x55b6fda7dd10 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b6fda78b20 con 0x55b6fda868f0 2024-03-24T09:27:37.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda860a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda860a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52754/0 (socket says 172.21.15.116:52754) 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 -- 172.21.15.116:0/1930099611 learned_addr learned my addr 172.21.15.116:0/1930099611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7dd10 0x55b6fda841e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e85786640 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda9e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 -- 172.21.15.116:0/1930099611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 msgr2=0x55b6fda9e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda9e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 -- 172.21.15.116:0/1930099611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7dd10 msgr2=0x55b6fda841e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7dd10 0x55b6fda841e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 -- 172.21.15.116:0/1930099611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6fd97ec80 con 0x55b6fda7d3f0 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.994+0000 7f9e84f85640 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda860a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9e7409aa70 tx=0x7f9e7400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8764d427bf6609 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e6a7fc640 1 -- 172.21.15.116:0/1930099611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7408b070 con 0x55b6fda7d3f0 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e6a7fc640 1 -- 172.21.15.116:0/1930099611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e740b92c0 con 0x55b6fda7d3f0 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e6a7fc640 1 -- 172.21.15.116:0/1930099611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7409b230 con 0x55b6fda7d3f0 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 -- 172.21.15.116:0/1930099611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 msgr2=0x55b6fda860a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda860a0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f9e7409aa70 tx=0x7f9e7400cf50 comp rx=0 tx=0).stop 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 -- 172.21.15.116:0/1930099611 shutdown_connections 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda868f0 0x55b6fda9e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda7dd10 0x55b6fda841e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/1930099611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7d3f0 0x55b6fda860a0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 -- 172.21.15.116:0/1930099611 >> 172.21.15.116:0/1930099611 conn(0x55b6fda77b20 msgr2=0x55b6fda78e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 -- 172.21.15.116:0/1930099611 shutdown_connections 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 -- 172.21.15.116:0/1930099611 wait complete. 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.995+0000 7f9e89dc8740 1 -- start start 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 Processor -- start 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7d3f0 0x55b6fda80600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda80b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda81080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b6fdafbd20 con 0x55b6fda7dd10 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6fdafbe60 con 0x55b6fda868f0 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b6fdafbff0 con 0x55b6fda7d3f0 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e89dc8740 10 client.0 _ll_register_callbacks cb 0x7f9e7400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda80b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda80b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52758/0 (socket says 172.21.15.116:52758) 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 -- 172.21.15.116:0/4089949972 learned_addr learned my addr 172.21.15.116:0/4089949972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e85786640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda81080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e84f85640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7d3f0 0x55b6fda80600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 -- 172.21.15.116:0/4089949972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7d3f0 msgr2=0x55b6fda80600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7d3f0 0x55b6fda80600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 -- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 msgr2=0x55b6fda81080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda81080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.996+0000 7f9e7ffff640 1 -- 172.21.15.116:0/4089949972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6fd9e2bd0 con 0x55b6fda7dd10 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e7ffff640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda80b40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x55b6fda7cec0 tx=0x7f9e6c009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0784ab1f1011d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e5ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4089949972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0197e0 con 0x55b6fda7dd10 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4089949972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f9e6c02c070 con 0x55b6fda7dd10 2024-03-24T09:27:37.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e6c004390 con 0x55b6fda7dd10 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4089949972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9e6c005a70 con 0x55b6fda7dd10 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 10 client.17199 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b6fd9a4920 con 0x55b6fda7dd10 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4089949972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9e6c004390 con 0x55b6fda7dd10 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4089949972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9e6c026990 con 0x55b6fda7dd10 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 10 client.17199 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 10 client.17199 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 20 client.17199 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fdafd8b0 0x55b6fdaffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 4 client.17199 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6fd76f890 con 0x55b6fdafd8b0 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.997+0000 7f9e84f85640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fdafd8b0 0x55b6fdaffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e5ffff640 10 client.17199 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e84f85640 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fdafd8b0 0x55b6fdaffcd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=190369c0fa1ab019 server_cookie=5b2f078b6c773d4 in_seq=0 out_seq=0 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e5ffff640 1 -- 172.21.15.116:0/4089949972 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6fd76f890 con 0x55b6fdafd8b0 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e5ffff640 10 client.17199 handle_command_reply: tid=0 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 client.17199 shutdown 2024-03-24T09:27:37.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 msgr2=0x55b6fda80b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda80b40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x55b6fda7cec0 tx=0x7f9e6c009820 comp rx=0 tx=0).stop 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 shutdown_connections 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b6fda868f0 0x55b6fda81080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b6fdafd8b0 0x55b6fdaffcd0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b6fda7dd10 0x55b6fda80b40 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 --2- 172.21.15.116:0/4089949972 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b6fda7d3f0 0x55b6fda80600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e5ffff640 0 client.17199 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 >> 172.21.15.116:0/4089949972 conn(0x55b6fda77b20 msgr2=0x55b6fda79570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 shutdown_connections 2024-03-24T09:27:37.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 1 -- 172.21.15.116:0/4089949972 wait complete. 2024-03-24T09:27:37.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.998+0000 7f9e89dc8740 20 client.17199 trim_cache size 0 max 16384 2024-03-24T09:27:37.009 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:37.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e700a2230 msgr2=0x7f9e700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e700a2230 0x7f9e700a46f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9e78004af0 tx=0x7f9e78032fa0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 msgr2=0x7f9e8019d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e8019d900 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f9e7400e9f0 tx=0x7f9e7400edf0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 shutdown_connections 2024-03-24T09:27:37.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9e700a2230 0x7f9e700a46f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9e80107910 0x7f9e80197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e80100990 0x7f9e8019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 --2- 172.21.15.116:0/1502753620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e800fffd0 0x7f9e8019d900 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 >> 172.21.15.116:0/1502753620 conn(0x7f9e80075950 msgr2=0x7f9e800fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 shutdown_connections 2024-03-24T09:27:37.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:36.999+0000 7f9e871e9640 1 -- 172.21.15.116:0/1502753620 wait complete. 2024-03-24T09:27:37.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:27:37.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.114+0000 7fad60b8a640 1 -- 172.21.15.116:0/2275631278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c1005a0 msgr2=0x7fad5c100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.114+0000 7fad60b8a640 1 --2- 172.21.15.116:0/2275631278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c1005a0 0x7fad5c100a20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fad44009890 tx=0x7fad4402bb70 comp rx=0 tx=0).stop 2024-03-24T09:27:37.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 -- 172.21.15.116:0/2275631278 shutdown_connections 2024-03-24T09:27:37.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 --2- 172.21.15.116:0/2275631278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c100f60 0x7fad5c10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 --2- 172.21.15.116:0/2275631278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c1005a0 0x7fad5c100a20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 --2- 172.21.15.116:0/2275631278 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c106660 0x7fad5c106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 -- 172.21.15.116:0/2275631278 >> 172.21.15.116:0/2275631278 conn(0x7fad5c077710 msgr2=0x7fad5c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 -- 172.21.15.116:0/2275631278 shutdown_connections 2024-03-24T09:27:37.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.115+0000 7fad60b8a640 1 -- 172.21.15.116:0/2275631278 wait complete. 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 Processor -- start 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 -- start start 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 0x7fad5c19d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c100f60 0x7fad5c19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c106660 0x7fad5c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1980c0 con 0x7fad5c1005a0 2024-03-24T09:27:37.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c198230 con 0x7fad5c106660 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad60b8a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fad5c1983f0 con 0x7fad5c100f60 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 0x7fad5c19d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 0x7fad5c19d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52774/0 (socket says 172.21.15.116:52774) 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c106660 0x7fad5c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 -- 172.21.15.116:0/1596588091 learned_addr learned my addr 172.21.15.116:0/1596588091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad53fff640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c100f60 0x7fad5c19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 -- 172.21.15.116:0/1596588091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c100f60 msgr2=0x7fad5c19ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c100f60 0x7fad5c19ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 -- 172.21.15.116:0/1596588091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c106660 msgr2=0x7fad5c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c106660 0x7fad5c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 -- 172.21.15.116:0/1596588091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad44009490 con 0x7fad5c1005a0 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c106660 0x7fad5c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad53fff640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c100f60 0x7fad5c19ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 0x7fad5c19d870 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00e9f0 tx=0x7fad4c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.116+0000 7fad537fe640 1 -- 172.21.15.116:0/1596588091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c00ccc0 con 0x7fad5c1005a0 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.117+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5c1986d0 con 0x7fad5c1005a0 2024-03-24T09:27:37.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.117+0000 7fad537fe640 1 -- 172.21.15.116:0/1596588091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad4c00ce20 con 0x7fad5c1005a0 2024-03-24T09:27:37.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.117+0000 7fad537fe640 1 -- 172.21.15.116:0/1596588091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4c010430 con 0x7fad5c1005a0 2024-03-24T09:27:37.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.117+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5c1a4910 con 0x7fad5c1005a0 2024-03-24T09:27:37.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad537fe640 1 -- 172.21.15.116:0/1596588091 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fad4c004f90 con 0x7fad5c1005a0 2024-03-24T09:27:37.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 Processor -- start 2024-03-24T09:27:37.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 -- start start 2024-03-24T09:27:37.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7a3f0 0x5645f8f7a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7ad10 0x5645f8f833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f85ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5645f8ee8840 con 0x5645f8f7a3f0 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5645f8ee5d70 con 0x5645f8f838f0 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.119+0000 7fad6376c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5645f8ee1550 con 0x5645f8f7ad10 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad537fe640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fad480a2230 0x7fad480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f85ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f85ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:47840/0 (socket says 172.21.15.116:47840) 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7a3f0 0x5645f8f7a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 -- 172.21.15.116:0/1266093467 learned_addr learned my addr 172.21.15.116:0/1266093467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad5a59c640 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7ad10 0x5645f8f833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fad480a2230 0x7fad480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad537fe640 1 -- 172.21.15.116:0/1596588091 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fad4c014070 con 0x7fad5c1005a0 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 -- 172.21.15.116:0/1266093467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7ad10 msgr2=0x5645f8f833b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7ad10 0x5645f8f833b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 -- 172.21.15.116:0/1266093467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7a3f0 msgr2=0x5645f8f7a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7a3f0 0x5645f8f7a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 -- 172.21.15.116:0/1266093467 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645f8e76de0 con 0x5645f8f838f0 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad53fff640 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f85ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad4402c0b0 tx=0x7fad44005890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c6f525aef5a907 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fad480a2230 0x7fad480a46f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad540099e0 tx=0x7fad54009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad50ff9640 1 -- 172.21.15.116:0/1266093467 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44034640 con 0x5645f8f838f0 2024-03-24T09:27:37.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad50ff9640 1 -- 172.21.15.116:0/1266093467 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad4402ce90 con 0x5645f8f838f0 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.120+0000 7fad50ff9640 1 -- 172.21.15.116:0/1266093467 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4402ca50 con 0x5645f8f838f0 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 -- 172.21.15.116:0/1266093467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 msgr2=0x5645f8f85ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f85ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fad4402c0b0 tx=0x7fad44005890 comp rx=0 tx=0).stop 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 -- 172.21.15.116:0/1266093467 shutdown_connections 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f85ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7ad10 0x5645f8f833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 --2- 172.21.15.116:0/1266093467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7a3f0 0x5645f8f7a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 -- 172.21.15.116:0/1266093467 >> 172.21.15.116:0/1266093467 conn(0x5645f8f74b20 msgr2=0x5645f8f74f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 -- 172.21.15.116:0/1266093467 shutdown_connections 2024-03-24T09:27:37.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 -- 172.21.15.116:0/1266093467 wait complete. 2024-03-24T09:27:37.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.121+0000 7fad6376c740 1 -- start start 2024-03-24T09:27:37.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 Processor -- start 2024-03-24T09:27:37.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f808b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f80df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f81330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5645f8bb4430 con 0x5645f8f7ad10 2024-03-24T09:27:37.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5645f8b70e00 con 0x5645f8f838f0 2024-03-24T09:27:37.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5645f8e66590 con 0x5645f8f7a3f0 2024-03-24T09:27:37.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad6376c740 10 client.0 _ll_register_callbacks cb 0x5645f8ed5fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f80df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f80df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52800/0 (socket says 172.21.15.116:52800) 2024-03-24T09:27:37.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 -- 172.21.15.116:0/2708221203 learned_addr learned my addr 172.21.15.116:0/2708221203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f81330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad5a59c640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f808b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 -- 172.21.15.116:0/2708221203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 msgr2=0x5645f8f808b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f808b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 -- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 msgr2=0x5645f8f81330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f81330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 -- 172.21.15.116:0/2708221203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad5c10dd20 con 0x5645f8f7ad10 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad53fff640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f80df0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fad44005940 tx=0x7fad4402c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09a4f31e323eabc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.122+0000 7fad41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad41ffb640 1 -- 172.21.15.116:0/2708221203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44004450 con 0x5645f8f7ad10 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad41ffb640 1 -- 172.21.15.116:0/2708221203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fad440035f0 con 0x5645f8f7ad10 2024-03-24T09:27:37.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5645f8be07e0 con 0x5645f8f7ad10 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad41ffb640 1 -- 172.21.15.116:0/2708221203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fad440347b0 con 0x5645f8f7ad10 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 10 client.17211 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5645f8edfbd0 con 0x5645f8f7ad10 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad41ffb640 1 -- 172.21.15.116:0/2708221203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5645f8be07e0 con 0x5645f8f7ad10 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad41ffb640 1 -- 172.21.15.116:0/2708221203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fad44032670 con 0x5645f8f7ad10 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 10 client.17211 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 10 client.17211 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 20 client.17211 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8f935e0 0x5645f8f95a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 4 client.17211 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645f8c1d720 con 0x5645f8f935e0 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.123+0000 7fad5a59c640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8f935e0 0x5645f8f95a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad41ffb640 10 client.17211 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad5a59c640 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8f935e0 0x5645f8f95a00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4ce7cd0658ececdd server_cookie=17d3f7d35b1737bb in_seq=0 out_seq=0 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad41ffb640 1 -- 172.21.15.116:0/2708221203 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5645f8c1d720 con 0x5645f8f935e0 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad41ffb640 10 client.17211 handle_command_reply: tid=0 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 client.17211 shutdown 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 msgr2=0x5645f8f80df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f80df0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fad44005940 tx=0x7fad4402c980 comp rx=0 tx=0).stop 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 shutdown_connections 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8f935e0 0x5645f8f95a00 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f81330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad41ffb640 0 client.17211 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f80df0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 --2- 172.21.15.116:0/2708221203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f808b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 >> 172.21.15.116:0/2708221203 conn(0x5645f8f74b20 msgr2=0x5645f8f76870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.124+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 shutdown_connections 2024-03-24T09:27:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.125+0000 7fad6376c740 1 -- 172.21.15.116:0/2708221203 wait complete. 2024-03-24T09:27:37.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.125+0000 7fad6376c740 20 client.17211 trim_cache size 0 max 16384 2024-03-24T09:27:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 Processor -- start 2024-03-24T09:27:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 -- start start 2024-03-24T09:27:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f82fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f7ad10 0x5645f8f8f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 0x5645f8f73cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5645f8f70250 con 0x5645f8f838f0 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5645f8f75b20 con 0x5645f8f7ad10 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad6376c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5645f8f74950 con 0x5645f8f7a3f0 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 0x5645f8f73cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 0x5645f8f73cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52814/0 (socket says 172.21.15.116:52814) 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2961376154 learned_addr learned my addr 172.21.15.116:0/2961376154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5a59c640 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f82fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad53fff640 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f7ad10 0x5645f8f8f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2961376154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 msgr2=0x5645f8f82fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f82fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2961376154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f7ad10 msgr2=0x5645f8f8f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f7ad10 0x5645f8f8f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 -- 172.21.15.116:0/2961376154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645f8e7bc80 con 0x5645f8f838f0 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad5ad9d640 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 0x5645f8f73cc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fad54011ed0 tx=0x7fad5400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10227b274efd221f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad43fff640 1 -- 172.21.15.116:0/2961376154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54026040 con 0x5645f8f838f0 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad43fff640 1 -- 172.21.15.116:0/2961376154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad540045d0 con 0x5645f8f838f0 2024-03-24T09:27:37.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.138+0000 7fad43fff640 1 -- 172.21.15.116:0/2961376154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5401fbf0 con 0x5645f8f838f0 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 -- 172.21.15.116:0/2961376154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 msgr2=0x5645f8f73cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 0x5645f8f73cc0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fad54011ed0 tx=0x7fad5400e000 comp rx=0 tx=0).stop 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 -- 172.21.15.116:0/2961376154 shutdown_connections 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f838f0 0x5645f8f73cc0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f7ad10 0x5645f8f8f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 --2- 172.21.15.116:0/2961376154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f82fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 -- 172.21.15.116:0/2961376154 >> 172.21.15.116:0/2961376154 conn(0x5645f8f74b20 msgr2=0x5645f8f74f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 -- 172.21.15.116:0/2961376154 shutdown_connections 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 -- 172.21.15.116:0/2961376154 wait complete. 2024-03-24T09:27:37.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 -- start start 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.139+0000 7fad6376c740 1 Processor -- start 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f8b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f88720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f88c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5645f8f8bc90 con 0x5645f8f7ad10 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5645f8f8bdd0 con 0x5645f8f838f0 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5645f8f8bf90 con 0x5645f8f7a3f0 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 10 client.0 _ll_register_callbacks cb 0x7fad54011760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f88720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad5ad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f88c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f88720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52822/0 (socket says 172.21.15.116:52822) 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 -- 172.21.15.116:0/3162486847 learned_addr learned my addr 172.21.15.116:0/3162486847 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad5a59c640 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f8b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 -- 172.21.15.116:0/3162486847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 msgr2=0x5645f8f8b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f8b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 -- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 msgr2=0x5645f8f88c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f88c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 -- 172.21.15.116:0/3162486847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad54019040 con 0x5645f8f7ad10 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad53fff640 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f88720 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fad4402f020 tx=0x7fad44005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cedaf709d867d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad41ffb640 1 -- 172.21.15.116:0/3162486847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad44032800 con 0x5645f8f7ad10 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad41ffb640 1 -- 172.21.15.116:0/3162486847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fad44000e40 con 0x5645f8f7ad10 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad4402ce90 con 0x5645f8f7ad10 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad41ffb640 1 -- 172.21.15.116:0/3162486847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fad44032340 con 0x5645f8f7ad10 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 10 client.17217 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5645f8ea1920 con 0x5645f8f7ad10 2024-03-24T09:27:37.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.140+0000 7fad41ffb640 1 -- 172.21.15.116:0/3162486847 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fad4402ce90 con 0x5645f8f7ad10 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad41ffb640 1 -- 172.21.15.116:0/3162486847 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fad44004370 con 0x5645f8f7ad10 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 10 client.17217 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 10 client.17217 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 20 client.17217 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8ffa760 0x5645f8ffcb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 4 client.17217 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645f8c6c890 con 0x5645f8ffa760 2024-03-24T09:27:37.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad5a59c640 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8ffa760 0x5645f8ffcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad5a59c640 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8ffa760 0x5645f8ffcb80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dd9e92e9350767d0 server_cookie=97bcadc7e979fb00 in_seq=0 out_seq=0 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad41ffb640 10 client.17217 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad41ffb640 1 -- 172.21.15.116:0/3162486847 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645f8c6c890 con 0x5645f8ffa760 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad41ffb640 10 client.17217 handle_command_reply: tid=0 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 client.17217 shutdown 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 msgr2=0x5645f8f88720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f88720 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fad4402f020 tx=0x7fad44005cf0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 shutdown_connections 2024-03-24T09:27:37.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5645f8ffa760 0x5645f8ffcb80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5645f8f838f0 0x5645f8f88c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5645f8f7ad10 0x5645f8f88720 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad41ffb640 0 client.17217 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.141+0000 7fad6376c740 1 --2- 172.21.15.116:0/3162486847 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5645f8f7a3f0 0x5645f8f8b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 >> 172.21.15.116:0/3162486847 conn(0x5645f8f74b20 msgr2=0x5645f8f76700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 shutdown_connections 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad6376c740 1 -- 172.21.15.116:0/3162486847 wait complete. 2024-03-24T09:27:37.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad6376c740 20 client.17217 trim_cache size 0 max 16384 2024-03-24T09:27:37.154 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:37.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fad480a2230 msgr2=0x7fad480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fad480a2230 0x7fad480a46f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad540099e0 tx=0x7fad54009500 comp rx=0 tx=0).stop 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 msgr2=0x7fad5c19d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 0x7fad5c19d870 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fad4c00e9f0 tx=0x7fad4c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 shutdown_connections 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad5c106660 0x7fad5c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fad480a2230 0x7fad480a46f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fad5c100f60 0x7fad5c19ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 --2- 172.21.15.116:0/1596588091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad5c1005a0 0x7fad5c19d870 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 >> 172.21.15.116:0/1596588091 conn(0x7fad5c077710 msgr2=0x7fad5c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 shutdown_connections 2024-03-24T09:27:37.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.142+0000 7fad60b8a640 1 -- 172.21.15.116:0/1596588091 wait complete. 2024-03-24T09:27:37.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:36 smithi116 ceph-mon[32207]: pgmap v168: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:37.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:36 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1763586540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:27:37.157 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:27:37.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 -- 172.21.15.116:0/3663483064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 msgr2=0x7fb068107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 --2- 172.21.15.116:0/3663483064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068107b40 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb058009930 tx=0x7fb05802bc30 comp rx=0 tx=0).stop 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 -- 172.21.15.116:0/3663483064 shutdown_connections 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 --2- 172.21.15.116:0/3663483064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb0680692d0 0x7fb06810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 --2- 172.21.15.116:0/3663483064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb068068910 0x7fb068068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 --2- 172.21.15.116:0/3663483064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068107b40 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.259+0000 7fb06d925640 1 -- 172.21.15.116:0/3663483064 >> 172.21.15.116:0/3663483064 conn(0x7fb068077710 msgr2=0x7fb068077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 -- 172.21.15.116:0/3663483064 shutdown_connections 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 -- 172.21.15.116:0/3663483064 wait complete. 2024-03-24T09:27:37.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 Processor -- start 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 -- start start 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb068068910 0x7fb06819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0680692d0 0x7fb06819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681980b0 con 0x7fb068107760 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb068198220 con 0x7fb0680692d0 2024-03-24T09:27:37.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.260+0000 7fb06d925640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb0681983e0 con 0x7fb068068910 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52850/0 (socket says 172.21.15.116:52850) 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 -- 172.21.15.116:0/1423677650 learned_addr learned my addr 172.21.15.116:0/1423677650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb066ffd640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb068068910 0x7fb06819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0667fc640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0680692d0 0x7fb06819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 -- 172.21.15.116:0/1423677650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb068068910 msgr2=0x7fb06819d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb068068910 0x7fb06819d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 -- 172.21.15.116:0/1423677650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0680692d0 msgr2=0x7fb06819ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0680692d0 0x7fb06819ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 -- 172.21.15.116:0/1423677650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb058009560 con 0x7fb068107760 2024-03-24T09:27:37.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb0677fe640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068197a60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb06000eb10 tx=0x7fb06000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb06c923640 1 -- 172.21.15.116:0/1423677650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060004b20 con 0x7fb068107760 2024-03-24T09:27:37.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb06c923640 1 -- 172.21.15.116:0/1423677650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb060004c80 con 0x7fb068107760 2024-03-24T09:27:37.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0681986c0 con 0x7fb068107760 2024-03-24T09:27:37.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb06c923640 1 -- 172.21.15.116:0/1423677650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb060010690 con 0x7fb068107760 2024-03-24T09:27:37.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.261+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0681a4950 con 0x7fb068107760 2024-03-24T09:27:37.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.263+0000 7fb06c923640 1 -- 172.21.15.116:0/1423677650 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fb060005020 con 0x7fb068107760 2024-03-24T09:27:37.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 Processor -- start 2024-03-24T09:27:37.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 -- start start 2024-03-24T09:27:37.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0bf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df0cace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5607df02d840 con 0x5607df0c88f0 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5607df02ad70 con 0x5607df0bf3f0 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0705e4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5607df026550 con 0x5607df0bfd10 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb06c923640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb0500a2230 0x7fb0500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df0cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df0cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52858/0 (socket says 172.21.15.116:52858) 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 -- 172.21.15.116:0/4147639365 learned_addr learned my addr 172.21.15.116:0/4147639365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb066ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0bf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0677fe640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb0500a2230 0x7fb0500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb06c923640 1 -- 172.21.15.116:0/1423677650 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb060014070 con 0x7fb068107760 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 -- 172.21.15.116:0/4147639365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 msgr2=0x5607df0c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 -- 172.21.15.116:0/4147639365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 msgr2=0x5607df0bf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0bf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 -- 172.21.15.116:0/4147639365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607defbbde0 con 0x5607df0c88f0 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb066ffd640 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.264+0000 7fb0667fc640 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df0cace0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0058c0 tx=0x7fb05c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acffdb916f53ab34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0677fe640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb0500a2230 0x7fb0500a46f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0600cbe10 tx=0x7fb0600665a0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb04dffb640 1 -- 172.21.15.116:0/4147639365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c004030 con 0x5607df0c88f0 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb04dffb640 1 -- 172.21.15.116:0/4147639365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb05c00f790 con 0x5607df0c88f0 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb04dffb640 1 -- 172.21.15.116:0/4147639365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c0039f0 con 0x5607df0c88f0 2024-03-24T09:27:37.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 -- 172.21.15.116:0/4147639365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 msgr2=0x5607df0cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df0cace0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb05c0058c0 tx=0x7fb05c005850 comp rx=0 tx=0).stop 2024-03-24T09:27:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 -- 172.21.15.116:0/4147639365 shutdown_connections 2024-03-24T09:27:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df0cace0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 --2- 172.21.15.116:0/4147639365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0bf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 -- 172.21.15.116:0/4147639365 >> 172.21.15.116:0/4147639365 conn(0x5607df0b9b20 msgr2=0x5607df0b9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 -- 172.21.15.116:0/4147639365 shutdown_connections 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.265+0000 7fb0705e4740 1 -- 172.21.15.116:0/4147639365 wait complete. 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 -- start start 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 Processor -- start 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 0x5607df0c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0c88f0 0x5607df0c6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5607decf9430 con 0x5607df0bfd10 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5607decb5e00 con 0x5607df0bf3f0 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5607defab590 con 0x5607df0c88f0 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0705e4740 10 client.0 _ll_register_callbacks cb 0x5607df01afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb066ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 0x5607df0c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb066ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 0x5607df0c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52866/0 (socket says 172.21.15.116:52866) 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb066ffd640 1 -- 172.21.15.116:0/3018391760 learned_addr learned my addr 172.21.15.116:0/3018391760 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0667fc640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.266+0000 7fb0667fc640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0c88f0 0x5607df0c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb066ffd640 1 -- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0c88f0 msgr2=0x5607df0c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb066ffd640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0c88f0 0x5607df0c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb066ffd640 1 -- 172.21.15.116:0/3018391760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 msgr2=0x5607df0c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb066ffd640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb066ffd640 1 -- 172.21.15.116:0/3018391760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c019040 con 0x5607df0bfd10 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0667fc640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0c88f0 0x5607df0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb066ffd640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 0x5607df0c5ee0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb058036040 tx=0x7fb05802c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8da97028a7fc66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb046ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb046ffd640 1 -- 172.21.15.116:0/3018391760 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb058004250 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb046ffd640 1 -- 172.21.15.116:0/3018391760 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb0580043b0 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5607ded257e0 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb046ffd640 1 -- 172.21.15.116:0/3018391760 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb0580035f0 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 10 client.17229 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5607df024bd0 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb046ffd640 1 -- 172.21.15.116:0/3018391760 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5607ded257e0 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb046ffd640 1 -- 172.21.15.116:0/3018391760 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb05803c040 con 0x5607df0bfd10 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 10 client.17229 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 10 client.17229 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 20 client.17229 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df0d8a80 0x5607df0daea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 4 client.17229 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.267+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607ded62720 con 0x5607df0d8a80 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.268+0000 7fb0667fc640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df0d8a80 0x5607df0daea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.268+0000 7fb046ffd640 10 client.17229 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.268+0000 7fb0667fc640 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df0d8a80 0x5607df0daea0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae431780346e0af0 server_cookie=694dbb87376ffe48 in_seq=0 out_seq=0 2024-03-24T09:27:37.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb046ffd640 1 -- 172.21.15.116:0/3018391760 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5607ded62720 con 0x5607df0d8a80 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb046ffd640 10 client.17229 handle_command_reply: tid=0 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 client.17229 shutdown 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 msgr2=0x5607df0c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 0x5607df0c5ee0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb058036040 tx=0x7fb05802c9d0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 shutdown_connections 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0c88f0 0x5607df0c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bfd10 0x5607df0c5ee0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df0d8a80 0x5607df0daea0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 --2- 172.21.15.116:0/3018391760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb046ffd640 0 client.17229 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:37.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 >> 172.21.15.116:0/3018391760 conn(0x5607df0b9b20 msgr2=0x5607df0bb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 shutdown_connections 2024-03-24T09:27:37.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 1 -- 172.21.15.116:0/3018391760 wait complete. 2024-03-24T09:27:37.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.269+0000 7fb0705e4740 20 client.17229 trim_cache size 0 max 16384 2024-03-24T09:27:37.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:36 smithi143 ceph-mon[35965]: pgmap v168: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:37.285 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:36 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1763586540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 Processor -- start 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 -- start start 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 0x5607df0d4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0c88f0 0x5607df0e07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5607df0b5480 con 0x5607df0bf3f0 2024-03-24T09:27:37.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5607df0c6e60 con 0x5607df0c88f0 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0705e4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5607df0d5500 con 0x5607df0bfd10 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 0x5607df0d4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 0x5607df0d4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52882/0 (socket says 172.21.15.116:52882) 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 -- 172.21.15.116:0/2702328019 learned_addr learned my addr 172.21.15.116:0/2702328019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb066ffd640 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0c88f0 0x5607df0e07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 -- 172.21.15.116:0/2702328019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 msgr2=0x5607df0b8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0b8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 -- 172.21.15.116:0/2702328019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0c88f0 msgr2=0x5607df0e07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0c88f0 0x5607df0e07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.282+0000 7fb0667fc640 1 -- 172.21.15.116:0/2702328019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607defc0c80 con 0x5607df0bf3f0 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0667fc640 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 0x5607df0d4180 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00fee0 tx=0x7fb05c003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c5de6534019c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb04d7fa640 1 -- 172.21.15.116:0/2702328019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c02fbc0 con 0x5607df0bf3f0 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb04d7fa640 1 -- 172.21.15.116:0/2702328019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb05c011040 con 0x5607df0bf3f0 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb04d7fa640 1 -- 172.21.15.116:0/2702328019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb05c005570 con 0x5607df0bf3f0 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 -- 172.21.15.116:0/2702328019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 msgr2=0x5607df0d4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 0x5607df0d4180 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00fee0 tx=0x7fb05c003690 comp rx=0 tx=0).stop 2024-03-24T09:27:37.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 -- 172.21.15.116:0/2702328019 shutdown_connections 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0c88f0 0x5607df0e07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0b8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 --2- 172.21.15.116:0/2702328019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0bf3f0 0x5607df0d4180 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 -- 172.21.15.116:0/2702328019 >> 172.21.15.116:0/2702328019 conn(0x5607df0b9b20 msgr2=0x5607df0b8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 -- 172.21.15.116:0/2702328019 shutdown_connections 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.283+0000 7fb0705e4740 1 -- 172.21.15.116:0/2702328019 wait complete. 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 -- start start 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 Processor -- start 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df13dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5607df0c38b0 con 0x5607df0c88f0 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5607df13e320 con 0x5607df0bf3f0 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5607df13e490 con 0x5607df0bfd10 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0705e4740 10 client.0 _ll_register_callbacks cb 0x7fb058009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0667fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df13dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb0667fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56190/0 (socket says 172.21.15.116:56190) 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 -- 172.21.15.116:0/346823623 learned_addr learned my addr 172.21.15.116:0/346823623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 -- 172.21.15.116:0/346823623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 msgr2=0x5607df0c2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 -- 172.21.15.116:0/346823623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 msgr2=0x5607df13dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df13dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.284+0000 7fb066ffd640 1 -- 172.21.15.116:0/346823623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c029040 con 0x5607df0bfd10 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb066ffd640 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c31b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb05802c170 tx=0x7fb05802cab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d4a0b9fdb5ca922 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb046ffd640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:27:37.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb046ffd640 1 -- 172.21.15.116:0/346823623 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0580037f0 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb05c00f790 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb046ffd640 1 -- 172.21.15.116:0/346823623 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb058004a60 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb046ffd640 1 -- 172.21.15.116:0/346823623 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb058005660 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 10 client.26926 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5607defe6920 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb046ffd640 1 -- 172.21.15.116:0/346823623 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb05c00f790 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb046ffd640 1 -- 172.21.15.116:0/346823623 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb05803c040 con 0x5607df0bfd10 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 10 client.26926 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 10 client.26926 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 20 client.26926 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df13fab0 0x5607df141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 4 client.26926 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.285+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5607dedb1890 con 0x5607df13fab0 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0667fc640 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df13fab0 0x5607df141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb046ffd640 10 client.26926 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0667fc640 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df13fab0 0x5607df141ed0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=185c199eabbc1f9a server_cookie=ebc04183071cbcd3 in_seq=0 out_seq=0 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb046ffd640 1 -- 172.21.15.116:0/346823623 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5607dedb1890 con 0x5607df13fab0 2024-03-24T09:27:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb046ffd640 10 client.26926 handle_command_reply: tid=0 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 client.26926 shutdown 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 msgr2=0x5607df0c31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c31b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb05802c170 tx=0x7fb05802cab0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 shutdown_connections 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607df0c88f0 0x5607df13dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5607df0bfd10 0x5607df0c31b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5607df13fab0 0x5607df141ed0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 --2- 172.21.15.116:0/346823623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5607df0bf3f0 0x5607df0c2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb046ffd640 0 client.26926 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:27:37.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 >> 172.21.15.116:0/346823623 conn(0x5607df0b9b20 msgr2=0x5607df0b8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 shutdown_connections 2024-03-24T09:27:37.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 1 -- 172.21.15.116:0/346823623 wait complete. 2024-03-24T09:27:37.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.286+0000 7fb0705e4740 20 client.26926 trim_cache size 0 max 16384 2024-03-24T09:27:37.299 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:37.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb0500a2230 msgr2=0x7fb0500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb0500a2230 0x7fb0500a46f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb0600cbe10 tx=0x7fb0600665a0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 msgr2=0x7fb068197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068197a60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb06000eb10 tx=0x7fb06000c2d0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 shutdown_connections 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb068107760 0x7fb068197a60 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0680692d0 0x7fb06819ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb0500a2230 0x7fb0500a46f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 --2- 172.21.15.116:0/1423677650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb068068910 0x7fb06819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 >> 172.21.15.116:0/1423677650 conn(0x7fb068077710 msgr2=0x7fb0680fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 shutdown_connections 2024-03-24T09:27:37.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.287+0000 7fb06d925640 1 -- 172.21.15.116:0/1423677650 wait complete. 2024-03-24T09:27:37.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:27:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:36 smithi117 ceph-mon[35788]: pgmap v168: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:36 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1763586540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:27:37.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 -- 172.21.15.116:0/3587566464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 msgr2=0x7f137010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 --2- 172.21.15.116:0/3587566464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137010d440 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f1368009ca0 tx=0x7f136802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 -- 172.21.15.116:0/3587566464 shutdown_connections 2024-03-24T09:27:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 --2- 172.21.15.116:0/3587566464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137010d440 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 --2- 172.21.15.116:0/3587566464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13701005c0 0x7f1370100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 --2- 172.21.15.116:0/3587566464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1370106680 0x7f1370106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 -- 172.21.15.116:0/3587566464 >> 172.21.15.116:0/3587566464 conn(0x7f13700776e0 msgr2=0x7f1370077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 -- 172.21.15.116:0/3587566464 shutdown_connections 2024-03-24T09:27:37.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.407+0000 7f1374e55640 1 -- 172.21.15.116:0/3587566464 wait complete. 2024-03-24T09:27:37.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 Processor -- start 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 -- start start 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13701005c0 0x7f137019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1370106680 0x7f13701979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1370198030 con 0x7f1370100f80 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13701981a0 con 0x7f13701005c0 2024-03-24T09:27:37.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f1374e55640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1370198360 con 0x7f1370106680 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52904/0 (socket says 172.21.15.116:52904) 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 -- 172.21.15.116:0/3945208620 learned_addr learned my addr 172.21.15.116:0/3945208620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136e59c640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13701005c0 0x7f137019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136ed9d640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1370106680 0x7f13701979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 -- 172.21.15.116:0/3945208620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1370106680 msgr2=0x7f13701979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1370106680 0x7f13701979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 -- 172.21.15.116:0/3945208620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13701005c0 msgr2=0x7f137019d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13701005c0 0x7f137019d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 -- 172.21.15.116:0/3945208620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1364009490 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f136dd9b640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137019dd20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f136000e9f0 tx=0x7f136000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f135f7fe640 1 -- 172.21.15.116:0/3945208620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360009720 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.408+0000 7f135f7fe640 1 -- 172.21.15.116:0/3945208620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f136000cd80 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.409+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1368009900 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.409+0000 7f135f7fe640 1 -- 172.21.15.116:0/3945208620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360010430 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.409+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13701a46a0 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f135f7fe640 1 -- 172.21.15.116:0/3945208620 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f1360005020 con 0x7f1370100f80 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f1377a36740 1 Processor -- start 2024-03-24T09:27:37.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f135f7fe640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f13580a2230 0x7f13580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f136ed9d640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f13580a2230 0x7f13580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f135f7fe640 1 -- 172.21.15.116:0/3945208620 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1360014070 con 0x7f1370100f80 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f1377a36740 1 -- start start 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f1377a36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b523f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.411+0000 7f1377a36740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b52483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f1377a36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b52488f0 0x5633b524ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f1377a36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633b51ad840 con 0x5633b523f3f0 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f1377a36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633b51aad70 con 0x5633b52488f0 2024-03-24T09:27:37.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f1377a36740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633b51a6550 con 0x5633b523fd10 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b523f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b523f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52920/0 (socket says 172.21.15.116:52920) 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 -- 172.21.15.116:0/2404736287 learned_addr learned my addr 172.21.15.116:0/2404736287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136ed9d640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f13580a2230 0x7f13580a46f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1368004af0 tx=0x7f1368005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136dd9b640 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b52483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136ed9d640 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b52488f0 0x5633b524ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 -- 172.21.15.116:0/2404736287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 msgr2=0x5633b52483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b52483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 -- 172.21.15.116:0/2404736287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b52488f0 msgr2=0x5633b524ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b52488f0 0x5633b524ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 -- 172.21.15.116:0/2404736287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b513bde0 con 0x5633b523f3f0 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f136e59c640 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b523f7d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1364009e20 tx=0x7f1364011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6660a91f0e7d7138 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f135cff9640 1 -- 172.21.15.116:0/2404736287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640042f0 con 0x5633b523f3f0 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f135cff9640 1 -- 172.21.15.116:0/2404736287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1364004450 con 0x5633b523f3f0 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.412+0000 7f135cff9640 1 -- 172.21.15.116:0/2404736287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364019850 con 0x5633b523f3f0 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 -- 172.21.15.116:0/2404736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 msgr2=0x5633b523f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b523f7d0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1364009e20 tx=0x7f1364011810 comp rx=0 tx=0).stop 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 -- 172.21.15.116:0/2404736287 shutdown_connections 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b52488f0 0x5633b524ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b52483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 --2- 172.21.15.116:0/2404736287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b523f7d0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 -- 172.21.15.116:0/2404736287 >> 172.21.15.116:0/2404736287 conn(0x5633b5239b20 msgr2=0x5633b5239f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 -- 172.21.15.116:0/2404736287 shutdown_connections 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 -- 172.21.15.116:0/2404736287 wait complete. 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 -- start start 2024-03-24T09:27:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.413+0000 7f1377a36740 1 Processor -- start 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523f3f0 0x5633b5245950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b5245e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633b4e79430 con 0x5633b52488f0 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633b4e35e00 con 0x5633b523f3f0 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633b512b590 con 0x5633b523fd10 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 10 client.0 _ll_register_callbacks cb 0x5633b519afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523f3f0 0x5633b5245950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52463d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52932/0 (socket says 172.21.15.116:52932) 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 -- 172.21.15.116:0/1333725543 learned_addr learned my addr 172.21.15.116:0/1333725543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136dd9b640 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b5245e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 -- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 msgr2=0x5633b5245e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b5245e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 -- 172.21.15.116:0/1333725543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523f3f0 msgr2=0x5633b5245950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523f3f0 0x5633b5245950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 -- 172.21.15.116:0/1333725543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1364009a80 con 0x5633b52488f0 2024-03-24T09:27:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f136ed9d640 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52463d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1368009ca0 tx=0x7f13680095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11d26327cfb948d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1355ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1355ffb640 1 -- 172.21.15.116:0/1333725543 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1368003b70 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1355ffb640 1 -- 172.21.15.116:0/1333725543 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1368047070 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633b4ea57e0 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1355ffb640 1 -- 172.21.15.116:0/1333725543 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1368003cd0 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 10 client.17247 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.414+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633b51a4bd0 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1355ffb640 1 -- 172.21.15.116:0/1333725543 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5633b4ea57e0 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1355ffb640 1 -- 172.21.15.116:0/1333725543 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f136803f4b0 con 0x5633b52488f0 2024-03-24T09:27:37.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1377a36740 10 client.17247 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1377a36740 10 client.17247 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1377a36740 20 client.17247 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1377a36740 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b5258880 0x5633b525aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1377a36740 4 client.17247 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633b4ee2720 con 0x5633b5258880 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f136e59c640 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b5258880 0x5633b525aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f136e59c640 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b5258880 0x5633b525aca0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=42be49adaab65fb6 server_cookie=d23f9cf4d3b825f9 in_seq=0 out_seq=0 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.415+0000 7f1355ffb640 10 client.17247 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:37.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1355ffb640 1 -- 172.21.15.116:0/1333725543 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633b4ee2720 con 0x5633b5258880 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1355ffb640 10 client.17247 handle_command_reply: tid=0 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 client.17247 shutdown 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 msgr2=0x5633b52463d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52463d0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1368009ca0 tx=0x7f13680095c0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 shutdown_connections 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b5258880 0x5633b525aca0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52463d0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1355ffb640 0 client.17247 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:37.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523fd10 0x5633b5245e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 --2- 172.21.15.116:0/1333725543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523f3f0 0x5633b5245950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 >> 172.21.15.116:0/1333725543 conn(0x5633b5239b20 msgr2=0x5633b523b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 shutdown_connections 2024-03-24T09:27:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 1 -- 172.21.15.116:0/1333725543 wait complete. 2024-03-24T09:27:37.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.416+0000 7f1377a36740 20 client.17247 trim_cache size 0 max 16384 2024-03-24T09:27:37.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 Processor -- start 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 -- start start 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b5248080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b52461f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b52488f0 0x5633b5260590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633b52554d0 con 0x5633b523f3f0 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633b523a920 con 0x5633b523fd10 2024-03-24T09:27:37.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.429+0000 7f1377a36740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633b522ae70 con 0x5633b52488f0 2024-03-24T09:27:37.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b5248080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b5248080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52936/0 (socket says 172.21.15.116:52936) 2024-03-24T09:27:37.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 -- 172.21.15.116:0/2848856324 learned_addr learned my addr 172.21.15.116:0/2848856324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136dd9b640 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b52461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136ed9d640 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b52488f0 0x5633b5260590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 -- 172.21.15.116:0/2848856324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b52488f0 msgr2=0x5633b5260590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b52488f0 0x5633b5260590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 -- 172.21.15.116:0/2848856324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 msgr2=0x5633b52461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b52461f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 -- 172.21.15.116:0/2848856324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633b5140c80 con 0x5633b523f3f0 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f136e59c640 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b5248080 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1364004250 tx=0x7f1364005fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2337393370cfff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f1357fff640 1 -- 172.21.15.116:0/2848856324 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364005050 con 0x5633b523f3f0 2024-03-24T09:27:37.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f1357fff640 1 -- 172.21.15.116:0/2848856324 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1364005660 con 0x5633b523f3f0 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.430+0000 7f1357fff640 1 -- 172.21.15.116:0/2848856324 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640045c0 con 0x5633b523f3f0 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- 172.21.15.116:0/2848856324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 msgr2=0x5633b5248080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b5248080 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1364004250 tx=0x7f1364005fb0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- 172.21.15.116:0/2848856324 shutdown_connections 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b52488f0 0x5633b5260590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b52461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- 172.21.15.116:0/2848856324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b523f3f0 0x5633b5248080 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- 172.21.15.116:0/2848856324 >> 172.21.15.116:0/2848856324 conn(0x5633b5239b20 msgr2=0x5633b523ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- 172.21.15.116:0/2848856324 shutdown_connections 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- 172.21.15.116:0/2848856324 wait complete. 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- start start 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 Processor -- start 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523f3f0 0x5633b524c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b524c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52bdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633b524d2c0 con 0x5633b52488f0 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633b52be100 con 0x5633b523fd10 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633b52be240 con 0x5633b523f3f0 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f1377a36740 10 client.0 _ll_register_callbacks cb 0x7f1360005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f136ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52bdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f136dd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b524c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f136ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52bdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52940/0 (socket says 172.21.15.116:52940) 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f136e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523f3f0 0x5633b524c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.431+0000 7f136ed9d640 1 -- 172.21.15.116:0/2730996679 learned_addr learned my addr 172.21.15.116:0/2730996679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136ed9d640 1 -- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523f3f0 msgr2=0x5633b524c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136ed9d640 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523f3f0 0x5633b524c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136ed9d640 1 -- 172.21.15.116:0/2730996679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 msgr2=0x5633b524c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136ed9d640 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b524c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136ed9d640 1 -- 172.21.15.116:0/2730996679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136401e040 con 0x5633b52488f0 2024-03-24T09:27:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136dd9b640 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b524c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136ed9d640 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52bdbc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1368009ca0 tx=0x7f136803fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f269628e7fe9337b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1355ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1355ffb640 1 -- 172.21.15.116:0/2730996679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1368003b70 con 0x5633b52488f0 2024-03-24T09:27:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1364004450 con 0x5633b52488f0 2024-03-24T09:27:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1355ffb640 1 -- 172.21.15.116:0/2730996679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f1368047040 con 0x5633b52488f0 2024-03-24T09:27:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1355ffb640 1 -- 172.21.15.116:0/2730996679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f13680329f0 con 0x5633b52488f0 2024-03-24T09:27:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 10 client.17253 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633b5166920 con 0x5633b52488f0 2024-03-24T09:27:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1355ffb640 1 -- 172.21.15.116:0/2730996679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1364004450 con 0x5633b52488f0 2024-03-24T09:27:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1355ffb640 1 -- 172.21.15.116:0/2730996679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1368062440 con 0x5633b52488f0 2024-03-24T09:27:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 10 client.17253 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 10 client.17253 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:27:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 20 client.17253 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b52bf6c0 0x5633b52c1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 4 client.17253 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633b4f31890 con 0x5633b52bf6c0 2024-03-24T09:27:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.432+0000 7f136e59c640 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b52bf6c0 0x5633b52c1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f136e59c640 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b52bf6c0 0x5633b52c1b00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5ee4f158960fa0 server_cookie=af77082ca199a1f in_seq=0 out_seq=0 2024-03-24T09:27:37.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1355ffb640 10 client.17253 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:37.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1355ffb640 1 -- 172.21.15.116:0/2730996679 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633b4f31890 con 0x5633b52bf6c0 2024-03-24T09:27:37.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1355ffb640 10 client.17253 handle_command_reply: tid=0 2024-03-24T09:27:37.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 client.17253 shutdown 2024-03-24T09:27:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 msgr2=0x5633b52bdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52bdbc0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1368009ca0 tx=0x7f136803fce0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 shutdown_connections 2024-03-24T09:27:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633b52bf6c0 0x5633b52c1b00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633b52488f0 0x5633b52bdbc0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1355ffb640 0 client.17253 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:27:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633b523fd10 0x5633b524c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 --2- 172.21.15.116:0/2730996679 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633b523f3f0 0x5633b524c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 >> 172.21.15.116:0/2730996679 conn(0x5633b5239b20 msgr2=0x5633b523b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 shutdown_connections 2024-03-24T09:27:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 1 -- 172.21.15.116:0/2730996679 wait complete. 2024-03-24T09:27:37.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.433+0000 7f1377a36740 20 client.17253 trim_cache size 0 max 16384 2024-03-24T09:27:37.451 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f13580a2230 msgr2=0x7f13580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f13580a2230 0x7f13580a46f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f1368004af0 tx=0x7f1368005c50 comp rx=0 tx=0).stop 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 msgr2=0x7f137019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137019dd20 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f136000e9f0 tx=0x7f136000edf0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 shutdown_connections 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f13580a2230 0x7f13580a46f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1370106680 0x7f13701979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1370100f80 0x7f137019dd20 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 --2- 172.21.15.116:0/3945208620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13701005c0 0x7f137019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 >> 172.21.15.116:0/3945208620 conn(0x7f13700776e0 msgr2=0x7f13700ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 shutdown_connections 2024-03-24T09:27:37.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.434+0000 7f1374e55640 1 -- 172.21.15.116:0/3945208620 wait complete. 2024-03-24T09:27:37.454 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 -- 172.21.15.116:0/2152376182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c10bd50 msgr2=0x7fbb0c10e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 --2- 172.21.15.116:0/2152376182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c10bd50 0x7fbb0c10e1c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fbb04009ca0 tx=0x7fbb0402c180 comp rx=0 tx=0).stop 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 -- 172.21.15.116:0/2152376182 shutdown_connections 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 --2- 172.21.15.116:0/2152376182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c10bd50 0x7fbb0c10e1c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 --2- 172.21.15.116:0/2152376182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c1032a0 0x7fbb0c10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 --2- 172.21.15.116:0/2152376182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb0c102980 0x7fbb0c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 -- 172.21.15.116:0/2152376182 >> 172.21.15.116:0/2152376182 conn(0x7fbb0c077770 msgr2=0x7fbb0c077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 -- 172.21.15.116:0/2152376182 shutdown_connections 2024-03-24T09:27:37.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.558+0000 7fbb12695640 1 -- 172.21.15.116:0/2152376182 wait complete. 2024-03-24T09:27:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 Processor -- start 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 -- start start 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 0x7fbb0c19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c1032a0 0x7fbb0c19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb0c10bd50 0x7fbb0c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c198140 con 0x7fbb0c102980 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c1982b0 con 0x7fbb0c10bd50 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb12695640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c198470 con 0x7fbb0c1032a0 2024-03-24T09:27:37.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 0x7fbb0c19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 0x7fbb0c19d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52956/0 (socket says 172.21.15.116:52956) 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb0bfff640 1 -- 172.21.15.116:0/324153316 learned_addr learned my addr 172.21.15.116:0/324153316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb0b7fe640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c1032a0 0x7fbb0c19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.559+0000 7fbb10c32640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb0c10bd50 0x7fbb0c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb0bfff640 1 -- 172.21.15.116:0/324153316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c1032a0 msgr2=0x7fbb0c19de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c1032a0 0x7fbb0c19de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb0bfff640 1 -- 172.21.15.116:0/324153316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb0c10bd50 msgr2=0x7fbb0c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb0c10bd50 0x7fbb0c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb0bfff640 1 -- 172.21.15.116:0/324153316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbafc009490 con 0x7fbb0c102980 2024-03-24T09:27:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 0x7fbb0c19d8f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00ea60 tx=0x7fbafc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb097fa640 1 -- 172.21.15.116:0/324153316 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc00cde0 con 0x7fbb0c102980 2024-03-24T09:27:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb097fa640 1 -- 172.21.15.116:0/324153316 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbafc0044d0 con 0x7fbb0c102980 2024-03-24T09:27:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb04009900 con 0x7fbb0c102980 2024-03-24T09:27:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb097fa640 1 -- 172.21.15.116:0/324153316 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc010690 con 0x7fbb0c102980 2024-03-24T09:27:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.560+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c1a47b0 con 0x7fbb0c102980 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.562+0000 7fbb097fa640 1 -- 172.21.15.116:0/324153316 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fbafc003e40 con 0x7fbb0c102980 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.562+0000 7fbb15271740 1 Processor -- start 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 -- start start 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb097fa640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbaf80a2230 0x7fbaf80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a826967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a82604840 con 0x564a8269f8f0 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a82601d70 con 0x564a826963f0 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb15271740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a825fd550 con 0x564a82696d10 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb10c32640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbaf80a2230 0x7fbaf80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb097fa640 1 -- 172.21.15.116:0/324153316 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbafc014070 con 0x7fbb0c102980 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a826967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a826967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:47968/0 (socket says 172.21.15.116:47968) 2024-03-24T09:27:37.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 -- 172.21.15.116:0/3658472227 learned_addr learned my addr 172.21.15.116:0/3658472227 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0b7fe640 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb10c32640 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 -- 172.21.15.116:0/3658472227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 msgr2=0x564a8269f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 -- 172.21.15.116:0/3658472227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 msgr2=0x564a826a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 -- 172.21.15.116:0/3658472227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a82592de0 con 0x564a826963f0 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb10c32640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbaf80a2230 0x7fbaf80a46f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb04004af0 tx=0x7fbb04034e30 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a826967d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0a3180 tx=0x7fbafc09ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f1bc786d54e913 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.563+0000 7fbb027fc640 1 -- 172.21.15.116:0/3658472227 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc09af10 con 0x564a826963f0 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb027fc640 1 -- 172.21.15.116:0/3658472227 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbafc09f060 con 0x564a826963f0 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb027fc640 1 -- 172.21.15.116:0/3658472227 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0cbef0 con 0x564a826963f0 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 -- 172.21.15.116:0/3658472227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 msgr2=0x564a826967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a826967d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0a3180 tx=0x7fbafc09ab10 comp rx=0 tx=0).stop 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 -- 172.21.15.116:0/3658472227 shutdown_connections 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 --2- 172.21.15.116:0/3658472227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a826967d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 -- 172.21.15.116:0/3658472227 >> 172.21.15.116:0/3658472227 conn(0x564a82690b20 msgr2=0x564a82690f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 -- 172.21.15.116:0/3658472227 shutdown_connections 2024-03-24T09:27:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 -- 172.21.15.116:0/3658472227 wait complete. 2024-03-24T09:27:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.564+0000 7fbb15271740 1 -- start start 2024-03-24T09:27:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 Processor -- start 2024-03-24T09:27:37.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a8269c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a8269d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a822d0430 con 0x564a8269f8f0 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a8228ce00 con 0x564a826963f0 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a82582590 con 0x564a82696d10 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb15271740 10 client.0 _ll_register_callbacks cb 0x564a825f1fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a8269d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a8269c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a8269d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52978/0 (socket says 172.21.15.116:52978) 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 -- 172.21.15.116:0/2674564683 learned_addr learned my addr 172.21.15.116:0/2674564683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a8269c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:47972/0 (socket says 172.21.15.116:47972) 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb0b7fe640 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 -- 172.21.15.116:0/2674564683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 msgr2=0x564a8269ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 -- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 msgr2=0x564a8269c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a8269c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.565+0000 7fbb10c32640 1 -- 172.21.15.116:0/2674564683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c1a47b0 con 0x564a8269f8f0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb10c32640 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a8269d360 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbb04036000 tx=0x7fbb0402c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922d7708050fe1cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbaeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/2674564683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04032880 con 0x564a8269f8f0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/2674564683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb04034890 con 0x564a8269f8f0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564a822fc7e0 con 0x564a8269f8f0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/2674564683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbb040329e0 con 0x564a8269f8f0 2024-03-24T09:27:37.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 10 client.17265 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564a825fbbd0 con 0x564a8269f8f0 2024-03-24T09:27:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/2674564683 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x564a822fc7e0 con 0x564a8269f8f0 2024-03-24T09:27:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/2674564683 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fbb040045d0 con 0x564a8269f8f0 2024-03-24T09:27:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 10 client.17265 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 10 client.17265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:37.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 20 client.17265 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a826afa50 0x564a826b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 4 client.17265 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a82339720 con 0x564a826afa50 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.566+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a826afa50 0x564a826b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.567+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a826afa50 0x564a826b1e90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d776bca90ae0715a server_cookie=6c45dad88b2fc363 in_seq=0 out_seq=0 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.567+0000 7fbaeb7fe640 10 client.17265 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.567+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/2674564683 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a82339720 con 0x564a826afa50 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.567+0000 7fbaeb7fe640 10 client.17265 handle_command_reply: tid=0 2024-03-24T09:27:37.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.567+0000 7fbb15271740 1 client.17265 shutdown 2024-03-24T09:27:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 msgr2=0x564a8269d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a8269d360 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fbb04036000 tx=0x7fbb0402c180 comp rx=0 tx=0).stop 2024-03-24T09:27:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 shutdown_connections 2024-03-24T09:27:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a8269d360 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a82696d10 0x564a8269ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a826afa50 0x564a826b1e90 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 --2- 172.21.15.116:0/2674564683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a826963f0 0x564a8269c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbaeb7fe640 0 client.17265 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 >> 172.21.15.116:0/2674564683 conn(0x564a82690b20 msgr2=0x564a82692870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 shutdown_connections 2024-03-24T09:27:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 1 -- 172.21.15.116:0/2674564683 wait complete. 2024-03-24T09:27:37.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.568+0000 7fbb15271740 20 client.17265 trim_cache size 0 max 16384 2024-03-24T09:27:37.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 Processor -- start 2024-03-24T09:27:37.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 -- start start 2024-03-24T09:27:37.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826ab180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a8269d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a8268c210 con 0x564a8269f8f0 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a826903d0 con 0x564a82696d10 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb15271740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a82690990 con 0x564a826963f0 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826b7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52980/0 (socket says 172.21.15.116:52980) 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 -- 172.21.15.116:0/4214239185 learned_addr learned my addr 172.21.15.116:0/4214239185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb0b7fe640 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a8269d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 -- 172.21.15.116:0/4214239185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 msgr2=0x564a826ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 -- 172.21.15.116:0/4214239185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 msgr2=0x564a8269d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a8269d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 -- 172.21.15.116:0/4214239185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a82597c80 con 0x564a8269f8f0 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb10c32640 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826b7760 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbb04034ec0 tx=0x7fbb04034d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd1219eb6ab8b08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.581+0000 7fbb01ffb640 1 -- 172.21.15.116:0/4214239185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04005bd0 con 0x564a8269f8f0 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb01ffb640 1 -- 172.21.15.116:0/4214239185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb04032460 con 0x564a8269f8f0 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb01ffb640 1 -- 172.21.15.116:0/4214239185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04032a70 con 0x564a8269f8f0 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 -- 172.21.15.116:0/4214239185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 msgr2=0x564a826b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826b7760 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fbb04034ec0 tx=0x7fbb04034d90 comp rx=0 tx=0).stop 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 -- 172.21.15.116:0/4214239185 shutdown_connections 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826b7760 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a8269d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 --2- 172.21.15.116:0/4214239185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 -- 172.21.15.116:0/4214239185 >> 172.21.15.116:0/4214239185 conn(0x564a82690b20 msgr2=0x564a8268fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 -- 172.21.15.116:0/4214239185 shutdown_connections 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 -- 172.21.15.116:0/4214239185 wait complete. 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.582+0000 7fbb15271740 1 -- start start 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 Processor -- start 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826a7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a826a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a826a3850 con 0x564a8269f8f0 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564a826a39c0 con 0x564a82696d10 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564a826a3b80 con 0x564a826963f0 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb15271740 10 client.0 _ll_register_callbacks cb 0x7fbafc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb0b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a826a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52990/0 (socket says 172.21.15.116:52990) 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb0bfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 -- 172.21.15.116:0/267197711 learned_addr learned my addr 172.21.15.116:0/267197711 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb0b7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a826a2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:47994/0 (socket says 172.21.15.116:47994) 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 -- 172.21.15.116:0/267197711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 msgr2=0x564a826a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 -- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 msgr2=0x564a826a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a826a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 -- 172.21.15.116:0/267197711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a825fbbd0 con 0x564a8269f8f0 2024-03-24T09:27:37.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb0b7fe640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a826a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbb10c32640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a3290 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbb04064040 tx=0x7fbb04004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce1abd2d42b1075 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.583+0000 7fbaeb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/267197711 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04004650 con 0x564a8269f8f0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/267197711 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fbb04032680 con 0x564a8269f8f0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbafc09f060 con 0x564a8269f8f0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/267197711 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbb040047b0 con 0x564a8269f8f0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 10 client.17271 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564a825bd920 con 0x564a8269f8f0 2024-03-24T09:27:37.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/267197711 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbafc09f060 con 0x564a8269f8f0 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/267197711 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fbb04068050 con 0x564a8269f8f0 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 10 client.17271 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 10 client.17271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 20 client.17271 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a82715f40 0x564a82718360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 4 client.17271 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:27:37.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a82388890 con 0x564a82715f40 2024-03-24T09:27:37.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.584+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a82715f40 0x564a82718360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:37.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb0bfff640 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a82715f40 0x564a82718360 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7485f31cba21dd27 server_cookie=da9e69d541f648f4 in_seq=0 out_seq=0 2024-03-24T09:27:37.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbaeb7fe640 10 client.17271 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:37.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbaeb7fe640 1 -- 172.21.15.116:0/267197711 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a82388890 con 0x564a82715f40 2024-03-24T09:27:37.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbaeb7fe640 10 client.17271 handle_command_reply: tid=0 2024-03-24T09:27:37.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 client.17271 shutdown 2024-03-24T09:27:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 msgr2=0x564a826a3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a3290 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fbb04064040 tx=0x7fbb04004250 comp rx=0 tx=0).stop 2024-03-24T09:27:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 shutdown_connections 2024-03-24T09:27:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a8269f8f0 0x564a826a3290 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564a82715f40 0x564a82718360 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564a82696d10 0x564a826a2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbaeb7fe640 0 client.17271 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 --2- 172.21.15.116:0/267197711 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564a826963f0 0x564a826a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 >> 172.21.15.116:0/267197711 conn(0x564a82690b20 msgr2=0x564a82691ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 shutdown_connections 2024-03-24T09:27:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 1 -- 172.21.15.116:0/267197711 wait complete. 2024-03-24T09:27:37.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.585+0000 7fbb15271740 20 client.17271 trim_cache size 0 max 16384 2024-03-24T09:27:37.601 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbaf80a2230 msgr2=0x7fbaf80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbaf80a2230 0x7fbaf80a46f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbb04004af0 tx=0x7fbb04034e30 comp rx=0 tx=0).stop 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 msgr2=0x7fbb0c19d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 0x7fbb0c19d8f0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00ea60 tx=0x7fbafc00ef20 comp rx=0 tx=0).stop 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 shutdown_connections 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb0c10bd50 0x7fbb0c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbaf80a2230 0x7fbaf80a46f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb0c1032a0 0x7fbb0c19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 --2- 172.21.15.116:0/324153316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c102980 0x7fbb0c19d8f0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:37.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 >> 172.21.15.116:0/324153316 conn(0x7fbb0c077770 msgr2=0x7fbb0c0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:37.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 shutdown_connections 2024-03-24T09:27:37.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:37.586+0000 7fbb12695640 1 -- 172.21.15.116:0/324153316 wait complete. 2024-03-24T09:27:38.605 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:27:38.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:27:38.605 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:27:38.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 -- 172.21.15.116:0/3564809156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee741005a0 msgr2=0x7fee74100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/3564809156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee741005a0 0x7fee74100a20 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fee64009900 tx=0x7fee6402bc30 comp rx=0 tx=0).stop 2024-03-24T09:27:38.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 -- 172.21.15.116:0/3564809156 shutdown_connections 2024-03-24T09:27:38.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/3564809156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee74100f60 0x7fee7410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/3564809156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee741005a0 0x7fee74100a20 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/3564809156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee74106660 0x7fee74106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 -- 172.21.15.116:0/3564809156 >> 172.21.15.116:0/3564809156 conn(0x7fee74077710 msgr2=0x7fee74077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 -- 172.21.15.116:0/3564809156 shutdown_connections 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 -- 172.21.15.116:0/3564809156 wait complete. 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 Processor -- start 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.720+0000 7fee7ae2a640 1 -- start start 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee741005a0 0x7fee7419d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 0x7fee7419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee74106660 0x7fee74197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee74198090 con 0x7fee74100f60 2024-03-24T09:27:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fee74198200 con 0x7fee74106660 2024-03-24T09:27:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fee741983c0 con 0x7fee741005a0 2024-03-24T09:27:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 0x7fee7419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 0x7fee7419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53012/0 (socket says 172.21.15.116:53012) 2024-03-24T09:27:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee741005a0 0x7fee7419d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 -- 172.21.15.116:0/1885903203 learned_addr learned my addr 172.21.15.116:0/1885903203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee793c7640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee74106660 0x7fee74197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 -- 172.21.15.116:0/1885903203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee741005a0 msgr2=0x7fee7419d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee741005a0 0x7fee7419d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 -- 172.21.15.116:0/1885903203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee74106660 msgr2=0x7fee74197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee74106660 0x7fee74197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 -- 172.21.15.116:0/1885903203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee64009560 con 0x7fee74100f60 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee6bfff640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 0x7fee7419ddd0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fee64005940 tx=0x7fee64005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee69ffb640 1 -- 172.21.15.116:0/1885903203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64004350 con 0x7fee74100f60 2024-03-24T09:27:38.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee69ffb640 1 -- 172.21.15.116:0/1885903203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee640044b0 con 0x7fee74100f60 2024-03-24T09:27:38.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee69ffb640 1 -- 172.21.15.116:0/1885903203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640053c0 con 0x7fee74100f60 2024-03-24T09:27:38.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.721+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee74198640 con 0x7fee74100f60 2024-03-24T09:27:38.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.722+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee741a4930 con 0x7fee74100f60 2024-03-24T09:27:38.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee69ffb640 1 -- 172.21.15.116:0/1885903203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fee64005520 con 0x7fee74100f60 2024-03-24T09:27:38.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 Processor -- start 2024-03-24T09:27:38.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 -- start start 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee69ffb640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fee5c0a22b0 0x7fee5c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d8eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d97720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d97c60 0x55a730d9a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a730cfc530 con 0x55a730d97c60 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a730cf9a60 con 0x55a730d8e760 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee7da06740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a730cf5240 con 0x55a730d8f080 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee793c7640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fee5c0a22b0 0x7fee5c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d97720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d8eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d8eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48022/0 (socket says 172.21.15.116:48022) 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d97720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56348/0 (socket says 172.21.15.116:56348) 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee78bc6640 1 -- 172.21.15.116:0/1887685688 learned_addr learned my addr 172.21.15.116:0/1887685688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee69ffb640 1 -- 172.21.15.116:0/1885903203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fee640f03e0 con 0x7fee74100f60 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.724+0000 7fee793c7640 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d97c60 0x55a730d9a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee78bc6640 1 -- 172.21.15.116:0/1887685688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 msgr2=0x55a730d97720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee78bc6640 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d97720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee78bc6640 1 -- 172.21.15.116:0/1887685688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d97c60 msgr2=0x55a730d9a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee78bc6640 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d97c60 0x55a730d9a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee78bc6640 1 -- 172.21.15.116:0/1887685688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a730cf38c0 con 0x55a730d8e760 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee6bfff640 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d97720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee793c7640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fee5c0a22b0 0x7fee5c0a4770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee6c005fd0 tx=0x7fee6c009f10 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee78bc6640 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d8eb40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fee70009ca0 tx=0x7fee70011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93eb877f486dcca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee5affd640 1 -- 172.21.15.116:0/1887685688 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70005220 con 0x55a730d8e760 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee5affd640 1 -- 172.21.15.116:0/1887685688 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee70004c50 con 0x55a730d8e760 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.725+0000 7fee5affd640 1 -- 172.21.15.116:0/1887685688 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70019630 con 0x55a730d8e760 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 -- 172.21.15.116:0/1887685688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 msgr2=0x55a730d8eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d8eb40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fee70009ca0 tx=0x7fee70011810 comp rx=0 tx=0).stop 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 -- 172.21.15.116:0/1887685688 shutdown_connections 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d97c60 0x55a730d9a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d97720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 --2- 172.21.15.116:0/1887685688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d8eb40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 -- 172.21.15.116:0/1887685688 >> 172.21.15.116:0/1887685688 conn(0x55a730d88e90 msgr2=0x55a730d89270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 -- 172.21.15.116:0/1887685688 shutdown_connections 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 -- 172.21.15.116:0/1887685688 wait complete. 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 -- start start 2024-03-24T09:27:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.726+0000 7fee7da06740 1 Processor -- start 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d94c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d951a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d956e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a730c99f80 con 0x55a730d8e760 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a7309c84b0 con 0x55a730d97c60 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a730c7aad0 con 0x55a730d8f080 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 10 client.0 _ll_register_callbacks cb 0x55a730cadb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d94c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d94c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53032/0 (socket says 172.21.15.116:53032) 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 -- 172.21.15.116:0/2376709137 learned_addr learned my addr 172.21.15.116:0/2376709137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee793c7640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d956e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee6bfff640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 -- 172.21.15.116:0/2376709137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 msgr2=0x55a730d951a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d951a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 -- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 msgr2=0x55a730d956e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d956e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 -- 172.21.15.116:0/2376709137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee700098d0 con 0x55a730d8e760 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee793c7640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d956e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:38.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee78bc6640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d94c60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fee700055d0 tx=0x7fee700048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dcdddb595f76e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee53fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee53fff640 1 -- 172.21.15.116:0/2376709137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70023840 con 0x55a730d8e760 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee53fff640 1 -- 172.21.15.116:0/2376709137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fee70004dc0 con 0x55a730d8e760 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.727+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a730d197a0 con 0x55a730d8e760 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee53fff640 1 -- 172.21.15.116:0/2376709137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fee70004ae0 con 0x55a730d8e760 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 10 client.17283 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a730cab6d0 con 0x55a730d8e760 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee53fff640 1 -- 172.21.15.116:0/2376709137 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a730d197a0 con 0x55a730d8e760 2024-03-24T09:27:38.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee53fff640 1 -- 172.21.15.116:0/2376709137 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fee70021040 con 0x55a730d8e760 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 10 client.17283 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 10 client.17283 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 20 client.17283 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730da7c80 0x55a730daa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 4 client.17283 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a730ceb1f0 con 0x55a730da7c80 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.728+0000 7fee6bfff640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730da7c80 0x55a730daa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee6bfff640 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730da7c80 0x55a730daa090 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dffa0103acd792a server_cookie=36884c278262cdf in_seq=0 out_seq=0 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee53fff640 10 client.17283 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee53fff640 1 -- 172.21.15.116:0/2376709137 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a730ceb1f0 con 0x55a730da7c80 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee53fff640 10 client.17283 handle_command_reply: tid=0 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 client.17283 shutdown 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 msgr2=0x55a730d94c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d94c60 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fee700055d0 tx=0x7fee700048f0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 shutdown_connections 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730da7c80 0x55a730daa090 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d956e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee53fff640 0 client.17283 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d951a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 --2- 172.21.15.116:0/2376709137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d94c60 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.729+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 >> 172.21.15.116:0/2376709137 conn(0x55a730d88e90 msgr2=0x55a730d8abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.730+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 shutdown_connections 2024-03-24T09:27:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.730+0000 7fee7da06740 1 -- 172.21.15.116:0/2376709137 wait complete. 2024-03-24T09:27:38.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.730+0000 7fee7da06740 20 client.17283 trim_cache size 0 max 16384 2024-03-24T09:27:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 Processor -- start 2024-03-24T09:27:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 -- start start 2024-03-24T09:27:38.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d97390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 0x55a730da34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d97c60 0x55a730daf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a730d87350 con 0x55a730d8f080 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a730da47a0 con 0x55a730d8e760 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee7da06740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a730d84620 con 0x55a730d97c60 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 0x55a730da34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 0x55a730da34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53048/0 (socket says 172.21.15.116:53048) 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 -- 172.21.15.116:0/4160807799 learned_addr learned my addr 172.21.15.116:0/4160807799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee78bc6640 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d97390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee793c7640 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d97c60 0x55a730daf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 -- 172.21.15.116:0/4160807799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d97c60 msgr2=0x55a730daf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d97c60 0x55a730daf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 -- 172.21.15.116:0/4160807799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 msgr2=0x55a730d97390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d97390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee6bfff640 1 -- 172.21.15.116:0/4160807799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a730c8fc60 con 0x55a730d8f080 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.750+0000 7fee78bc6640 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d97390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee6bfff640 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 0x55a730da34f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fee640b9590 tx=0x7fee64031810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faff9e8cdc09cf5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee5a7fc640 1 -- 172.21.15.116:0/4160807799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640bcbc0 con 0x55a730d8f080 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee5a7fc640 1 -- 172.21.15.116:0/4160807799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee640c0d10 con 0x55a730d8f080 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee5a7fc640 1 -- 172.21.15.116:0/4160807799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee640baec0 con 0x55a730d8f080 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 -- 172.21.15.116:0/4160807799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 msgr2=0x55a730da34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 0x55a730da34f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fee640b9590 tx=0x7fee64031810 comp rx=0 tx=0).stop 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 -- 172.21.15.116:0/4160807799 shutdown_connections 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d97c60 0x55a730daf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8f080 0x55a730da34f0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 --2- 172.21.15.116:0/4160807799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d8e760 0x55a730d97390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.751+0000 7fee7da06740 1 -- 172.21.15.116:0/4160807799 >> 172.21.15.116:0/4160807799 conn(0x55a730d88e90 msgr2=0x55a730d87660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 -- 172.21.15.116:0/4160807799 shutdown_connections 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 -- 172.21.15.116:0/4160807799 wait complete. 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 -- start start 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 Processor -- start 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d9fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d9caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d9d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a730da00b0 con 0x55a730d8e760 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a730da0220 con 0x55a730d97c60 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a730d9d570 con 0x55a730d8f080 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee7da06740 10 client.0 _ll_register_callbacks cb 0x7fee70009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d9fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee78bc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d9fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53062/0 (socket says 172.21.15.116:53062) 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee78bc6640 1 -- 172.21.15.116:0/2670059851 learned_addr learned my addr 172.21.15.116:0/2670059851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee6bfff640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d9caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.752+0000 7fee793c7640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d9d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee78bc6640 1 -- 172.21.15.116:0/2670059851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 msgr2=0x55a730d9caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee78bc6640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d9caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee78bc6640 1 -- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 msgr2=0x55a730d9d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee78bc6640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d9d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee78bc6640 1 -- 172.21.15.116:0/2670059851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a730cab6d0 con 0x55a730d8e760 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee78bc6640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d9fb20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fee70019ae0 tx=0x7fee70009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62d5b25aec0b8b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee53fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee53fff640 1 -- 172.21.15.116:0/2670059851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70011d10 con 0x55a730d8e760 2024-03-24T09:27:38.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee53fff640 1 -- 172.21.15.116:0/2670059851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fee70000e40 con 0x55a730d8e760 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee70004c50 con 0x55a730d8e760 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee53fff640 1 -- 172.21.15.116:0/2670059851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fee70009e50 con 0x55a730d8e760 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 10 client.17289 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7fee74198640 con 0x55a730d8e760 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee53fff640 1 -- 172.21.15.116:0/2670059851 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fee70004c50 con 0x55a730d8e760 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee53fff640 1 -- 172.21.15.116:0/2670059851 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fee70023760 con 0x55a730d8e760 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 10 client.17289 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 10 client.17289 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:38.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 20 client.17289 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730e0ee80 0x55a730e112a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 4 client.17289 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.753+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a7309f4860 con 0x55a730e0ee80 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.754+0000 7fee6bfff640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730e0ee80 0x55a730e112a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.754+0000 7fee6bfff640 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730e0ee80 0x55a730e112a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80a3db4c08979777 server_cookie=66c3d4e1ea8ce0fc in_seq=0 out_seq=0 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.754+0000 7fee53fff640 10 client.17289 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee53fff640 1 -- 172.21.15.116:0/2670059851 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a7309f4860 con 0x55a730e0ee80 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee53fff640 10 client.17289 handle_command_reply: tid=0 2024-03-24T09:27:38.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 client.17289 shutdown 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 msgr2=0x55a730d9fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d9fb20 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fee70019ae0 tx=0x7fee70009c30 comp rx=0 tx=0).stop 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 shutdown_connections 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55a730e0ee80 0x55a730e112a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a730d97c60 0x55a730d9d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a730d8f080 0x55a730d9caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee53fff640 0 client.17289 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 --2- 172.21.15.116:0/2670059851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a730d8e760 0x55a730d9fb20 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 >> 172.21.15.116:0/2670059851 conn(0x55a730d88e90 msgr2=0x55a730d95490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 shutdown_connections 2024-03-24T09:27:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.755+0000 7fee7da06740 1 -- 172.21.15.116:0/2670059851 wait complete. 2024-03-24T09:27:38.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7da06740 20 client.17289 trim_cache size 0 max 16384 2024-03-24T09:27:38.766 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:27:38.766 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:27:38.766 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "17a6910b-d981-4e55-af0b-d30f64f83e7f", 2024-03-24T09:27:38.767 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:27:38.767 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:27:38.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fee5c0a22b0 msgr2=0x7fee5c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fee5c0a22b0 0x7fee5c0a4770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee6c005fd0 tx=0x7fee6c009f10 comp rx=0 tx=0).stop 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 msgr2=0x7fee7419ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 0x7fee7419ddd0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fee64005940 tx=0x7fee64005df0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 shutdown_connections 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee74106660 0x7fee74197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee74100f60 0x7fee7419ddd0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fee5c0a22b0 0x7fee5c0a4770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 --2- 172.21.15.116:0/1885903203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fee741005a0 0x7fee7419d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 >> 172.21.15.116:0/1885903203 conn(0x7fee74077710 msgr2=0x7fee740ff350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 shutdown_connections 2024-03-24T09:27:38.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.756+0000 7fee7ae2a640 1 -- 172.21.15.116:0/1885903203 wait complete. 2024-03-24T09:27:38.770 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:27:38.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:27:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 -- 172.21.15.116:0/2339054571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581016a0 msgr2=0x7f3258101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 --2- 172.21.15.116:0/2339054571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581016a0 0x7f3258101b20 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3244009890 tx=0x7f324402bb70 comp rx=0 tx=0).stop 2024-03-24T09:27:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 -- 172.21.15.116:0/2339054571 shutdown_connections 2024-03-24T09:27:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 --2- 172.21.15.116:0/2339054571 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3258102060 0x7f325810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 --2- 172.21.15.116:0/2339054571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581016a0 0x7f3258101b20 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 --2- 172.21.15.116:0/2339054571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32581076b0 0x7f3258107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 -- 172.21.15.116:0/2339054571 >> 172.21.15.116:0/2339054571 conn(0x7f3258078b10 msgr2=0x7f3258078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 -- 172.21.15.116:0/2339054571 shutdown_connections 2024-03-24T09:27:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.890+0000 7f325ea9d640 1 -- 172.21.15.116:0/2339054571 wait complete. 2024-03-24T09:27:38.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 Processor -- start 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 -- start start 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32581016a0 0x7f325819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3258102060 0x7f325819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 0x7f3258197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32581980e0 con 0x7f32581076b0 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3258198250 con 0x7f32581016a0 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325ea9d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3258198410 con 0x7f3258102060 2024-03-24T09:27:38.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325d03a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 0x7f3258197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325d03a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 0x7f3258197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53082/0 (socket says 172.21.15.116:53082) 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325d03a640 1 -- 172.21.15.116:0/1042397706 learned_addr learned my addr 172.21.15.116:0/1042397706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.891+0000 7f325c839640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32581016a0 0x7f325819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f3257fff640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3258102060 0x7f325819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325d03a640 1 -- 172.21.15.116:0/1042397706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3258102060 msgr2=0x7f325819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325d03a640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3258102060 0x7f325819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325d03a640 1 -- 172.21.15.116:0/1042397706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32581016a0 msgr2=0x7f325819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325d03a640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32581016a0 0x7f325819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325d03a640 1 -- 172.21.15.116:0/1042397706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3244009490 con 0x7f32581076b0 2024-03-24T09:27:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325d03a640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 0x7f3258197a90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f325000eb10 tx=0x7f325000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f3255ffb640 1 -- 172.21.15.116:0/1042397706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325000ce20 con 0x7f32581076b0 2024-03-24T09:27:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f3255ffb640 1 -- 172.21.15.116:0/1042397706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3250005020 con 0x7f32581076b0 2024-03-24T09:27:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32581986f0 con 0x7f32581076b0 2024-03-24T09:27:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f3255ffb640 1 -- 172.21.15.116:0/1042397706 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250010690 con 0x7f32581076b0 2024-03-24T09:27:38.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.892+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32581a4980 con 0x7f32581076b0 2024-03-24T09:27:38.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.895+0000 7f3255ffb640 1 -- 172.21.15.116:0/1042397706 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f3250005190 con 0x7f32581076b0 2024-03-24T09:27:38.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 Processor -- start 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 -- start start 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3255ffb640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32480a2330 0x7f32480a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a401a860 0x5621a401ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4008830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a4022730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621a3f32f30 con 0x5621a401a860 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621a3f7c810 con 0x5621a40083b0 2024-03-24T09:27:38.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.896+0000 7f3261684740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5621a3f79d40 con 0x5621a4008d70 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f3257fff640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32480a2330 0x7f32480a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f3257fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4008830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a4022730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a4022730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:56398/0 (socket says 172.21.15.116:56398) 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f3257fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4008830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:48046/0 (socket says 172.21.15.116:48046) 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 -- 172.21.15.116:0/31113252 learned_addr learned my addr 172.21.15.116:0/31113252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325c839640 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a401a860 0x5621a401ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f3255ffb640 1 -- 172.21.15.116:0/1042397706 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3250014070 con 0x7f32581076b0 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 -- 172.21.15.116:0/31113252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 msgr2=0x5621a4008830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4008830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 -- 172.21.15.116:0/31113252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a401a860 msgr2=0x5621a401ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a401a860 0x5621a401ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f325d03a640 1 -- 172.21.15.116:0/31113252 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a3f36ea0 con 0x5621a4008d70 2024-03-24T09:27:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f3257fff640 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4008830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.897+0000 7f3257fff640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32480a2330 0x7f32480a47f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3244037040 tx=0x7f3244035b40 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.898+0000 7f325d03a640 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a4022730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f32500a3270 tx=0x7f325009a960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=778547fcc05f08fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.898+0000 7f3242ffd640 1 -- 172.21.15.116:0/31113252 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325009f150 con 0x5621a4008d70 2024-03-24T09:27:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.898+0000 7f3242ffd640 1 -- 172.21.15.116:0/31113252 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32500a37a0 con 0x5621a4008d70 2024-03-24T09:27:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.898+0000 7f3242ffd640 1 -- 172.21.15.116:0/31113252 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250097670 con 0x5621a4008d70 2024-03-24T09:27:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 -- 172.21.15.116:0/31113252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 msgr2=0x5621a4022730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a4022730 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f32500a3270 tx=0x7f325009a960 comp rx=0 tx=0).stop 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 -- 172.21.15.116:0/31113252 shutdown_connections 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a4022730 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4008830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 --2- 172.21.15.116:0/31113252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a401a860 0x5621a401ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 -- 172.21.15.116:0/31113252 >> 172.21.15.116:0/31113252 conn(0x5621a400ae70 msgr2=0x5621a400b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 -- 172.21.15.116:0/31113252 shutdown_connections 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 -- 172.21.15.116:0/31113252 wait complete. 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 -- start start 2024-03-24T09:27:38.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.899+0000 7f3261684740 1 Processor -- start 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 0x5621a401fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a40201a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a401a860 0x5621a40206e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621a3f75520 con 0x5621a40083b0 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621a3c48430 con 0x5621a401a860 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5621a3c04e00 con 0x5621a4008d70 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3261684740 10 client.0 _ll_register_callbacks cb 0x5621a3f13d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 0x5621a401fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 0x5621a401fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53106/0 (socket says 172.21.15.116:53106) 2024-03-24T09:27:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 -- 172.21.15.116:0/3322941 learned_addr learned my addr 172.21.15.116:0/3322941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325d03a640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a401a860 0x5621a40206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3257fff640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a40201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 -- 172.21.15.116:0/3322941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 msgr2=0x5621a40201a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a40201a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 -- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a401a860 msgr2=0x5621a40206e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a401a860 0x5621a40206e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 -- 172.21.15.116:0/3322941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32581a4980 con 0x5621a40083b0 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f325c839640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 0x5621a401fc60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f324c0126b0 tx=0x7f324c012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5b3034680a8f34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.900+0000 7f3237fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3237fff640 1 -- 172.21.15.116:0/3322941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c017070 con 0x5621a40083b0 2024-03-24T09:27:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3237fff640 1 -- 172.21.15.116:0/3322941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f324c004030 con 0x5621a40083b0 2024-03-24T09:27:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5621a3c747e0 con 0x5621a40083b0 2024-03-24T09:27:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3237fff640 1 -- 172.21.15.116:0/3322941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f324c0122e0 con 0x5621a40083b0 2024-03-24T09:27:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 10 client.17301 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:38.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5621a3f73ba0 con 0x5621a40083b0 2024-03-24T09:27:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3237fff640 1 -- 172.21.15.116:0/3322941 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5621a3c747e0 con 0x5621a40083b0 2024-03-24T09:27:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3237fff640 1 -- 172.21.15.116:0/3322941 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f324c003880 con 0x5621a40083b0 2024-03-24T09:27:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 10 client.17301 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 10 client.17301 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 20 client.17301 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40414a0 0x5621a40438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 4 client.17301 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:27:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.901+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621a3cb1720 con 0x5621a40414a0 2024-03-24T09:27:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.902+0000 7f3257fff640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40414a0 0x5621a40438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.902+0000 7f3257fff640 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40414a0 0x5621a40438e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=357699e325085ec1 server_cookie=dc06d215177d2ce6 in_seq=0 out_seq=0 2024-03-24T09:27:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.903+0000 7f3237fff640 10 client.17301 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3237fff640 1 -- 172.21.15.116:0/3322941 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621a3cb1720 con 0x5621a40414a0 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3237fff640 10 client.17301 handle_command_reply: tid=0 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 client.17301 shutdown 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 msgr2=0x5621a401fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 0x5621a401fc60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f324c0126b0 tx=0x7f324c012b70 comp rx=0 tx=0).stop 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 shutdown_connections 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40414a0 0x5621a40438e0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a401a860 0x5621a40206e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3237fff640 0 client.17301 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a4008d70 0x5621a40201a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 --2- 172.21.15.116:0/3322941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a40083b0 0x5621a401fc60 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 >> 172.21.15.116:0/3322941 conn(0x5621a400ae70 msgr2=0x5621a4009d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 shutdown_connections 2024-03-24T09:27:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 1 -- 172.21.15.116:0/3322941 wait complete. 2024-03-24T09:27:38.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.904+0000 7f3261684740 20 client.17301 trim_cache size 0 max 16384 2024-03-24T09:27:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.928+0000 7f3261684740 1 Processor -- start 2024-03-24T09:27:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 -- start start 2024-03-24T09:27:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4022390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a400a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40182d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3257fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a400a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3257fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a400a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53108/0 (socket says 172.21.15.116:53108) 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3257fff640 1 -- 172.21.15.116:0/2944861848 learned_addr learned my addr 172.21.15.116:0/2944861848 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621a3efa430 con 0x5621a4008d70 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621a4009670 con 0x5621a40083b0 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5621a4004550 con 0x5621a401a860 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f325c839640 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4022390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.929+0000 7f325d03a640 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40182d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f3257fff640 1 -- 172.21.15.116:0/2944861848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 msgr2=0x5621a40182d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f3257fff640 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40182d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f3257fff640 1 -- 172.21.15.116:0/2944861848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 msgr2=0x5621a4022390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f3257fff640 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4022390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f3257fff640 1 -- 172.21.15.116:0/2944861848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a3f0fb20 con 0x5621a4008d70 2024-03-24T09:27:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f325d03a640 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40182d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:27:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f3257fff640 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a400a390 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f324402c0b0 tx=0x7f324404a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ac130b797c7a0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f32427fc640 1 -- 172.21.15.116:0/2944861848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244047070 con 0x5621a4008d70 2024-03-24T09:27:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f32427fc640 1 -- 172.21.15.116:0/2944861848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f324402ce90 con 0x5621a4008d70 2024-03-24T09:27:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.930+0000 7f32427fc640 1 -- 172.21.15.116:0/2944861848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324402c770 con 0x5621a4008d70 2024-03-24T09:27:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 msgr2=0x5621a400a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a400a390 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f324402c0b0 tx=0x7f324404a000 comp rx=0 tx=0).stop 2024-03-24T09:27:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 shutdown_connections 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40182d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a400a390 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 --2- 172.21.15.116:0/2944861848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a4022390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 >> 172.21.15.116:0/2944861848 conn(0x5621a400ae70 msgr2=0x5621a4009c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.931+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 shutdown_connections 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 -- 172.21.15.116:0/2944861848 wait complete. 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 -- start start 2024-03-24T09:27:38.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 Processor -- start 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a401f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a401fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40202e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5621a40a67e0 con 0x5621a4008d70 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621a40a6920 con 0x5621a40083b0 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5621a40a6a60 con 0x5621a401a860 2024-03-24T09:27:38.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3261684740 10 client.0 _ll_register_callbacks cb 0x7f324c009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.932+0000 7f3257fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a401fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a401fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53118/0 (socket says 172.21.15.116:53118) 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 -- 172.21.15.116:0/2456337189 learned_addr learned my addr 172.21.15.116:0/2456337189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f325c839640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a401f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f325d03a640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 -- 172.21.15.116:0/2456337189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 msgr2=0x5621a40202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 -- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 msgr2=0x5621a401f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a401f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 -- 172.21.15.116:0/2456337189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a3f36ea0 con 0x5621a4008d70 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3257fff640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a401fda0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3244005b60 tx=0x7f3244004130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa7942035403f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3237fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3237fff640 1 -- 172.21.15.116:0/2456337189 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244047070 con 0x5621a4008d70 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3237fff640 1 -- 172.21.15.116:0/2456337189 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3244055780 con 0x5621a4008d70 2024-03-24T09:27:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32500a37a0 con 0x5621a4008d70 2024-03-24T09:27:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3237fff640 1 -- 172.21.15.116:0/2456337189 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3244055250 con 0x5621a4008d70 2024-03-24T09:27:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3261684740 10 client.17307 fetch_fsmap learned FSMap version 35 2024-03-24T09:27:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.933+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7f3244030070 con 0x5621a4008d70 2024-03-24T09:27:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3237fff640 1 -- 172.21.15.116:0/2456337189 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f32500a37a0 con 0x5621a4008d70 2024-03-24T09:27:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3237fff640 1 -- 172.21.15.116:0/2456337189 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3244066070 con 0x5621a4008d70 2024-03-24T09:27:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3261684740 10 client.17307 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:27:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3261684740 10 client.17307 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:27:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3261684740 20 client.17307 populate_metadata read hostname 'smithi116' 2024-03-24T09:27:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3261684740 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40a7f90 0x5621a40aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:27:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3261684740 4 client.17307 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:27:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5621a3d00890 con 0x5621a40a7f90 2024-03-24T09:27:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.934+0000 7f325c839640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40a7f90 0x5621a40aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:27:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.935+0000 7f3237fff640 10 client.17307 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.935+0000 7f325c839640 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40a7f90 0x5621a40aa3b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3a359f0e440c60f server_cookie=5e243ee742ac0c95 in_seq=0 out_seq=0 2024-03-24T09:27:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.935+0000 7f3237fff640 1 -- 172.21.15.116:0/2456337189 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5621a3d00890 con 0x5621a40a7f90 2024-03-24T09:27:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.935+0000 7f3237fff640 10 client.17307 handle_command_reply: tid=0 2024-03-24T09:27:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.935+0000 7f3261684740 1 client.17307 shutdown 2024-03-24T09:27:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.935+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 msgr2=0x5621a401fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a401fda0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f3244005b60 tx=0x7f3244004130 comp rx=0 tx=0).stop 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 shutdown_connections 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5621a401a860 0x5621a40202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5621a4008d70 0x5621a401fda0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5621a40a7f90 0x5621a40aa3b0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 --2- 172.21.15.116:0/2456337189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621a40083b0 0x5621a401f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 >> 172.21.15.116:0/2456337189 conn(0x5621a400ae70 msgr2=0x5621a40072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3237fff640 0 client.17307 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 shutdown_connections 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 1 -- 172.21.15.116:0/2456337189 wait complete. 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.936+0000 7f3261684740 20 client.17307 trim_cache size 0 max 16384 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:27:38.946 INFO:teuthology.orchestra.run.smithi116.stdout: "17a6910b-d981-4e55-af0b-d30f64f83e7f": { 2024-03-24T09:27:38.947 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:27:38.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "17a6910b-d981-4e55-af0b-d30f64f83e7f", 2024-03-24T09:27:38.947 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:27:38.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:27:38.947 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:27:38.947 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32480a2330 msgr2=0x7f32480a47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32480a2330 0x7f32480a47f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3244037040 tx=0x7f3244035b40 comp rx=0 tx=0).stop 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 msgr2=0x7f3258197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 0x7f3258197a90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f325000eb10 tx=0x7f325000ef10 comp rx=0 tx=0).stop 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 shutdown_connections 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32581076b0 0x7f3258197a90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f32480a2330 0x7f32480a47f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3258102060 0x7f325819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 --2- 172.21.15.116:0/1042397706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32581016a0 0x7f325819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:27:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 >> 172.21.15.116:0/1042397706 conn(0x7f3258078b10 msgr2=0x7f3258104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:27:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 shutdown_connections 2024-03-24T09:27:38.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:27:38.937+0000 7f325ea9d640 1 -- 172.21.15.116:0/1042397706 wait complete. 2024-03-24T09:27:38.950 INFO:tasks.cephfs.filesystem:scrub status for tag:17a6910b-d981-4e55-af0b-d30f64f83e7f - {'path': '/', 'tag': '17a6910b-d981-4e55-af0b-d30f64f83e7f', 'options': 'recursive,force'} 2024-03-24T09:27:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:38 smithi143 ceph-mon[35965]: pgmap v169: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:38 smithi117 ceph-mon[35788]: pgmap v169: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:38 smithi116 ceph-mon[32207]: pgmap v169: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:40.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:27:40.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:39 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:27:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:40.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:39 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:27:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:40.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:40.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:39 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:27:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:40 smithi143 ceph-mon[35965]: pgmap v170: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:40 smithi117 ceph-mon[35788]: pgmap v170: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:40 smithi116 ceph-mon[32207]: pgmap v170: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:43.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:42 smithi143 ceph-mon[35965]: pgmap v171: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:42 smithi117 ceph-mon[35788]: pgmap v171: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:42 smithi116 ceph-mon[32207]: pgmap v171: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:44 smithi143 ceph-mon[35965]: pgmap v172: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:44 smithi117 ceph-mon[35788]: pgmap v172: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:44 smithi116 ceph-mon[32207]: pgmap v172: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:46 smithi143 ceph-mon[35965]: pgmap v173: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:46 smithi117 ceph-mon[35788]: pgmap v173: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:46 smithi116 ceph-mon[32207]: pgmap v173: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:49.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:48 smithi143 ceph-mon[35965]: pgmap v174: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:48 smithi117 ceph-mon[35788]: pgmap v174: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:48 smithi116 ceph-mon[32207]: pgmap v174: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:51.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:51 smithi143 ceph-mon[35965]: pgmap v175: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:51 smithi117 ceph-mon[35788]: pgmap v175: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:51.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:51 smithi116 ceph-mon[32207]: pgmap v175: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:27:53.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:53 smithi143 ceph-mon[35965]: pgmap v176: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:53.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:53 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:53 smithi117 ceph-mon[35788]: pgmap v176: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:53 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:53 smithi116 ceph-mon[32207]: pgmap v176: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:27:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:53 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:54.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:54 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:54.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:54 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:54.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:54 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:54.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:54 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:27:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:54 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:54.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:54 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:54.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:54 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:54.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:54 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:27:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:54 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:54 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:54 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:54.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:54 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:27:55.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:55 smithi143 ceph-mon[35965]: pgmap v177: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:55 smithi117 ceph-mon[35788]: pgmap v177: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:55.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:55 smithi116 ceph-mon[32207]: pgmap v177: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:57.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:57 smithi143 ceph-mon[35965]: pgmap v178: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:57 smithi117 ceph-mon[35788]: pgmap v178: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:57.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:57 smithi116 ceph-mon[32207]: pgmap v178: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:27:59.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:27:59 smithi143 ceph-mon[35965]: pgmap v179: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:59.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:27:59 smithi117 ceph-mon[35788]: pgmap v179: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:27:59.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:27:59 smithi116 ceph-mon[32207]: pgmap v179: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:01.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:01 smithi143 ceph-mon[35965]: pgmap v180: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:01 smithi117 ceph-mon[35788]: pgmap v180: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:01 smithi116 ceph-mon[32207]: pgmap v180: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:03.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:03 smithi143 ceph-mon[35965]: pgmap v181: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:03 smithi117 ceph-mon[35788]: pgmap v181: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:03.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:03 smithi116 ceph-mon[32207]: pgmap v181: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:05.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:05 smithi143 ceph-mon[35965]: pgmap v182: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:05 smithi117 ceph-mon[35788]: pgmap v182: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:05.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:05 smithi116 ceph-mon[32207]: pgmap v182: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:07 smithi117 ceph-mon[35788]: pgmap v183: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:07.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:07 smithi116 ceph-mon[32207]: pgmap v183: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:07.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:07 smithi143 ceph-mon[35965]: pgmap v183: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:08.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:28:08.953 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:28:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 -- 172.21.15.116:0/1723282652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 msgr2=0x7fcdb810da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 --2- 172.21.15.116:0/1723282652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb810da20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009900 tx=0x7fcdac02bcd0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 -- 172.21.15.116:0/1723282652 shutdown_connections 2024-03-24T09:28:09.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 --2- 172.21.15.116:0/1723282652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb810df60 0x7fcdb8110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 --2- 172.21.15.116:0/1723282652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb810da20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 --2- 172.21.15.116:0/1723282652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdb8073ac0 0x7fcdb8073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 -- 172.21.15.116:0/1723282652 >> 172.21.15.116:0/1723282652 conn(0x7fcdb8078b10 msgr2=0x7fcdb8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 -- 172.21.15.116:0/1723282652 shutdown_connections 2024-03-24T09:28:09.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.070+0000 7fcdbe125640 1 -- 172.21.15.116:0/1723282652 wait complete. 2024-03-24T09:28:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 Processor -- start 2024-03-24T09:28:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 -- start start 2024-03-24T09:28:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb8073ac0 0x7fcdb8199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdb810df60 0x7fcdb819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb819ecf0 con 0x7fcdb80743e0 2024-03-24T09:28:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb819ee60 con 0x7fcdb810df60 2024-03-24T09:28:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdbe125640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb819f020 con 0x7fcdb8073ac0 2024-03-24T09:28:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdb6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdb6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb819a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37172/0 (socket says 172.21.15.116:37172) 2024-03-24T09:28:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/2070044078 learned_addr learned my addr 172.21.15.116:0/2070044078 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdb810df60 0x7fcdb819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.071+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb8073ac0 0x7fcdb8199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/2070044078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb8073ac0 msgr2=0x7fcdb8199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb8073ac0 0x7fcdb8199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/2070044078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdb810df60 msgr2=0x7fcdb819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdb810df60 0x7fcdb819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/2070044078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac009560 con 0x7fcdb80743e0 2024-03-24T09:28:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb819a370 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcdac006010 tx=0x7fcdac005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb4ff9640 1 -- 172.21.15.116:0/2070044078 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0043d0 con 0x7fcdb80743e0 2024-03-24T09:28:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb4ff9640 1 -- 172.21.15.116:0/2070044078 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac003aa0 con 0x7fcdb80743e0 2024-03-24T09:28:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb819f2a0 con 0x7fcdb80743e0 2024-03-24T09:28:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdb4ff9640 1 -- 172.21.15.116:0/2070044078 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac005350 con 0x7fcdb80743e0 2024-03-24T09:28:09.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.072+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a6b80 con 0x7fcdb80743e0 2024-03-24T09:28:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.074+0000 7fcdb4ff9640 1 -- 172.21.15.116:0/2070044078 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fcdac003c10 con 0x7fcdb80743e0 2024-03-24T09:28:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 Processor -- start 2024-03-24T09:28:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 -- start start 2024-03-24T09:28:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb4ff9640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcda00a2230 0x7fcda00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da795b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da82190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da84ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4d99df30 con 0x558c4da826d0 2024-03-24T09:28:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4d9e7810 con 0x558c4da79af0 2024-03-24T09:28:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558c4d9e4d40 con 0x558c4da791d0 2024-03-24T09:28:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcda00a2230 0x7fcda00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb4ff9640 1 -- 172.21.15.116:0/2070044078 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcdac0f0ac0 con 0x7fcdb80743e0 2024-03-24T09:28:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da84ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb6ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da82190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da84ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37180/0 (socket says 172.21.15.116:37180) 2024-03-24T09:28:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da795b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45170/0 (socket says 172.21.15.116:45170) 2024-03-24T09:28:09.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb7fff640 1 -- 172.21.15.116:0/4228286346 learned_addr learned my addr 172.21.15.116:0/4228286346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/4228286346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 msgr2=0x558c4da795b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da795b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/4228286346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 msgr2=0x558c4da84ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da84ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.075+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/4228286346 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4d9a1ea0 con 0x558c4da79af0 2024-03-24T09:28:09.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcda00a2230 0x7fcda00a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009fd0 tx=0x7fcdb0009f40 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da82190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0edf70 tx=0x7fcdac0bce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d04f07b327cc38e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcd9dffb640 1 -- 172.21.15.116:0/4228286346 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0bd910 con 0x558c4da79af0 2024-03-24T09:28:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcd9dffb640 1 -- 172.21.15.116:0/4228286346 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac091760 con 0x558c4da79af0 2024-03-24T09:28:09.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcd9dffb640 1 -- 172.21.15.116:0/4228286346 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0b9880 con 0x558c4da79af0 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcdc0de6740 1 -- 172.21.15.116:0/4228286346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 msgr2=0x558c4da82190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.076+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da82190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0edf70 tx=0x7fcdac0bce10 comp rx=0 tx=0).stop 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- 172.21.15.116:0/4228286346 shutdown_connections 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da84ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da82190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/4228286346 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da795b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- 172.21.15.116:0/4228286346 >> 172.21.15.116:0/4228286346 conn(0x558c4da73900 msgr2=0x558c4da73ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- 172.21.15.116:0/4228286346 shutdown_connections 2024-03-24T09:28:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- 172.21.15.116:0/4228286346 wait complete. 2024-03-24T09:28:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- start start 2024-03-24T09:28:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 Processor -- start 2024-03-24T09:28:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 0x558c4da7f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da79af0 0x558c4da7fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da80230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4d9e0520 con 0x558c4da826d0 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4d6b3430 con 0x558c4da791d0 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558c4d66fe00 con 0x558c4da79af0 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.077+0000 7fcdc0de6740 10 client.0 _ll_register_callbacks cb 0x558c4d97ed90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da80230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 0x558c4da7f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da80230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37182/0 (socket says 172.21.15.116:37182) 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb77fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 0x558c4da7f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:58954/0 (socket says 172.21.15.116:58954) 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb77fe640 1 -- 172.21.15.116:0/3822023655 learned_addr learned my addr 172.21.15.116:0/3822023655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb6ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da79af0 0x558c4da7fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 -- 172.21.15.116:0/3822023655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da79af0 msgr2=0x558c4da7fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da79af0 0x558c4da7fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 -- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 msgr2=0x558c4da7f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 0x558c4da7f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 -- 172.21.15.116:0/3822023655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a6b80 con 0x558c4da826d0 2024-03-24T09:28:09.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 0x558c4da7f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da80230 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0005fd0 tx=0x7fcdb001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd0e510439d7ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcd96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcd96ffd640 1 -- 172.21.15.116:0/3822023655 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0016040 con 0x558c4da826d0 2024-03-24T09:28:09.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcd96ffd640 1 -- 172.21.15.116:0/3822023655 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcdb0004720 con 0x558c4da826d0 2024-03-24T09:28:09.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558c4d6df7e0 con 0x558c4da826d0 2024-03-24T09:28:09.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcd96ffd640 1 -- 172.21.15.116:0/3822023655 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcdb0004b20 con 0x558c4da826d0 2024-03-24T09:28:09.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdc0de6740 10 client.17319 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558c4d9deba0 con 0x558c4da826d0 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.078+0000 7fcd96ffd640 1 -- 172.21.15.116:0/3822023655 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x558c4d6df7e0 con 0x558c4da826d0 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcd96ffd640 1 -- 172.21.15.116:0/3822023655 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcdb0026030 con 0x558c4da826d0 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdc0de6740 10 client.17319 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdc0de6740 10 client.17319 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdc0de6740 20 client.17319 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4da92a50 0x558c4da94e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdc0de6740 4 client.17319 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c4d71c720 con 0x558c4da92a50 2024-03-24T09:28:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4da92a50 0x558c4da94e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4da92a50 0x558c4da94e40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=131283dc7cd44874 server_cookie=770afc0c0828c1d2 in_seq=0 out_seq=0 2024-03-24T09:28:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.079+0000 7fcd96ffd640 10 client.17319 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcd96ffd640 1 -- 172.21.15.116:0/3822023655 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c4d71c720 con 0x558c4da92a50 2024-03-24T09:28:09.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcd96ffd640 10 client.17319 handle_command_reply: tid=0 2024-03-24T09:28:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 client.17319 shutdown 2024-03-24T09:28:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 msgr2=0x558c4da80230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da80230 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0005fd0 tx=0x7fcdb001d000 comp rx=0 tx=0).stop 2024-03-24T09:28:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 shutdown_connections 2024-03-24T09:28:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da80230 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da79af0 0x558c4da7fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4da92a50 0x558c4da94e40 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/3822023655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da791d0 0x558c4da7f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcd96ffd640 0 client.17319 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 >> 172.21.15.116:0/3822023655 conn(0x558c4da73900 msgr2=0x558c4da72b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 shutdown_connections 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 1 -- 172.21.15.116:0/3822023655 wait complete. 2024-03-24T09:28:09.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.080+0000 7fcdc0de6740 20 client.17319 trim_cache size 0 max 16384 2024-03-24T09:28:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.097+0000 7fcdc0de6740 1 Processor -- start 2024-03-24T09:28:09.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 -- start start 2024-03-24T09:28:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da8dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 0x558c4da80050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da826d0 0x558c4da9a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4d965430 con 0x558c4da79af0 2024-03-24T09:28:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4da78970 con 0x558c4da826d0 2024-03-24T09:28:09.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558c4da78e30 con 0x558c4da791d0 2024-03-24T09:28:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 0x558c4da80050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da826d0 0x558c4da9a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 0x558c4da80050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37196/0 (socket says 172.21.15.116:37196) 2024-03-24T09:28:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/13953399 learned_addr learned my addr 172.21.15.116:0/13953399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da8dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/13953399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 msgr2=0x558c4da8dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da8dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/13953399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da826d0 msgr2=0x558c4da9a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da826d0 0x558c4da9a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 -- 172.21.15.116:0/13953399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c4d97ab20 con 0x558c4da79af0 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da8dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da826d0 0x558c4da9a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 0x558c4da80050 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0ee020 tx=0x7fcdac005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d34dc1addebcd31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcd9d7fa640 1 -- 172.21.15.116:0/13953399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0c1160 con 0x558c4da79af0 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcd9d7fa640 1 -- 172.21.15.116:0/13953399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac0bd910 con 0x558c4da79af0 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.098+0000 7fcd9d7fa640 1 -- 172.21.15.116:0/13953399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac0b9a10 con 0x558c4da79af0 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 -- 172.21.15.116:0/13953399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 msgr2=0x558c4da80050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 0x558c4da80050 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0ee020 tx=0x7fcdac005e80 comp rx=0 tx=0).stop 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 -- 172.21.15.116:0/13953399 shutdown_connections 2024-03-24T09:28:09.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da826d0 0x558c4da9a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da79af0 0x558c4da80050 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/13953399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da8dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 -- 172.21.15.116:0/13953399 >> 172.21.15.116:0/13953399 conn(0x558c4da73900 msgr2=0x558c4da72b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 -- 172.21.15.116:0/13953399 shutdown_connections 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 -- 172.21.15.116:0/13953399 wait complete. 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.099+0000 7fcdc0de6740 1 -- start start 2024-03-24T09:28:09.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 Processor -- start 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da85c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da861c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da86700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c4daf7d60 con 0x558c4da826d0 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558c4daf7ea0 con 0x558c4da79af0 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558c4daf7fe0 con 0x558c4da791d0 2024-03-24T09:28:09.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdc0de6740 10 client.0 _ll_register_callbacks cb 0x7fcdb0011560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da86700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da85c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da86700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37212/0 (socket says 172.21.15.116:37212) 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 -- 172.21.15.116:0/1379152586 learned_addr learned my addr 172.21.15.116:0/1379152586 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb6ffd640 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da861c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 -- 172.21.15.116:0/1379152586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 msgr2=0x558c4da85c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da85c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 -- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 msgr2=0x558c4da861c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da861c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 -- 172.21.15.116:0/1379152586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda8009df0 con 0x558c4da826d0 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcdb7fff640 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da86700 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000fc20 tx=0x7fcdb00039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0790f1d95f1dba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcd96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.100+0000 7fcd96ffd640 1 -- 172.21.15.116:0/1379152586 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0016070 con 0x558c4da826d0 2024-03-24T09:28:09.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcd96ffd640 1 -- 172.21.15.116:0/1379152586 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fcdb0011610 con 0x558c4da826d0 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdac091760 con 0x558c4da826d0 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcd96ffd640 1 -- 172.21.15.116:0/1379152586 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcdb00059c0 con 0x558c4da826d0 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 10 client.17325 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558c4d9deba0 con 0x558c4da826d0 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcd96ffd640 1 -- 172.21.15.116:0/1379152586 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcdac091760 con 0x558c4da826d0 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcd96ffd640 1 -- 172.21.15.116:0/1379152586 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcdb0026040 con 0x558c4da826d0 2024-03-24T09:28:09.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 10 client.17325 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 10 client.17325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 20 client.17325 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4daf9690 0x558c4dafbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 4 client.17325 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c4d76b890 con 0x558c4daf9690 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.101+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4daf9690 0x558c4dafbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcd96ffd640 10 client.17325 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdb77fe640 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4daf9690 0x558c4dafbab0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a97f79a33a36828 server_cookie=25f4efa34f6e5859 in_seq=0 out_seq=0 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcd96ffd640 1 -- 172.21.15.116:0/1379152586 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c4d76b890 con 0x558c4daf9690 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcd96ffd640 10 client.17325 handle_command_reply: tid=0 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 client.17325 shutdown 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 msgr2=0x558c4da86700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da86700 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000fc20 tx=0x7fcdb00039f0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 shutdown_connections 2024-03-24T09:28:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c4da826d0 0x558c4da86700 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558c4daf9690 0x558c4dafbab0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558c4da79af0 0x558c4da861c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 --2- 172.21.15.116:0/1379152586 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558c4da791d0 0x558c4da85c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcd96ffd640 0 client.17325 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:09.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 >> 172.21.15.116:0/1379152586 conn(0x558c4da73900 msgr2=0x558c4da74a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 shutdown_connections 2024-03-24T09:28:09.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 1 -- 172.21.15.116:0/1379152586 wait complete. 2024-03-24T09:28:09.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdc0de6740 20 client.17325 trim_cache size 0 max 16384 2024-03-24T09:28:09.122 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:28:09.123 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:28:09.123 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:28:09.123 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:28:09.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcda00a2230 msgr2=0x7fcda00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdbe125640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcda00a2230 0x7fcda00a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0009fd0 tx=0x7fcdb0009f40 comp rx=0 tx=0).stop 2024-03-24T09:28:09.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 msgr2=0x7fcdb819a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.102+0000 7fcdbe125640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb819a370 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fcdac006010 tx=0x7fcdac005e10 comp rx=0 tx=0).stop 2024-03-24T09:28:09.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 shutdown_connections 2024-03-24T09:28:09.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdb810df60 0x7fcdb819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdb80743e0 0x7fcdb819a370 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcda00a2230 0x7fcda00a46f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 --2- 172.21.15.116:0/2070044078 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcdb8073ac0 0x7fcdb8199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 >> 172.21.15.116:0/2070044078 conn(0x7fcdb8078b10 msgr2=0x7fcdb80ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 shutdown_connections 2024-03-24T09:28:09.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.103+0000 7fcdbe125640 1 -- 172.21.15.116:0/2070044078 wait complete. 2024-03-24T09:28:09.127 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:28:09.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:28:09.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 -- 172.21.15.116:0/3728893587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0102940 msgr2=0x7f2be0102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 --2- 172.21.15.116:0/3728893587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0102940 0x7f2be0102d20 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009900 tx=0x7f2bd002bcd0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 -- 172.21.15.116:0/3728893587 shutdown_connections 2024-03-24T09:28:09.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 --2- 172.21.15.116:0/3728893587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2be010bd10 0x7f2be010e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 --2- 172.21.15.116:0/3728893587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be0103260 0x7f2be010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 --2- 172.21.15.116:0/3728893587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0102940 0x7f2be0102d20 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 -- 172.21.15.116:0/3728893587 >> 172.21.15.116:0/3728893587 conn(0x7f2be0076990 msgr2=0x7f2be0076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 -- 172.21.15.116:0/3728893587 shutdown_connections 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.228+0000 7f2be49c2640 1 -- 172.21.15.116:0/3728893587 wait complete. 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 Processor -- start 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 -- start start 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2be0102940 0x7f2be019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 0x7f2be019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be010bd10 0x7f2be0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2be0198120 con 0x7f2be0103260 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2be0198290 con 0x7f2be0102940 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2be49c2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2be0198450 con 0x7f2be010bd10 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2bddd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 0x7f2be019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2bddd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 0x7f2be019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37226/0 (socket says 172.21.15.116:37226) 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2bddd9b640 1 -- 172.21.15.116:0/4084617694 learned_addr learned my addr 172.21.15.116:0/4084617694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2bde59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2be0102940 0x7f2be019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.229+0000 7f2bded9d640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be010bd10 0x7f2be0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bddd9b640 1 -- 172.21.15.116:0/4084617694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be010bd10 msgr2=0x7f2be0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bddd9b640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be010bd10 0x7f2be0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bddd9b640 1 -- 172.21.15.116:0/4084617694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2be0102940 msgr2=0x7f2be019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bddd9b640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2be0102940 0x7f2be019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bddd9b640 1 -- 172.21.15.116:0/4084617694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd0009560 con 0x7f2be0103260 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bddd9b640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 0x7f2be019de60 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400e9f0 tx=0x7f2bd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4009720 con 0x7f2be0103260 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2bd400cd80 con 0x7f2be0103260 2024-03-24T09:28:09.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be0198730 con 0x7f2be0103260 2024-03-24T09:28:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4010430 con 0x7f2be0103260 2024-03-24T09:28:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.230+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be01a49c0 con 0x7f2be0103260 2024-03-24T09:28:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.231+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be006a560 con 0x7f2be0103260 2024-03-24T09:28:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.232+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2bd4005020 con 0x7f2be0103260 2024-03-24T09:28:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.232+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2bc80a2230 0x7f2bc80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.232+0000 7f2bde59c640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2bc80a2230 0x7f2bc80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.233+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2bd4014070 con 0x7f2be0103260 2024-03-24T09:28:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.233+0000 7f2bde59c640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2bc80a2230 0x7f2bc80a46f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0006010 tx=0x7f2bd0005df0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.237+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2bd4097050 con 0x7f2be0103260 2024-03-24T09:28:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:09 smithi117 ceph-mon[35788]: pgmap v184: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:09 smithi116 ceph-mon[32207]: pgmap v184: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.448+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2be0198fc0 con 0x7f2be0103260 2024-03-24T09:28:09.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.449+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/4084617694 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f2bd409a710 con 0x7f2be0103260 2024-03-24T09:28:09.450 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:28:09.450 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:28:09.451 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:28:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2bc80a2230 msgr2=0x7f2bc80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2bc80a2230 0x7f2bc80a46f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0006010 tx=0x7f2bd0005df0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 msgr2=0x7f2be019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 0x7f2be019de60 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400e9f0 tx=0x7f2bd400edf0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 shutdown_connections 2024-03-24T09:28:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2bc80a2230 0x7f2bc80a46f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2be010bd10 0x7f2be0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2be0103260 0x7f2be019de60 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 --2- 172.21.15.116:0/4084617694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2be0102940 0x7f2be019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 >> 172.21.15.116:0/4084617694 conn(0x7f2be0076990 msgr2=0x7f2be00fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 shutdown_connections 2024-03-24T09:28:09.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.451+0000 7f2be49c2640 1 -- 172.21.15.116:0/4084617694 wait complete. 2024-03-24T09:28:09.463 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:28:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:09 smithi143 ceph-mon[35965]: pgmap v184: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:09.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 -- 172.21.15.116:0/192010558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780106660 msgr2=0x7fb780106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 --2- 172.21.15.116:0/192010558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780106660 0x7fb780106a40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009930 tx=0x7fb77c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 -- 172.21.15.116:0/192010558 shutdown_connections 2024-03-24T09:28:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 --2- 172.21.15.116:0/192010558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780100f80 0x7fb78010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 --2- 172.21.15.116:0/192010558 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7801005c0 0x7fb780100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 --2- 172.21.15.116:0/192010558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780106660 0x7fb780106a40 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 -- 172.21.15.116:0/192010558 >> 172.21.15.116:0/192010558 conn(0x7fb780076990 msgr2=0x7fb780076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 -- 172.21.15.116:0/192010558 shutdown_connections 2024-03-24T09:28:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.566+0000 7fb788255640 1 -- 172.21.15.116:0/192010558 wait complete. 2024-03-24T09:28:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 Processor -- start 2024-03-24T09:28:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 -- start start 2024-03-24T09:28:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7801005c0 0x7fb78019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 0x7fb78019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780106660 0x7fb7801968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb780196f30 con 0x7fb780100f80 2024-03-24T09:28:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7801970a0 con 0x7fb780106660 2024-03-24T09:28:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb788255640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb780197260 con 0x7fb7801005c0 2024-03-24T09:28:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 0x7fb78019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 0x7fb78019cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37250/0 (socket says 172.21.15.116:37250) 2024-03-24T09:28:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 -- 172.21.15.116:0/1545262512 learned_addr learned my addr 172.21.15.116:0/1545262512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb785ff1640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7801005c0 0x7fb78019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7867f2640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780106660 0x7fb7801968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 -- 172.21.15.116:0/1545262512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7801005c0 msgr2=0x7fb78019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7801005c0 0x7fb78019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 -- 172.21.15.116:0/1545262512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780106660 msgr2=0x7fb7801968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780106660 0x7fb7801968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.567+0000 7fb7857f0640 1 -- 172.21.15.116:0/1545262512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c009560 con 0x7fb780100f80 2024-03-24T09:28:09.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.568+0000 7fb7857f0640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 0x7fb78019cc70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fb77000e9f0 tx=0x7fb77000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.568+0000 7fb776ffd640 1 -- 172.21.15.116:0/1545262512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770009700 con 0x7fb780100f80 2024-03-24T09:28:09.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.568+0000 7fb776ffd640 1 -- 172.21.15.116:0/1545262512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb77000cd80 con 0x7fb780100f80 2024-03-24T09:28:09.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.568+0000 7fb776ffd640 1 -- 172.21.15.116:0/1545262512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770018840 con 0x7fb780100f80 2024-03-24T09:28:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.568+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb780197540 con 0x7fb780100f80 2024-03-24T09:28:09.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.568+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7801a37d0 con 0x7fb780100f80 2024-03-24T09:28:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.570+0000 7fb776ffd640 1 -- 172.21.15.116:0/1545262512 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fb7700189a0 con 0x7fb780100f80 2024-03-24T09:28:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 Processor -- start 2024-03-24T09:28:09.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 -- start start 2024-03-24T09:28:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 0x55e42083d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083dd10 0x55e4208463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e420848ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e4207ab840 con 0x55e4208468f0 2024-03-24T09:28:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e4207a8d70 con 0x55e42083dd10 2024-03-24T09:28:09.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e4207a4550 con 0x55e42083d3f0 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb7857f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e420848ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb776ffd640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb76c0a2230 0x7fb76c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb7867f2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 0x55e42083d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb7867f2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 0x55e42083d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45228/0 (socket says 172.21.15.116:45228) 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb7857f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e420848ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37256/0 (socket says 172.21.15.116:37256) 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb7867f2640 1 -- 172.21.15.116:0/4107503094 learned_addr learned my addr 172.21.15.116:0/4107503094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb785ff1640 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083dd10 0x55e4208463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb7867f2640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb76c0a2230 0x7fb76c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb785ff1640 1 -- 172.21.15.116:0/4107503094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 msgr2=0x55e42083d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb785ff1640 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 0x55e42083d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb785ff1640 1 -- 172.21.15.116:0/4107503094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 msgr2=0x55e420848ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.571+0000 7fb785ff1640 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e420848ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb785ff1640 1 -- 172.21.15.116:0/4107503094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e420739de0 con 0x55e42083dd10 2024-03-24T09:28:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb7867f2640 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 0x55e42083d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb776ffd640 1 -- 172.21.15.116:0/1545262512 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb770014070 con 0x7fb780100f80 2024-03-24T09:28:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb785ff1640 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083dd10 0x55e4208463b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009230 tx=0x7fb77c005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=578044d06e4bf61f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb7867f2640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb76c0a2230 0x7fb76c0a46f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb778009fd0 tx=0x7fb778004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb76bfff640 1 -- 172.21.15.116:0/4107503094 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c004880 con 0x55e42083dd10 2024-03-24T09:28:09.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb76bfff640 1 -- 172.21.15.116:0/4107503094 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb77c02cc20 con 0x55e42083dd10 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.572+0000 7fb76bfff640 1 -- 172.21.15.116:0/4107503094 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c003850 con 0x55e42083dd10 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 -- 172.21.15.116:0/4107503094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083dd10 msgr2=0x55e4208463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083dd10 0x55e4208463b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb77c009230 tx=0x7fb77c005dd0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 -- 172.21.15.116:0/4107503094 shutdown_connections 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e420848ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083dd10 0x55e4208463b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/4107503094 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083d3f0 0x55e42083d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 -- 172.21.15.116:0/4107503094 >> 172.21.15.116:0/4107503094 conn(0x55e420837b20 msgr2=0x55e420837f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 -- 172.21.15.116:0/4107503094 shutdown_connections 2024-03-24T09:28:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 -- 172.21.15.116:0/4107503094 wait complete. 2024-03-24T09:28:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 -- start start 2024-03-24T09:28:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.573+0000 7fb78ae3b740 1 Processor -- start 2024-03-24T09:28:09.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420843930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420843e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e4208443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e420477430 con 0x55e42083d3f0 2024-03-24T09:28:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e420433e00 con 0x55e4208468f0 2024-03-24T09:28:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e420729590 con 0x55e42083dd10 2024-03-24T09:28:09.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 10 client.0 _ll_register_callbacks cb 0x55e420798fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420843930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420843930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37262/0 (socket says 172.21.15.116:37262) 2024-03-24T09:28:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 -- 172.21.15.116:0/3122475413 learned_addr learned my addr 172.21.15.116:0/3122475413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7867f2640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e4208443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7857f0640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420843e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 -- 172.21.15.116:0/3122475413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 msgr2=0x55e420843e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420843e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 -- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 msgr2=0x55e4208443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e4208443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 -- 172.21.15.116:0/3122475413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb78010dd90 con 0x55e42083d3f0 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7867f2640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e4208443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb785ff1640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420843930 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb77c005c40 tx=0x7fb77c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce8c40fd79f38b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7697fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7697fa640 1 -- 172.21.15.116:0/3122475413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0051d0 con 0x55e42083d3f0 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7697fa640 1 -- 172.21.15.116:0/3122475413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb77c02de40 con 0x55e42083d3f0 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e4204a37e0 con 0x55e42083d3f0 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb7697fa640 1 -- 172.21.15.116:0/3122475413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb77c005330 con 0x55e42083d3f0 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 10 client.17343 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.574+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e4207a2bd0 con 0x55e42083d3f0 2024-03-24T09:28:09.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb7697fa640 1 -- 172.21.15.116:0/3122475413 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e4204a37e0 con 0x55e42083d3f0 2024-03-24T09:28:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb7697fa640 1 -- 172.21.15.116:0/3122475413 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb77c03c050 con 0x55e42083d3f0 2024-03-24T09:28:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb78ae3b740 10 client.17343 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb78ae3b740 10 client.17343 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:28:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb78ae3b740 20 client.17343 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e420856570 0x55e420858990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb78ae3b740 4 client.17343 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4204e0720 con 0x55e420856570 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb7857f0640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e420856570 0x55e420858990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb7697fa640 10 client.17343 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.575+0000 7fb7857f0640 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e420856570 0x55e420858990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c77b6b225182baef server_cookie=3cf6e2dc75752b12 in_seq=0 out_seq=0 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb7697fa640 1 -- 172.21.15.116:0/3122475413 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e4204e0720 con 0x55e420856570 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb7697fa640 10 client.17343 handle_command_reply: tid=0 2024-03-24T09:28:09.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 client.17343 shutdown 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 msgr2=0x55e420843930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420843930 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb77c005c40 tx=0x7fb77c009260 comp rx=0 tx=0).stop 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 shutdown_connections 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e4208443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420843e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e420856570 0x55e420858990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3122475413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420843930 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb7697fa640 0 client.17343 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:09.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 >> 172.21.15.116:0/3122475413 conn(0x55e420837b20 msgr2=0x55e420839870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 shutdown_connections 2024-03-24T09:28:09.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3122475413 wait complete. 2024-03-24T09:28:09.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.576+0000 7fb78ae3b740 20 client.17343 trim_cache size 0 max 16384 2024-03-24T09:28:09.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 Processor -- start 2024-03-24T09:28:09.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 -- start start 2024-03-24T09:28:09.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420852120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420836b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e42085e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e4204a37e0 con 0x55e42083d3f0 2024-03-24T09:28:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e420844900 con 0x55e4208468f0 2024-03-24T09:28:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e420853410 con 0x55e42083dd10 2024-03-24T09:28:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420852120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420852120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37274/0 (socket says 172.21.15.116:37274) 2024-03-24T09:28:09.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 -- 172.21.15.116:0/1948755375 learned_addr learned my addr 172.21.15.116:0/1948755375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb7857f0640 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420836b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb7867f2640 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e42085e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 -- 172.21.15.116:0/1948755375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 msgr2=0x55e420836b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420836b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 -- 172.21.15.116:0/1948755375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 msgr2=0x55e42085e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e42085e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 -- 172.21.15.116:0/1948755375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e42073ec80 con 0x55e42083d3f0 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb785ff1640 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420852120 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb77c005890 tx=0x7fb77c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62140db09d08e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb76b7fe640 1 -- 172.21.15.116:0/1948755375 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c056940 con 0x55e42083d3f0 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb76b7fe640 1 -- 172.21.15.116:0/1948755375 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb77c0560d0 con 0x55e42083d3f0 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.590+0000 7fb76b7fe640 1 -- 172.21.15.116:0/1948755375 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c005210 con 0x55e42083d3f0 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 -- 172.21.15.116:0/1948755375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 msgr2=0x55e420852120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420852120 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fb77c005890 tx=0x7fb77c000bc0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 -- 172.21.15.116:0/1948755375 shutdown_connections 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e4208468f0 0x55e42085e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e420836b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/1948755375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e42083d3f0 0x55e420852120 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 -- 172.21.15.116:0/1948755375 >> 172.21.15.116:0/1948755375 conn(0x55e420837b20 msgr2=0x55e420836430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 -- 172.21.15.116:0/1948755375 shutdown_connections 2024-03-24T09:28:09.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 -- 172.21.15.116:0/1948755375 wait complete. 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 -- start start 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.591+0000 7fb78ae3b740 1 Processor -- start 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083d3f0 0x55e42084a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e42084a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e42084ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e4208bb8d0 con 0x55e4208468f0 2024-03-24T09:28:09.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e4208bba10 con 0x55e42083d3f0 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e4208bbb80 con 0x55e42083dd10 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb78ae3b740 10 client.0 _ll_register_callbacks cb 0x7fb77009ac90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e42084ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb785ff1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083d3f0 0x55e42084a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e42084ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37284/0 (socket says 172.21.15.116:37284) 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 -- 172.21.15.116:0/3638952399 learned_addr learned my addr 172.21.15.116:0/3638952399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7857f0640 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e42084a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 -- 172.21.15.116:0/3638952399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 msgr2=0x55e42084a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e42084a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 -- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083d3f0 msgr2=0x55e42084a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083d3f0 0x55e42084a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 -- 172.21.15.116:0/3638952399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4207a2bd0 con 0x55e4208468f0 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7867f2640 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e42084ab80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb7780047e0 tx=0x7fb778004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8012e60c3130f262 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.592+0000 7fb7697fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb7697fa640 1 -- 172.21.15.116:0/3638952399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb778027070 con 0x55e4208468f0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb7697fa640 1 -- 172.21.15.116:0/3638952399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fb778022440 con 0x55e4208468f0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb77c02cc20 con 0x55e4208468f0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb7697fa640 1 -- 172.21.15.116:0/3638952399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb778022600 con 0x55e4208468f0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 10 client.17349 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e420764920 con 0x55e4208468f0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb7697fa640 1 -- 172.21.15.116:0/3638952399 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb77c02cc20 con 0x55e4208468f0 2024-03-24T09:28:09.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb7697fa640 1 -- 172.21.15.116:0/3638952399 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fb7780216e0 con 0x55e4208468f0 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 10 client.17349 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 10 client.17349 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 20 client.17349 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e4208bd4e0 0x55e4208bf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 4 client.17349 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e42052f890 con 0x55e4208bd4e0 2024-03-24T09:28:09.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb785ff1640 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e4208bd4e0 0x55e4208bf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb7697fa640 10 client.17349 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.593+0000 7fb785ff1640 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e4208bd4e0 0x55e4208bf900 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e0e07575dd00cd44 server_cookie=781a452fa9015351 in_seq=0 out_seq=0 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb7697fa640 1 -- 172.21.15.116:0/3638952399 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e42052f890 con 0x55e4208bd4e0 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb7697fa640 10 client.17349 handle_command_reply: tid=0 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 client.17349 shutdown 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 msgr2=0x55e42084ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e42084ab80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb7780047e0 tx=0x7fb778004a40 comp rx=0 tx=0).stop 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 shutdown_connections 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55e4208bd4e0 0x55e4208bf900 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e4208468f0 0x55e42084ab80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e42083dd10 0x55e42084a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 --2- 172.21.15.116:0/3638952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e42083d3f0 0x55e42084a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb7697fa640 0 client.17349 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:09.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 >> 172.21.15.116:0/3638952399 conn(0x55e420837b20 msgr2=0x55e420836cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 shutdown_connections 2024-03-24T09:28:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 1 -- 172.21.15.116:0/3638952399 wait complete. 2024-03-24T09:28:09.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb78ae3b740 20 client.17349 trim_cache size 0 max 16384 2024-03-24T09:28:09.616 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb76c0a2230 msgr2=0x7fb76c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb76c0a2230 0x7fb76c0a46f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb778009fd0 tx=0x7fb778004ab0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 msgr2=0x7fb78019cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 0x7fb78019cc70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fb77000e9f0 tx=0x7fb77000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 shutdown_connections 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb780106660 0x7fb7801968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb780100f80 0x7fb78019cc70 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb76c0a2230 0x7fb76c0a46f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 --2- 172.21.15.116:0/1545262512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb7801005c0 0x7fb78019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.594+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 >> 172.21.15.116:0/1545262512 conn(0x7fb780076990 msgr2=0x7fb7800fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.595+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 shutdown_connections 2024-03-24T09:28:09.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.595+0000 7fb788255640 1 -- 172.21.15.116:0/1545262512 wait complete. 2024-03-24T09:28:09.620 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:28:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 -- 172.21.15.116:0/1017761535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22400fffd0 msgr2=0x7f2240100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 --2- 172.21.15.116:0/1017761535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22400fffd0 0x7f2240100450 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f22280098c0 tx=0x7f222802bbe0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 -- 172.21.15.116:0/1017761535 shutdown_connections 2024-03-24T09:28:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 --2- 172.21.15.116:0/1017761535 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2240100990 0x7f224010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 --2- 172.21.15.116:0/1017761535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22400fffd0 0x7f2240100450 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 --2- 172.21.15.116:0/1017761535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2240107910 0x7f2240107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 -- 172.21.15.116:0/1017761535 >> 172.21.15.116:0/1017761535 conn(0x7f2240075950 msgr2=0x7f2240075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 -- 172.21.15.116:0/1017761535 shutdown_connections 2024-03-24T09:28:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.722+0000 7f2244d47640 1 -- 172.21.15.116:0/1017761535 wait complete. 2024-03-24T09:28:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 Processor -- start 2024-03-24T09:28:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 -- start start 2024-03-24T09:28:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 0x7f2240197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2240100990 0x7f22401982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2240107910 0x7f224019c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f224019cc20 con 0x7f22400fffd0 2024-03-24T09:28:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f224019cd90 con 0x7f2240100990 2024-03-24T09:28:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2244d47640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f224019cf50 con 0x7f2240107910 2024-03-24T09:28:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 0x7f2240197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 0x7f2240197d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37302/0 (socket says 172.21.15.116:37302) 2024-03-24T09:28:09.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f223e59c640 1 -- 172.21.15.116:0/4053541718 learned_addr learned my addr 172.21.15.116:0/4053541718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f2237fff640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2240100990 0x7f22401982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.723+0000 7f223ed9d640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2240107910 0x7f224019c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f223e59c640 1 -- 172.21.15.116:0/4053541718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2240107910 msgr2=0x7f224019c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f223e59c640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2240107910 0x7f224019c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f223e59c640 1 -- 172.21.15.116:0/4053541718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2240100990 msgr2=0x7f22401982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f223e59c640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2240100990 0x7f22401982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f223e59c640 1 -- 172.21.15.116:0/4053541718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2228009490 con 0x7f22400fffd0 2024-03-24T09:28:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f2237fff640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2240100990 0x7f22401982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f223e59c640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 0x7f2240197d60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f223000e9c0 tx=0x7f223000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f22377fe640 1 -- 172.21.15.116:0/4053541718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223000ccc0 con 0x7f22400fffd0 2024-03-24T09:28:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f224019d230 con 0x7f22400fffd0 2024-03-24T09:28:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f22377fe640 1 -- 172.21.15.116:0/4053541718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f223000ce20 con 0x7f22400fffd0 2024-03-24T09:28:09.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f22377fe640 1 -- 172.21.15.116:0/4053541718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2230010430 con 0x7f22400fffd0 2024-03-24T09:28:09.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.724+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22401a4ab0 con 0x7f22400fffd0 2024-03-24T09:28:09.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.726+0000 7f22377fe640 1 -- 172.21.15.116:0/4053541718 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2230004ee0 con 0x7f22400fffd0 2024-03-24T09:28:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.726+0000 7f22479fd740 1 Processor -- start 2024-03-24T09:28:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.726+0000 7f22479fd740 1 -- start start 2024-03-24T09:28:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22479fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b33f0 0x55e2051b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22479fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b3d10 0x55e2051bc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22479fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051bece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22479fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e205121840 con 0x55e2051b3d10 2024-03-24T09:28:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22479fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e20511ed70 con 0x55e2051bc8f0 2024-03-24T09:28:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22479fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e20511a550 con 0x55e2051b33f0 2024-03-24T09:28:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b3d10 0x55e2051bc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22377fe640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f222c0a2230 0x7f222c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b3d10 0x55e2051bc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37314/0 (socket says 172.21.15.116:37314) 2024-03-24T09:28:09.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f223e59c640 1 -- 172.21.15.116:0/3440466065 learned_addr learned my addr 172.21.15.116:0/3440466065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051bece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f223ed9d640 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b33f0 0x55e2051b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f223ed9d640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f222c0a2230 0x7f222c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f22377fe640 1 -- 172.21.15.116:0/4053541718 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2230014070 con 0x7f22400fffd0 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 -- 172.21.15.116:0/3440466065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b33f0 msgr2=0x55e2051b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b33f0 0x55e2051b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 -- 172.21.15.116:0/3440466065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b3d10 msgr2=0x55e2051bc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b3d10 0x55e2051bc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 -- 172.21.15.116:0/3440466065 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2050afde0 con 0x55e2051bc8f0 2024-03-24T09:28:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2237fff640 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051bece0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2228005940 tx=0x7f22280058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7024ba3335e7a704 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f223ed9d640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f222c0a2230 0x7f222c0a46f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2238005c40 tx=0x7f2238009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2234ff9640 1 -- 172.21.15.116:0/3440466065 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228004400 con 0x55e2051bc8f0 2024-03-24T09:28:09.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2234ff9640 1 -- 172.21.15.116:0/3440466065 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22280348f0 con 0x55e2051bc8f0 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.727+0000 7f2234ff9640 1 -- 172.21.15.116:0/3440466065 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22280035f0 con 0x55e2051bc8f0 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 -- 172.21.15.116:0/3440466065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 msgr2=0x55e2051bece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051bece0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2228005940 tx=0x7f22280058b0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 -- 172.21.15.116:0/3440466065 shutdown_connections 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051bece0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b3d10 0x55e2051bc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 --2- 172.21.15.116:0/3440466065 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b33f0 0x55e2051b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 -- 172.21.15.116:0/3440466065 >> 172.21.15.116:0/3440466065 conn(0x55e2051adb20 msgr2=0x55e2051adf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 -- 172.21.15.116:0/3440466065 shutdown_connections 2024-03-24T09:28:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.728+0000 7f22479fd740 1 -- 172.21.15.116:0/3440466065 wait complete. 2024-03-24T09:28:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 -- start start 2024-03-24T09:28:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 Processor -- start 2024-03-24T09:28:09.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b33f0 0x55e2051b98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051b9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 0x55e2051ba350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e204ded430 con 0x55e2051bc8f0 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e204da9e00 con 0x55e2051b33f0 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e20509f590 con 0x55e2051b3d10 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f22479fd740 10 client.0 _ll_register_callbacks cb 0x55e20510efc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 0x55e2051ba350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 0x55e2051ba350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37322/0 (socket says 172.21.15.116:37322) 2024-03-24T09:28:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 -- 172.21.15.116:0/550112033 learned_addr learned my addr 172.21.15.116:0/550112033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223e59c640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b33f0 0x55e2051b98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f2237fff640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051b9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 -- 172.21.15.116:0/550112033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 msgr2=0x55e2051b9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051b9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 -- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b33f0 msgr2=0x55e2051b98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b33f0 0x55e2051b98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 -- 172.21.15.116:0/550112033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f224010ee90 con 0x55e2051bc8f0 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f2237fff640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051b9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223e59c640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b33f0 0x55e2051b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f223ed9d640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 0x55e2051ba350 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f223800fb30 tx=0x7f2238004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231fd9067343750f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.729+0000 7f2225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f2225ffb640 1 -- 172.21.15.116:0/550112033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2238011020 con 0x55e2051bc8f0 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f2225ffb640 1 -- 172.21.15.116:0/550112033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f2238011180 con 0x55e2051bc8f0 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e204e197e0 con 0x55e2051bc8f0 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f2225ffb640 1 -- 172.21.15.116:0/550112033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2238004260 con 0x55e2051bc8f0 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 10 client.17361 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e205118bd0 con 0x55e2051bc8f0 2024-03-24T09:28:09.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f2225ffb640 1 -- 172.21.15.116:0/550112033 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e204e197e0 con 0x55e2051bc8f0 2024-03-24T09:28:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f2225ffb640 1 -- 172.21.15.116:0/550112033 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f223801eb20 con 0x55e2051bc8f0 2024-03-24T09:28:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 10 client.17361 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 10 client.17361 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:28:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 20 client.17361 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e2051cca40 0x55e2051cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 4 client.17361 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e204e56720 con 0x55e2051cca40 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.730+0000 7f223e59c640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e2051cca40 0x55e2051cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f2225ffb640 10 client.17361 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f223e59c640 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e2051cca40 0x55e2051cee60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b03fdb1b9f976c2 server_cookie=e2d3ca7971d82cc7 in_seq=0 out_seq=0 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f2225ffb640 1 -- 172.21.15.116:0/550112033 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e204e56720 con 0x55e2051cca40 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f2225ffb640 10 client.17361 handle_command_reply: tid=0 2024-03-24T09:28:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 client.17361 shutdown 2024-03-24T09:28:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 msgr2=0x55e2051ba350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 0x55e2051ba350 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f223800fb30 tx=0x7f2238004a40 comp rx=0 tx=0).stop 2024-03-24T09:28:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 shutdown_connections 2024-03-24T09:28:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e2051cca40 0x55e2051cee60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051bc8f0 0x55e2051ba350 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051b9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f2225ffb640 0 client.17361 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 --2- 172.21.15.116:0/550112033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b33f0 0x55e2051b98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 >> 172.21.15.116:0/550112033 conn(0x55e2051adb20 msgr2=0x55e2051af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 shutdown_connections 2024-03-24T09:28:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.731+0000 7f22479fd740 1 -- 172.21.15.116:0/550112033 wait complete. 2024-03-24T09:28:09.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.732+0000 7f22479fd740 20 client.17361 trim_cache size 0 max 16384 2024-03-24T09:28:09.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 Processor -- start 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 -- start start 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051c8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051bb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051d4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e204e19810 con 0x55e2051b33f0 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e2051cb1a0 con 0x55e2051bc8f0 2024-03-24T09:28:09.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f22479fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e2051aeb20 con 0x55e2051b3d10 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37334/0 (socket says 172.21.15.116:37334) 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 -- 172.21.15.116:0/3541399177 learned_addr learned my addr 172.21.15.116:0/3541399177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f2237fff640 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051bb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223ed9d640 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 -- 172.21.15.116:0/3541399177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 msgr2=0x55e2051bb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051bb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 -- 172.21.15.116:0/3541399177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 msgr2=0x55e2051d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051d4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 -- 172.21.15.116:0/3541399177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2050b4c80 con 0x55e2051b33f0 2024-03-24T09:28:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f223e59c640 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051c8120 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2230097610 tx=0x7f22300973e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7de9dfdf13edbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f2227fff640 1 -- 172.21.15.116:0/3541399177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223009b340 con 0x55e2051b33f0 2024-03-24T09:28:09.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f2227fff640 1 -- 172.21.15.116:0/3541399177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f223009ac40 con 0x55e2051b33f0 2024-03-24T09:28:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.745+0000 7f2227fff640 1 -- 172.21.15.116:0/3541399177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22300a2f50 con 0x55e2051b33f0 2024-03-24T09:28:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 -- 172.21.15.116:0/3541399177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 msgr2=0x55e2051c8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051c8120 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f2230097610 tx=0x7f22300973e0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 -- 172.21.15.116:0/3541399177 shutdown_connections 2024-03-24T09:28:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051bc8f0 0x55e2051d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051b3d10 0x55e2051bb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 --2- 172.21.15.116:0/3541399177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051c8120 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 -- 172.21.15.116:0/3541399177 >> 172.21.15.116:0/3541399177 conn(0x55e2051adb20 msgr2=0x55e2051baff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 -- 172.21.15.116:0/3541399177 shutdown_connections 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 -- 172.21.15.116:0/3541399177 wait complete. 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 -- start start 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.746+0000 7f22479fd740 1 Processor -- start 2024-03-24T09:28:09.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b3d10 0x55e2051b7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051bc8f0 0x55e205231da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e2052322e0 con 0x55e2051b33f0 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e205232420 con 0x55e2051b3d10 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e205232590 con 0x55e2051bc8f0 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 10 client.0 _ll_register_callbacks cb 0x7f2238009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051b6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37348/0 (socket says 172.21.15.116:37348) 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 -- 172.21.15.116:0/1293337667 learned_addr learned my addr 172.21.15.116:0/1293337667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f2237fff640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b3d10 0x55e2051b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223ed9d640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051bc8f0 0x55e205231da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 -- 172.21.15.116:0/1293337667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051bc8f0 msgr2=0x55e205231da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051bc8f0 0x55e205231da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 -- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b3d10 msgr2=0x55e2051b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b3d10 0x55e2051b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 -- 172.21.15.116:0/1293337667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22300ce040 con 0x55e2051b33f0 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f223e59c640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051b6c40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f22300058c0 tx=0x7f223009a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9fbc03f9629e369 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f2225ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f2225ffb640 1 -- 172.21.15.116:0/1293337667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223007d070 con 0x55e2051b33f0 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f2225ffb640 1 -- 172.21.15.116:0/1293337667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f223009ed90 con 0x55e2051b33f0 2024-03-24T09:28:09.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22280348f0 con 0x55e2051b33f0 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f2225ffb640 1 -- 172.21.15.116:0/1293337667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f22300046c0 con 0x55e2051b33f0 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 10 client.17367 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.747+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e2050da920 con 0x55e2051b33f0 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2225ffb640 1 -- 172.21.15.116:0/1293337667 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22280348f0 con 0x55e2051b33f0 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2225ffb640 1 -- 172.21.15.116:0/1293337667 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2230080030 con 0x55e2051b33f0 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 10 client.17367 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 10 client.17367 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:28:09.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 20 client.17367 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e205233860 0x55e205235c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 4 client.17367 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e204ea5890 con 0x55e205233860 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2237fff640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e205233860 0x55e205235c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2225ffb640 10 client.17367 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2237fff640 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e205233860 0x55e205235c50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=542aad2c2af4efa6 server_cookie=6d6578300bd70655 in_seq=0 out_seq=0 2024-03-24T09:28:09.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2225ffb640 1 -- 172.21.15.116:0/1293337667 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e204ea5890 con 0x55e205233860 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f2225ffb640 10 client.17367 handle_command_reply: tid=0 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 1 client.17367 shutdown 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 msgr2=0x55e2051b6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.748+0000 7f22479fd740 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051b6c40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f22300058c0 tx=0x7f223009a660 comp rx=0 tx=0).stop 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 shutdown_connections 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e205233860 0x55e205235c50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e2051bc8f0 0x55e205231da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e2051b3d10 0x55e2051b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 --2- 172.21.15.116:0/1293337667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e2051b33f0 0x55e2051b6c40 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2225ffb640 0 client.17367 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 >> 172.21.15.116:0/1293337667 conn(0x55e2051adb20 msgr2=0x55e2051accc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 shutdown_connections 2024-03-24T09:28:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 1 -- 172.21.15.116:0/1293337667 wait complete. 2024-03-24T09:28:09.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f22479fd740 20 client.17367 trim_cache size 0 max 16384 2024-03-24T09:28:09.771 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f222c0a2230 msgr2=0x7f222c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f222c0a2230 0x7f222c0a46f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2238005c40 tx=0x7f2238009500 comp rx=0 tx=0).stop 2024-03-24T09:28:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 msgr2=0x7f2240197d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 0x7f2240197d60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f223000e9c0 tx=0x7f223000ee80 comp rx=0 tx=0).stop 2024-03-24T09:28:09.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 shutdown_connections 2024-03-24T09:28:09.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f222c0a2230 0x7f222c0a46f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2240107910 0x7f224019c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2240100990 0x7f22401982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 --2- 172.21.15.116:0/4053541718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22400fffd0 0x7f2240197d60 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 >> 172.21.15.116:0/4053541718 conn(0x7f2240075950 msgr2=0x7f22400fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 shutdown_connections 2024-03-24T09:28:09.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.749+0000 7f2244d47640 1 -- 172.21.15.116:0/4053541718 wait complete. 2024-03-24T09:28:09.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:28:09.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 -- 172.21.15.116:0/474479636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100630 msgr2=0x7f3888100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 --2- 172.21.15.116:0/474479636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100630 0x7f3888100ab0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f387c009890 tx=0x7f387c02bb70 comp rx=0 tx=0).stop 2024-03-24T09:28:09.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 -- 172.21.15.116:0/474479636 shutdown_connections 2024-03-24T09:28:09.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 --2- 172.21.15.116:0/474479636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888100ff0 0x7f388810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 --2- 172.21.15.116:0/474479636 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100630 0x7f3888100ab0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 --2- 172.21.15.116:0/474479636 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3888106640 0x7f3888106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 -- 172.21.15.116:0/474479636 >> 172.21.15.116:0/474479636 conn(0x7f3888077770 msgr2=0x7f3888077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 -- 172.21.15.116:0/474479636 shutdown_connections 2024-03-24T09:28:09.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.879+0000 7f388dbf4640 1 -- 172.21.15.116:0/474479636 wait complete. 2024-03-24T09:28:09.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 Processor -- start 2024-03-24T09:28:09.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 -- start start 2024-03-24T09:28:09.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3888100630 0x7f388819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 0x7f388819de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888106640 0x7f3888197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3888198150 con 0x7f3888106640 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f38881982c0 con 0x7f3888100630 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f388dbf4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3888198480 con 0x7f3888100ff0 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f3886ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 0x7f388819de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f3887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888106640 0x7f3888197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f3886ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 0x7f388819de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45314/0 (socket says 172.21.15.116:45314) 2024-03-24T09:28:09.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f38877fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3888100630 0x7f388819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.880+0000 7f3886ffd640 1 -- 172.21.15.116:0/3701298543 learned_addr learned my addr 172.21.15.116:0/3701298543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3886ffd640 1 -- 172.21.15.116:0/3701298543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3888100630 msgr2=0x7f388819d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3886ffd640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3888100630 0x7f388819d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3886ffd640 1 -- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888106640 msgr2=0x7f3888197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3886ffd640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888106640 0x7f3888197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3886ffd640 1 -- 172.21.15.116:0/3701298543 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3880009900 con 0x7f3888100ff0 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3887fff640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888106640 0x7f3888197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3886ffd640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 0x7f388819de90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f387c02c180 tx=0x7f387c004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3884ff9640 1 -- 172.21.15.116:0/3701298543 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c034850 con 0x7f3888100ff0 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3884ff9640 1 -- 172.21.15.116:0/3701298543 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f387c02ce90 con 0x7f3888100ff0 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387c009490 con 0x7f3888100ff0 2024-03-24T09:28:09.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f3884ff9640 1 -- 172.21.15.116:0/3701298543 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387c003cd0 con 0x7f3888100ff0 2024-03-24T09:28:09.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.881+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38881a4810 con 0x7f3888100ff0 2024-03-24T09:28:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f3884ff9640 1 -- 172.21.15.116:0/3701298543 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f387c03b040 con 0x7f3888100ff0 2024-03-24T09:28:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 Processor -- start 2024-03-24T09:28:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 -- start start 2024-03-24T09:28:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 0x56292d7357d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d735d10 0x56292d73e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d740ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f3884ff9640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38700a2230 0x7f38700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56292d6a3840 con 0x56292d7353f0 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56292d6a0d70 con 0x56292d735d10 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.883+0000 7f38907cf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56292d69c550 con 0x56292d73e8f0 2024-03-24T09:28:09.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 0x56292d7357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 0x56292d7357d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37378/0 (socket says 172.21.15.116:37378) 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d735d10 0x56292d73e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3887fff640 1 -- 172.21.15.116:0/1942904950 learned_addr learned my addr 172.21.15.116:0/1942904950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3886ffd640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38700a2230 0x7f38700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3887fff640 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d740ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3884ff9640 1 -- 172.21.15.116:0/3701298543 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f387c0efcb0 con 0x7f3888100ff0 2024-03-24T09:28:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 -- 172.21.15.116:0/1942904950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 msgr2=0x56292d740ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d740ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 -- 172.21.15.116:0/1942904950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 msgr2=0x56292d7357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 0x56292d7357d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 -- 172.21.15.116:0/1942904950 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56292d631de0 con 0x56292d735d10 2024-03-24T09:28:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3887fff640 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 0x56292d7357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f3886ffd640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38700a2230 0x7f38700a46f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f387c0bc1b0 tx=0x7f387c0dc040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f38877fe640 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d735d10 0x56292d73e3b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3878009c20 tx=0x7f3878011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bde689d75d2feec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f386dffb640 1 -- 172.21.15.116:0/1942904950 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3878004050 con 0x56292d735d10 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f386dffb640 1 -- 172.21.15.116:0/1942904950 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3878005020 con 0x56292d735d10 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.884+0000 7f386dffb640 1 -- 172.21.15.116:0/1942904950 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3878004ac0 con 0x56292d735d10 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 -- 172.21.15.116:0/1942904950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d735d10 msgr2=0x56292d73e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d735d10 0x56292d73e3b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3878009c20 tx=0x7f3878011810 comp rx=0 tx=0).stop 2024-03-24T09:28:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 -- 172.21.15.116:0/1942904950 shutdown_connections 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d740ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d735d10 0x56292d73e3b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 --2- 172.21.15.116:0/1942904950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d7353f0 0x56292d7357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 -- 172.21.15.116:0/1942904950 >> 172.21.15.116:0/1942904950 conn(0x56292d72fb20 msgr2=0x56292d72ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 -- 172.21.15.116:0/1942904950 shutdown_connections 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 -- 172.21.15.116:0/1942904950 wait complete. 2024-03-24T09:28:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.885+0000 7f38907cf740 1 -- start start 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 Processor -- start 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d73b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d735d10 0x56292d73beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 0x56292d73c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56292d36f430 con 0x56292d73e8f0 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56292d32be00 con 0x56292d7353f0 2024-03-24T09:28:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56292d621590 con 0x56292d735d10 2024-03-24T09:28:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38907cf740 10 client.0 _ll_register_callbacks cb 0x56292d690fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d73b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 0x56292d73c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 0x56292d73c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37386/0 (socket says 172.21.15.116:37386) 2024-03-24T09:28:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 -- 172.21.15.116:0/869720337 learned_addr learned my addr 172.21.15.116:0/869720337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f38877fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d735d10 0x56292d73beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 -- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d735d10 msgr2=0x56292d73beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d735d10 0x56292d73beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 -- 172.21.15.116:0/869720337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 msgr2=0x56292d73b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d73b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 -- 172.21.15.116:0/869720337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3878009850 con 0x56292d73e8f0 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3887fff640 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 0x56292d73c3f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3880009800 tx=0x7f3880009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786c9121dda81d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.886+0000 7f3866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f3866ffd640 1 -- 172.21.15.116:0/869720337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3880013070 con 0x56292d73e8f0 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f3866ffd640 1 -- 172.21.15.116:0/869720337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f3880003a30 con 0x56292d73e8f0 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56292d39b7e0 con 0x56292d73e8f0 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f3866ffd640 1 -- 172.21.15.116:0/869720337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3880020450 con 0x56292d73e8f0 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 10 client.17379 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56292d69abd0 con 0x56292d73e8f0 2024-03-24T09:28:09.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f3866ffd640 1 -- 172.21.15.116:0/869720337 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56292d39b7e0 con 0x56292d73e8f0 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f3866ffd640 1 -- 172.21.15.116:0/869720337 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f388001f470 con 0x56292d73e8f0 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 10 client.17379 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 10 client.17379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 20 client.17379 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d74ea50 0x56292d750e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 4 client.17379 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:09.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56292d3d8720 con 0x56292d74ea50 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.887+0000 7f38877fe640 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d74ea50 0x56292d750e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f3866ffd640 10 client.17379 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f38877fe640 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d74ea50 0x56292d750e70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=543a3805a3805f15 server_cookie=fcb53f0c4bab7477 in_seq=0 out_seq=0 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f3866ffd640 1 -- 172.21.15.116:0/869720337 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56292d3d8720 con 0x56292d74ea50 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f3866ffd640 10 client.17379 handle_command_reply: tid=0 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f38907cf740 1 client.17379 shutdown 2024-03-24T09:28:09.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 msgr2=0x56292d73c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.888+0000 7f38907cf740 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 0x56292d73c3f0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f3880009800 tx=0x7f3880009ce0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 shutdown_connections 2024-03-24T09:28:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d73e8f0 0x56292d73c3f0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d735d10 0x56292d73beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d74ea50 0x56292d750e70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 --2- 172.21.15.116:0/869720337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d73b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f3866ffd640 0 client.17379 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 >> 172.21.15.116:0/869720337 conn(0x56292d72fb20 msgr2=0x56292d731870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 shutdown_connections 2024-03-24T09:28:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 1 -- 172.21.15.116:0/869720337 wait complete. 2024-03-24T09:28:09.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.889+0000 7f38907cf740 20 client.17379 trim_cache size 0 max 16384 2024-03-24T09:28:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 Processor -- start 2024-03-24T09:28:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 -- start start 2024-03-24T09:28:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d74a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d756720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d73d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56292d73b760 con 0x56292d735d10 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56292d730b20 con 0x56292d7353f0 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38907cf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56292d72b1c0 con 0x56292d73e8f0 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d756720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d756720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37392/0 (socket says 172.21.15.116:37392) 2024-03-24T09:28:09.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f3887fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d74a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 -- 172.21.15.116:0/1461215033 learned_addr learned my addr 172.21.15.116:0/1461215033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f3887fff640 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d73d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 -- 172.21.15.116:0/1461215033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 msgr2=0x56292d73d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d73d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 -- 172.21.15.116:0/1461215033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 msgr2=0x56292d74a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d74a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 -- 172.21.15.116:0/1461215033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56292d636c80 con 0x56292d735d10 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f38877fe640 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d756720 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3878009bf0 tx=0x7f38780097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d067ce735d0538 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.902+0000 7f386d7fa640 1 -- 172.21.15.116:0/1461215033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3878005190 con 0x56292d735d10 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f386d7fa640 1 -- 172.21.15.116:0/1461215033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f38780035f0 con 0x56292d735d10 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f386d7fa640 1 -- 172.21.15.116:0/1461215033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387802e860 con 0x56292d735d10 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 -- 172.21.15.116:0/1461215033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 msgr2=0x56292d756720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d756720 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3878009bf0 tx=0x7f38780097c0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 -- 172.21.15.116:0/1461215033 shutdown_connections 2024-03-24T09:28:09.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d73d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d756720 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 --2- 172.21.15.116:0/1461215033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d74a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 -- 172.21.15.116:0/1461215033 >> 172.21.15.116:0/1461215033 conn(0x56292d72fb20 msgr2=0x56292d736c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 -- 172.21.15.116:0/1461215033 shutdown_connections 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 -- 172.21.15.116:0/1461215033 wait complete. 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.903+0000 7f38907cf740 1 -- start start 2024-03-24T09:28:09.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 Processor -- start 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d7387b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d738cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d739230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56292d7b3ec0 con 0x56292d735d10 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56292d7b4030 con 0x56292d7353f0 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56292d7b41f0 con 0x56292d73e8f0 2024-03-24T09:28:09.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38907cf740 10 client.0 _ll_register_callbacks cb 0x7f388001ec90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d738cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d738cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37402/0 (socket says 172.21.15.116:37402) 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 -- 172.21.15.116:0/489017674 learned_addr learned my addr 172.21.15.116:0/489017674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f3887fff640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d739230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f3887fff640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d7387b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 -- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 msgr2=0x56292d739230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d739230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 -- 172.21.15.116:0/489017674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 msgr2=0x56292d7387b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d7387b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 -- 172.21.15.116:0/489017674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56292d631de0 con 0x56292d735d10 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f38877fe640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d738cf0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3878009c20 tx=0x7f3878009d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6754a98033f48e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.904+0000 7f3866ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f3866ffd640 1 -- 172.21.15.116:0/489017674 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387802eab0 con 0x56292d735d10 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f3866ffd640 1 -- 172.21.15.116:0/489017674 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f387802ec10 con 0x56292d735d10 2024-03-24T09:28:09.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3878005020 con 0x56292d735d10 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f3866ffd640 1 -- 172.21.15.116:0/489017674 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f387802edd0 con 0x56292d735d10 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 10 client.17385 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56292d65c920 con 0x56292d735d10 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f3866ffd640 1 -- 172.21.15.116:0/489017674 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3878005020 con 0x56292d735d10 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f3866ffd640 1 -- 172.21.15.116:0/489017674 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3878021040 con 0x56292d735d10 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 10 client.17385 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:09.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 10 client.17385 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 20 client.17385 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d7b5780 0x56292d7b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 4 client.17385 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56292d427890 con 0x56292d7b5780 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.905+0000 7f3887fff640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d7b5780 0x56292d7b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f3887fff640 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d7b5780 0x56292d7b7b70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e366a2d74102d28a server_cookie=d0d31dea65590bfa in_seq=0 out_seq=0 2024-03-24T09:28:09.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f3866ffd640 10 client.17385 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f3866ffd640 1 -- 172.21.15.116:0/489017674 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56292d427890 con 0x56292d7b5780 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f3866ffd640 10 client.17385 handle_command_reply: tid=0 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 client.17385 shutdown 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 msgr2=0x56292d738cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d738cf0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f3878009c20 tx=0x7f3878009d90 comp rx=0 tx=0).stop 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 shutdown_connections 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56292d73e8f0 0x56292d739230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56292d735d10 0x56292d738cf0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56292d7b5780 0x56292d7b7b70 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 --2- 172.21.15.116:0/489017674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56292d7353f0 0x56292d7387b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f3866ffd640 0 client.17385 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 >> 172.21.15.116:0/489017674 conn(0x56292d72fb20 msgr2=0x56292d72e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 shutdown_connections 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 1 -- 172.21.15.116:0/489017674 wait complete. 2024-03-24T09:28:09.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f38907cf740 20 client.17385 trim_cache size 0 max 16384 2024-03-24T09:28:09.926 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:09.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38700a2230 msgr2=0x7f38700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f388dbf4640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38700a2230 0x7f38700a46f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f387c0bc1b0 tx=0x7f387c0dc040 comp rx=0 tx=0).stop 2024-03-24T09:28:09.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 msgr2=0x7f388819de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.906+0000 7f388dbf4640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 0x7f388819de90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f387c02c180 tx=0x7f387c004550 comp rx=0 tx=0).stop 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 shutdown_connections 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3888106640 0x7f3888197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f38700a2230 0x7f38700a46f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3888100ff0 0x7f388819de90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 --2- 172.21.15.116:0/3701298543 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3888100630 0x7f388819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:09.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 >> 172.21.15.116:0/3701298543 conn(0x7f3888077770 msgr2=0x7f3888103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:09.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 shutdown_connections 2024-03-24T09:28:09.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:09.907+0000 7f388dbf4640 1 -- 172.21.15.116:0/3701298543 wait complete. 2024-03-24T09:28:09.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:28:10.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.036+0000 7fd6252ae640 1 -- 172.21.15.116:0/3593779141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 msgr2=0x7fd62010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.036+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3593779141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd62010d420 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd618009ca0 tx=0x7fd61802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- 172.21.15.116:0/3593779141 shutdown_connections 2024-03-24T09:28:10.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3593779141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd62010d420 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3593779141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6201005a0 0x7fd620100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3593779141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd620106660 0x7fd620106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- 172.21.15.116:0/3593779141 >> 172.21.15.116:0/3593779141 conn(0x7fd620077710 msgr2=0x7fd620077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- 172.21.15.116:0/3593779141 shutdown_connections 2024-03-24T09:28:10.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- 172.21.15.116:0/3593779141 wait complete. 2024-03-24T09:28:10.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 Processor -- start 2024-03-24T09:28:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- start start 2024-03-24T09:28:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6201005a0 0x7fd6201989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd620198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd620106660 0x7fd6201a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd620199770 con 0x7fd620100f60 2024-03-24T09:28:10.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd6201a3f70 con 0x7fd6201005a0 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.037+0000 7fd6252ae640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd6201a40b0 con 0x7fd620106660 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd620198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd620198ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37428/0 (socket says 172.21.15.116:37428) 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 -- 172.21.15.116:0/3004361650 learned_addr learned my addr 172.21.15.116:0/3004361650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd61effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6201005a0 0x7fd6201989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd620106660 0x7fd6201a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 -- 172.21.15.116:0/3004361650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd620106660 msgr2=0x7fd6201a3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd620106660 0x7fd6201a3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 -- 172.21.15.116:0/3004361650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6201005a0 msgr2=0x7fd6201989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6201005a0 0x7fd6201989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 -- 172.21.15.116:0/3004361650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd608009490 con 0x7fd620100f60 2024-03-24T09:28:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd617fff640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd620198ef0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd60800ea90 tx=0x7fd60800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd61cff9640 1 -- 172.21.15.116:0/3004361650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60800cde0 con 0x7fd620100f60 2024-03-24T09:28:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd618009900 con 0x7fd620100f60 2024-03-24T09:28:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd61cff9640 1 -- 172.21.15.116:0/3004361650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd6080044d0 con 0x7fd620100f60 2024-03-24T09:28:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd61cff9640 1 -- 172.21.15.116:0/3004361650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd608010430 con 0x7fd620100f60 2024-03-24T09:28:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.038+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6201a46e0 con 0x7fd620100f60 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.040+0000 7fd61cff9640 1 -- 172.21.15.116:0/3004361650 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd608003fe0 con 0x7fd620100f60 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 Processor -- start 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 -- start start 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61cff9640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd60c0a2230 0x7fd60c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd60c0a2230 0x7fd60c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61cff9640 1 -- 172.21.15.116:0/3004361650 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd608014070 con 0x7fd620100f60 2024-03-24T09:28:10.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 0x563f95cf07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cf93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf98f0 0x563f95cfbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f95c5e840 con 0x563f95cf98f0 2024-03-24T09:28:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f95c5bd70 con 0x563f95cf03f0 2024-03-24T09:28:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd627f6d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f95c57550 con 0x563f95cf0d10 2024-03-24T09:28:10.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 0x563f95cf07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd617fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cf93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 0x563f95cf07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59194/0 (socket says 172.21.15.116:59194) 2024-03-24T09:28:10.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd617fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cf93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45348/0 (socket says 172.21.15.116:45348) 2024-03-24T09:28:10.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61effd640 1 -- 172.21.15.116:0/274508036 learned_addr learned my addr 172.21.15.116:0/274508036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.041+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf98f0 0x563f95cfbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd60c0a2230 0x7fd60c0a46f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd618004af0 tx=0x7fd618005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61effd640 1 -- 172.21.15.116:0/274508036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 msgr2=0x563f95cf93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61effd640 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cf93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61effd640 1 -- 172.21.15.116:0/274508036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf98f0 msgr2=0x563f95cfbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61effd640 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf98f0 0x563f95cfbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61effd640 1 -- 172.21.15.116:0/274508036 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f95becde0 con 0x563f95cf03f0 2024-03-24T09:28:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd61effd640 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 0x563f95cf07d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd610009c70 tx=0x7fd610011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c316d3d09300445f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd6157fa640 1 -- 172.21.15.116:0/274508036 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610004230 con 0x563f95cf03f0 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd6157fa640 1 -- 172.21.15.116:0/274508036 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd610004390 con 0x563f95cf03f0 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.042+0000 7fd6157fa640 1 -- 172.21.15.116:0/274508036 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610019640 con 0x563f95cf03f0 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- 172.21.15.116:0/274508036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 msgr2=0x563f95cf07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 0x563f95cf07d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd610009c70 tx=0x7fd610011810 comp rx=0 tx=0).stop 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- 172.21.15.116:0/274508036 shutdown_connections 2024-03-24T09:28:10.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf98f0 0x563f95cfbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cf93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- 172.21.15.116:0/274508036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf03f0 0x563f95cf07d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- 172.21.15.116:0/274508036 >> 172.21.15.116:0/274508036 conn(0x563f95ceab20 msgr2=0x563f95ceaf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- 172.21.15.116:0/274508036 shutdown_connections 2024-03-24T09:28:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- 172.21.15.116:0/274508036 wait complete. 2024-03-24T09:28:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- start start 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 Processor -- start 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf03f0 0x563f95cf6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 0x563f95cf6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cf7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f9592a430 con 0x563f95cf0d10 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f958e6e00 con 0x563f95cf98f0 2024-03-24T09:28:10.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.043+0000 7fd627f6d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f95bdc590 con 0x563f95cf03f0 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd627f6d740 10 client.0 _ll_register_callbacks cb 0x563f95c4bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 0x563f95cf6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd61effd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf03f0 0x563f95cf6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd61f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cf7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 0x563f95cf6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37450/0 (socket says 172.21.15.116:37450) 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 -- 172.21.15.116:0/2644197612 learned_addr learned my addr 172.21.15.116:0/2644197612 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 -- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf03f0 msgr2=0x563f95cf6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf03f0 0x563f95cf6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 -- 172.21.15.116:0/2644197612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 msgr2=0x563f95cf7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cf7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 -- 172.21.15.116:0/2644197612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6100098d0 con 0x563f95cf0d10 2024-03-24T09:28:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cf7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd617fff640 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 0x563f95cf6e50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd60809ad20 tx=0x7fd60809b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3cda5361513de6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd6067fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd6067fc640 1 -- 172.21.15.116:0/2644197612 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6080cc000 con 0x563f95cf0d10 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd6067fc640 1 -- 172.21.15.116:0/2644197612 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd6080cc160 con 0x563f95cf0d10 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563f959567e0 con 0x563f95cf0d10 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd6067fc640 1 -- 172.21.15.116:0/2644197612 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd608097690 con 0x563f95cf0d10 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd627f6d740 10 client.17397 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563f95c55bd0 con 0x563f95cf0d10 2024-03-24T09:28:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.044+0000 7fd6067fc640 1 -- 172.21.15.116:0/2644197612 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd608082080 con 0x563f95cf0d10 2024-03-24T09:28:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd6067fc640 1 -- 172.21.15.116:0/2644197612 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd608095b90 con 0x563f95cf0d10 2024-03-24T09:28:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd627f6d740 10 client.17397 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd627f6d740 10 client.17397 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:28:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd627f6d740 20 client.17397 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd627f6d740 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d0beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd627f6d740 4 client.17397 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f95993720 con 0x563f95d09a90 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd61effd640 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d0beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd6067fc640 10 client.17397 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.045+0000 7fd61effd640 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d0beb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55f1bea60c716cbd server_cookie=a144b3fa45f7a6e0 in_seq=0 out_seq=0 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd6067fc640 1 -- 172.21.15.116:0/2644197612 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f95993720 con 0x563f95d09a90 2024-03-24T09:28:10.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd6067fc640 10 client.17397 handle_command_reply: tid=0 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 client.17397 shutdown 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 msgr2=0x563f95cf6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 0x563f95cf6e50 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd60809ad20 tx=0x7fd60809b860 comp rx=0 tx=0).stop 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 shutdown_connections 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d0beb0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cf7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd6067fc640 0 client.17397 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf0d10 0x563f95cf6e50 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 --2- 172.21.15.116:0/2644197612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf03f0 0x563f95cf6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 >> 172.21.15.116:0/2644197612 conn(0x563f95ceab20 msgr2=0x563f95cec870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 shutdown_connections 2024-03-24T09:28:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 1 -- 172.21.15.116:0/2644197612 wait complete. 2024-03-24T09:28:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.046+0000 7fd627f6d740 20 client.17397 trim_cache size 0 max 16384 2024-03-24T09:28:10.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 Processor -- start 2024-03-24T09:28:10.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 -- start start 2024-03-24T09:28:10.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95ce5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf0d10 0x563f95d05180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf98f0 0x563f95cf71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f95d098d0 con 0x563f95cf03f0 2024-03-24T09:28:10.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f95ce6090 con 0x563f95cf0d10 2024-03-24T09:28:10.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.059+0000 7fd627f6d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f95cdbe50 con 0x563f95cf98f0 2024-03-24T09:28:10.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95ce5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95ce5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37466/0 (socket says 172.21.15.116:37466) 2024-03-24T09:28:10.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 -- 172.21.15.116:0/4260871597 learned_addr learned my addr 172.21.15.116:0/4260871597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd617fff640 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf0d10 0x563f95d05180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61f7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf98f0 0x563f95cf71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 -- 172.21.15.116:0/4260871597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf98f0 msgr2=0x563f95cf71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf98f0 0x563f95cf71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 -- 172.21.15.116:0/4260871597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf0d10 msgr2=0x563f95d05180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf0d10 0x563f95d05180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 -- 172.21.15.116:0/4260871597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f95bf1c80 con 0x563f95cf03f0 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd617fff640 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf0d10 0x563f95d05180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd61effd640 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95ce5b50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd610009bc0 tx=0x7fd610009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2f47fc1781c586 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd614ff9640 1 -- 172.21.15.116:0/4260871597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610004020 con 0x563f95cf03f0 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd614ff9640 1 -- 172.21.15.116:0/4260871597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd610004180 con 0x563f95cf03f0 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.060+0000 7fd614ff9640 1 -- 172.21.15.116:0/4260871597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd610003c40 con 0x563f95cf03f0 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- 172.21.15.116:0/4260871597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 msgr2=0x563f95ce5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95ce5b50 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fd610009bc0 tx=0x7fd610009e20 comp rx=0 tx=0).stop 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- 172.21.15.116:0/4260871597 shutdown_connections 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf98f0 0x563f95cf71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf0d10 0x563f95d05180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- 172.21.15.116:0/4260871597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95ce5b50 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- 172.21.15.116:0/4260871597 >> 172.21.15.116:0/4260871597 conn(0x563f95ceab20 msgr2=0x563f95ce9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- 172.21.15.116:0/4260871597 shutdown_connections 2024-03-24T09:28:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- 172.21.15.116:0/4260871597 wait complete. 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- start start 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 Processor -- start 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95d017b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cfe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cfecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f95d01ed0 con 0x563f95cf03f0 2024-03-24T09:28:10.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f95cff200 con 0x563f95cf98f0 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f95cff3c0 con 0x563f95cf0d10 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd627f6d740 10 client.0 _ll_register_callbacks cb 0x7fd60800e9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd61effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95d017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd61effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95d017b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37470/0 (socket says 172.21.15.116:37470) 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd61effd640 1 -- 172.21.15.116:0/3325960271 learned_addr learned my addr 172.21.15.116:0/3325960271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.061+0000 7fd617fff640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cfe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61f7fe640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cfecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61effd640 1 -- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 msgr2=0x563f95cfe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61effd640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cfe780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61effd640 1 -- 172.21.15.116:0/3325960271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 msgr2=0x563f95cfecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61effd640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cfecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61effd640 1 -- 172.21.15.116:0/3325960271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f95c55bd0 con 0x563f95cf03f0 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd61effd640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95d017b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd610004330 tx=0x7fd6100040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5caddbbdc4a713 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd6067fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd6067fc640 1 -- 172.21.15.116:0/3325960271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6100035f0 con 0x563f95cf03f0 2024-03-24T09:28:10.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd6067fc640 1 -- 172.21.15.116:0/3325960271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd6100038c0 con 0x563f95cf03f0 2024-03-24T09:28:10.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd610004390 con 0x563f95cf03f0 2024-03-24T09:28:10.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd6067fc640 1 -- 172.21.15.116:0/3325960271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd610003a80 con 0x563f95cf03f0 2024-03-24T09:28:10.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd627f6d740 10 client.17403 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:10.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7fd61001e040 con 0x563f95cf03f0 2024-03-24T09:28:10.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd6067fc640 1 -- 172.21.15.116:0/3325960271 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd610004390 con 0x563f95cf03f0 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.062+0000 7fd6067fc640 1 -- 172.21.15.116:0/3325960271 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd610004650 con 0x563f95cf03f0 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 10 client.17403 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 10 client.17403 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 20 client.17403 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d72e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 4 client.17403 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f959e2890 con 0x563f95d09a90 2024-03-24T09:28:10.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd617fff640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d72e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd617fff640 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d72e70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce2f43ddc9679c28 server_cookie=40982b4a0a24aa3f in_seq=0 out_seq=0 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd6067fc640 10 client.17403 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd6067fc640 1 -- 172.21.15.116:0/3325960271 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f959e2890 con 0x563f95d09a90 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd6067fc640 10 client.17403 handle_command_reply: tid=0 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 client.17403 shutdown 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 msgr2=0x563f95d017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95d017b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fd610004330 tx=0x7fd6100040d0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 shutdown_connections 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563f95d09a90 0x563f95d72e70 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f95cf98f0 0x563f95cfecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f95cf0d10 0x563f95cfe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd6067fc640 0 client.17403 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 --2- 172.21.15.116:0/3325960271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f95cf03f0 0x563f95d017b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 >> 172.21.15.116:0/3325960271 conn(0x563f95ceab20 msgr2=0x563f95cebce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 shutdown_connections 2024-03-24T09:28:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.063+0000 7fd627f6d740 1 -- 172.21.15.116:0/3325960271 wait complete. 2024-03-24T09:28:10.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd627f6d740 20 client.17403 trim_cache size 0 max 16384 2024-03-24T09:28:10.086 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd60c0a2230 msgr2=0x7fd60c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd60c0a2230 0x7fd60c0a46f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd618004af0 tx=0x7fd618005c50 comp rx=0 tx=0).stop 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 msgr2=0x7fd620198ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd620198ef0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd60800ea90 tx=0x7fd60800ef50 comp rx=0 tx=0).stop 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 shutdown_connections 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd60c0a2230 0x7fd60c0a46f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd620106660 0x7fd6201a3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd620100f60 0x7fd620198ef0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 --2- 172.21.15.116:0/3004361650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6201005a0 0x7fd6201989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 >> 172.21.15.116:0/3004361650 conn(0x7fd620077710 msgr2=0x7fd6200ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 shutdown_connections 2024-03-24T09:28:10.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.064+0000 7fd6252ae640 1 -- 172.21.15.116:0/3004361650 wait complete. 2024-03-24T09:28:10.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:28:10.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- 172.21.15.116:0/1189601153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 msgr2=0x7fab4810b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- 172.21.15.116:0/1189601153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4810b750 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fab44009900 tx=0x7fab4402bcd0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- 172.21.15.116:0/1189601153 shutdown_connections 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- 172.21.15.116:0/1189601153 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab4810bd20 0x7fab4810e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- 172.21.15.116:0/1189601153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4810b750 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- 172.21.15.116:0/1189601153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab48102950 0x7fab48102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- 172.21.15.116:0/1189601153 >> 172.21.15.116:0/1189601153 conn(0x7fab48076990 msgr2=0x7fab48076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- 172.21.15.116:0/1189601153 shutdown_connections 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- 172.21.15.116:0/1189601153 wait complete. 2024-03-24T09:28:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 Processor -- start 2024-03-24T09:28:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- start start 2024-03-24T09:28:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab48102950 0x7fab4819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab4810bd20 0x7fab48197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab481980b0 con 0x7fab48103270 2024-03-24T09:28:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fab48198220 con 0x7fab4810bd20 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.194+0000 7fab4f76b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fab481983e0 con 0x7fab48102950 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4819ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37486/0 (socket says 172.21.15.116:37486) 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4d507640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab48102950 0x7fab4819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 -- 172.21.15.116:0/3952672938 learned_addr learned my addr 172.21.15.116:0/3952672938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab4810bd20 0x7fab48197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 -- 172.21.15.116:0/3952672938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab48102950 msgr2=0x7fab4819d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab48102950 0x7fab4819d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 -- 172.21.15.116:0/3952672938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab4810bd20 msgr2=0x7fab48197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab4810bd20 0x7fab48197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 -- 172.21.15.116:0/3952672938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab44009560 con 0x7fab48103270 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4819ddf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fab44006010 tx=0x7fab44005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab3e7fc640 1 -- 172.21.15.116:0/3952672938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab440043d0 con 0x7fab48103270 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab3e7fc640 1 -- 172.21.15.116:0/3952672938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab44003aa0 con 0x7fab48103270 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab48198660 con 0x7fab48103270 2024-03-24T09:28:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab3e7fc640 1 -- 172.21.15.116:0/3952672938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab44005350 con 0x7fab48103270 2024-03-24T09:28:10.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.195+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab481a4950 con 0x7fab48103270 2024-03-24T09:28:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.197+0000 7fab3e7fc640 1 -- 172.21.15.116:0/3952672938 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fab44003c10 con 0x7fab48103270 2024-03-24T09:28:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 Processor -- start 2024-03-24T09:28:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 -- start start 2024-03-24T09:28:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab3e7fc640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fab340a2230 0x7fab340a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a41ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5573059a4840 con 0x557305a36d10 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5573059a1d70 con 0x557305a363f0 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab5241f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55730599d550 con 0x557305a3f8f0 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4d507640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fab340a2230 0x7fab340a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37488/0 (socket says 172.21.15.116:37488) 2024-03-24T09:28:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4cd06640 1 -- 172.21.15.116:0/3644520989 learned_addr learned my addr 172.21.15.116:0/3644520989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a41ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab3e7fc640 1 -- 172.21.15.116:0/3952672938 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fab440f0ac0 con 0x7fab48103270 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 -- 172.21.15.116:0/3644520989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 msgr2=0x557305a41ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a41ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 -- 172.21.15.116:0/3644520989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 msgr2=0x557305a3f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4dd08640 1 -- 172.21.15.116:0/3644520989 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557305932de0 con 0x557305a363f0 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.198+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab4d507640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fab340a2230 0x7fab340a46f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fab48198ba0 tx=0x7fab38005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a367d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fab40005fd0 tx=0x7fab40004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=337b1b236079bade server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab337fe640 1 -- 172.21.15.116:0/3644520989 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab40011050 con 0x557305a363f0 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab337fe640 1 -- 172.21.15.116:0/3644520989 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab400111b0 con 0x557305a363f0 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab337fe640 1 -- 172.21.15.116:0/3644520989 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4000f9c0 con 0x557305a363f0 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 -- 172.21.15.116:0/3644520989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 msgr2=0x557305a367d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a367d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fab40005fd0 tx=0x7fab40004ab0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 -- 172.21.15.116:0/3644520989 shutdown_connections 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a41ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 --2- 172.21.15.116:0/3644520989 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a367d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.199+0000 7fab5241f740 1 -- 172.21.15.116:0/3644520989 >> 172.21.15.116:0/3644520989 conn(0x557305a30b20 msgr2=0x557305a30f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 -- 172.21.15.116:0/3644520989 shutdown_connections 2024-03-24T09:28:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 -- 172.21.15.116:0/3644520989 wait complete. 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 -- start start 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 Processor -- start 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a3c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a3d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557305670430 con 0x557305a36d10 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55730562ce00 con 0x557305a363f0 2024-03-24T09:28:10.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557305922590 con 0x557305a3f8f0 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab5241f740 10 client.0 _ll_register_callbacks cb 0x557305991fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37502/0 (socket says 172.21.15.116:37502) 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab4d507640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a3c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab4cd06640 1 -- 172.21.15.116:0/3169516157 learned_addr learned my addr 172.21.15.116:0/3169516157 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.200+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a3d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab4cd06640 1 -- 172.21.15.116:0/3169516157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 msgr2=0x557305a3d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a3d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab4cd06640 1 -- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 msgr2=0x557305a3c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a3c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab4cd06640 1 -- 172.21.15.116:0/3169516157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab40019040 con 0x557305a36d10 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3ce70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fab440bd1a0 tx=0x7fab440bcb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba8d11f81d24ee0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab30ff9640 1 -- 172.21.15.116:0/3169516157 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab440c6180 con 0x557305a36d10 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab30ff9640 1 -- 172.21.15.116:0/3169516157 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fab440c1120 con 0x557305a36d10 2024-03-24T09:28:10.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55730569c7e0 con 0x557305a36d10 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab30ff9640 1 -- 172.21.15.116:0/3169516157 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fab440bd210 con 0x557305a36d10 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 10 client.17415 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55730599bbd0 con 0x557305a36d10 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab30ff9640 1 -- 172.21.15.116:0/3169516157 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55730569c7e0 con 0x557305a36d10 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab30ff9640 1 -- 172.21.15.116:0/3169516157 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fab440d3030 con 0x557305a36d10 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 10 client.17415 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 10 client.17415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:10.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 20 client.17415 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305a4f760 0x557305a51b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 4 client.17415 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.201+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573056d9720 con 0x557305a4f760 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.202+0000 7fab4d507640 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305a4f760 0x557305a51b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.202+0000 7fab30ff9640 10 client.17415 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.202+0000 7fab4d507640 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305a4f760 0x557305a51b80 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ab1ab668232e3c server_cookie=688b7d2c19616633 in_seq=0 out_seq=0 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab30ff9640 1 -- 172.21.15.116:0/3169516157 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5573056d9720 con 0x557305a4f760 2024-03-24T09:28:10.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab30ff9640 10 client.17415 handle_command_reply: tid=0 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 client.17415 shutdown 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 msgr2=0x557305a3ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3ce70 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fab440bd1a0 tx=0x7fab440bcb70 comp rx=0 tx=0).stop 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 shutdown_connections 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a3d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a36d10 0x557305a3ce70 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305a4f760 0x557305a51b80 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 --2- 172.21.15.116:0/3169516157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a363f0 0x557305a3c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 >> 172.21.15.116:0/3169516157 conn(0x557305a30b20 msgr2=0x557305a32870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab30ff9640 0 client.17415 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 shutdown_connections 2024-03-24T09:28:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 1 -- 172.21.15.116:0/3169516157 wait complete. 2024-03-24T09:28:10.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.203+0000 7fab5241f740 20 client.17415 trim_cache size 0 max 16384 2024-03-24T09:28:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 Processor -- start 2024-03-24T09:28:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 -- start start 2024-03-24T09:28:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 0x557305a4b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a3d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a57490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557305a4bdf0 con 0x557305a363f0 2024-03-24T09:28:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557305a2c210 con 0x557305a36d10 2024-03-24T09:28:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab5241f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557305a30990 con 0x557305a3f8f0 2024-03-24T09:28:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab4d507640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 0x557305a4b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab4d507640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 0x557305a4b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37508/0 (socket says 172.21.15.116:37508) 2024-03-24T09:28:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab4d507640 1 -- 172.21.15.116:0/3127370343 learned_addr learned my addr 172.21.15.116:0/3127370343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab4cd06640 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a3d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.216+0000 7fab4dd08640 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a57490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab4d507640 1 -- 172.21.15.116:0/3127370343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 msgr2=0x557305a57490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab4d507640 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a57490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab4d507640 1 -- 172.21.15.116:0/3127370343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 msgr2=0x557305a3d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab4d507640 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a3d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab4d507640 1 -- 172.21.15.116:0/3127370343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557305937c80 con 0x557305a363f0 2024-03-24T09:28:10.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab4d507640 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 0x557305a4b150 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x557305a52400 tx=0x7fab38014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d46089f4cbf0d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab32ffd640 1 -- 172.21.15.116:0/3127370343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38022040 con 0x557305a363f0 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab32ffd640 1 -- 172.21.15.116:0/3127370343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab38003cc0 con 0x557305a363f0 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab32ffd640 1 -- 172.21.15.116:0/3127370343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3801d650 con 0x557305a363f0 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab5241f740 1 -- 172.21.15.116:0/3127370343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 msgr2=0x557305a4b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.217+0000 7fab5241f740 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 0x557305a4b150 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x557305a52400 tx=0x7fab38014000 comp rx=0 tx=0).stop 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- 172.21.15.116:0/3127370343 shutdown_connections 2024-03-24T09:28:10.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a3f8f0 0x557305a57490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a3d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 --2- 172.21.15.116:0/3127370343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a363f0 0x557305a4b150 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- 172.21.15.116:0/3127370343 >> 172.21.15.116:0/3127370343 conn(0x557305a30b20 msgr2=0x557305a327e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- 172.21.15.116:0/3127370343 shutdown_connections 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- 172.21.15.116:0/3127370343 wait complete. 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- start start 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 Processor -- start 2024-03-24T09:28:10.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a363f0 0x557305a39500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a39a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 0x557305a39f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x557305ab4ac0 con 0x557305a3f8f0 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557305ab4c00 con 0x557305a36d10 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x557305ab4d40 con 0x557305a363f0 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab5241f740 10 client.0 _ll_register_callbacks cb 0x7fab440edf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab4dd08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 0x557305a39f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab4cd06640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a39a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab4d507640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a363f0 0x557305a39500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab4dd08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 0x557305a39f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37512/0 (socket says 172.21.15.116:37512) 2024-03-24T09:28:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.218+0000 7fab4dd08640 1 -- 172.21.15.116:0/1290345618 learned_addr learned my addr 172.21.15.116:0/1290345618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4dd08640 1 -- 172.21.15.116:0/1290345618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a363f0 msgr2=0x557305a39500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4dd08640 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a363f0 0x557305a39500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4dd08640 1 -- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 msgr2=0x557305a39a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:10.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4dd08640 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a39a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4dd08640 1 -- 172.21.15.116:0/1290345618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab38017070 con 0x557305a3f8f0 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4cd06640 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a39a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab4dd08640 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 0x557305a39f80 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fab40005fd0 tx=0x7fab40005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba802a0f7ea8a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab30ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab30ff9640 1 -- 172.21.15.116:0/1290345618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab40011320 con 0x557305a3f8f0 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab30ff9640 1 -- 172.21.15.116:0/1290345618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fab4000f9a0 con 0x557305a3f8f0 2024-03-24T09:28:10.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab400111b0 con 0x557305a3f8f0 2024-03-24T09:28:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab30ff9640 1 -- 172.21.15.116:0/1290345618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fab4000fba0 con 0x557305a3f8f0 2024-03-24T09:28:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab5241f740 10 client.17421 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55730595d920 con 0x557305a3f8f0 2024-03-24T09:28:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab30ff9640 1 -- 172.21.15.116:0/1290345618 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fab400111b0 con 0x557305a3f8f0 2024-03-24T09:28:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab30ff9640 1 -- 172.21.15.116:0/1290345618 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fab4001f450 con 0x557305a3f8f0 2024-03-24T09:28:10.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab5241f740 10 client.17421 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab5241f740 10 client.17421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.219+0000 7fab5241f740 20 client.17421 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305ab6410 0x557305ab8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 4 client.17421 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557305728890 con 0x557305ab6410 2024-03-24T09:28:10.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab4d507640 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305ab6410 0x557305ab8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab30ff9640 10 client.17421 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab4d507640 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305ab6410 0x557305ab8830 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc2755cd045382c server_cookie=91947ce310d6a37d in_seq=0 out_seq=0 2024-03-24T09:28:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab30ff9640 1 -- 172.21.15.116:0/1290345618 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557305728890 con 0x557305ab6410 2024-03-24T09:28:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab30ff9640 10 client.17421 handle_command_reply: tid=0 2024-03-24T09:28:10.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 client.17421 shutdown 2024-03-24T09:28:10.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 msgr2=0x557305a39f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 0x557305a39f80 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fab40005fd0 tx=0x7fab40005c70 comp rx=0 tx=0).stop 2024-03-24T09:28:10.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 shutdown_connections 2024-03-24T09:28:10.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x557305a3f8f0 0x557305a39f80 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x557305ab6410 0x557305ab8830 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557305a36d10 0x557305a39a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 --2- 172.21.15.116:0/1290345618 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x557305a363f0 0x557305a39500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 >> 172.21.15.116:0/1290345618 conn(0x557305a30b20 msgr2=0x557305a2f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab30ff9640 0 client.17421 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 shutdown_connections 2024-03-24T09:28:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 1 -- 172.21.15.116:0/1290345618 wait complete. 2024-03-24T09:28:10.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.220+0000 7fab5241f740 20 client.17421 trim_cache size 0 max 16384 2024-03-24T09:28:10.237 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fab340a2230 msgr2=0x7fab340a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fab340a2230 0x7fab340a46f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fab48198ba0 tx=0x7fab38005f90 comp rx=0 tx=0).stop 2024-03-24T09:28:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 msgr2=0x7fab4819ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:10.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4819ddf0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fab44006010 tx=0x7fab44005e10 comp rx=0 tx=0).stop 2024-03-24T09:28:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 shutdown_connections 2024-03-24T09:28:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab4810bd20 0x7fab48197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab48103270 0x7fab4819ddf0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fab340a2230 0x7fab340a46f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 --2- 172.21.15.116:0/3952672938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fab48102950 0x7fab4819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 >> 172.21.15.116:0/3952672938 conn(0x7fab48076990 msgr2=0x7fab480fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 shutdown_connections 2024-03-24T09:28:10.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:10.221+0000 7fab4f76b640 1 -- 172.21.15.116:0/3952672938 wait complete. 2024-03-24T09:28:10.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:10 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4084617694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:28:10.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:10 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4084617694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:28:10.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:10 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4084617694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:28:11.242 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:28:11.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:28:11.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:28:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:11 smithi117 ceph-mon[35788]: pgmap v185: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:11.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:11 smithi116 ceph-mon[32207]: pgmap v185: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:11.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 -- 172.21.15.116:0/3707948470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 msgr2=0x7fa084101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 --2- 172.21.15.116:0/3707948470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa084101b20 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fa070009930 tx=0x7fa07002bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 -- 172.21.15.116:0/3707948470 shutdown_connections 2024-03-24T09:28:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 --2- 172.21.15.116:0/3707948470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa084102060 0x7fa08410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 --2- 172.21.15.116:0/3707948470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa084101b20 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 --2- 172.21.15.116:0/3707948470 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0841076b0 0x7fa084107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 -- 172.21.15.116:0/3707948470 >> 172.21.15.116:0/3707948470 conn(0x7fa084078b10 msgr2=0x7fa084078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 -- 172.21.15.116:0/3707948470 shutdown_connections 2024-03-24T09:28:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.362+0000 7fa08ab02640 1 -- 172.21.15.116:0/3707948470 wait complete. 2024-03-24T09:28:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 Processor -- start 2024-03-24T09:28:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 -- start start 2024-03-24T09:28:11.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa08419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa084102060 0x7fa08419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0841076b0 0x7fa084197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa084198100 con 0x7fa0841016a0 2024-03-24T09:28:11.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa084198270 con 0x7fa084102060 2024-03-24T09:28:11.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08ab02640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa084198430 con 0x7fa0841076b0 2024-03-24T09:28:11.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08889e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa08419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08889e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa08419d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37532/0 (socket says 172.21.15.116:37532) 2024-03-24T09:28:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08889e640 1 -- 172.21.15.116:0/2642001706 learned_addr learned my addr 172.21.15.116:0/2642001706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa08909f640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0841076b0 0x7fa084197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.363+0000 7fa083fff640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa084102060 0x7fa08419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08889e640 1 -- 172.21.15.116:0/2642001706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0841076b0 msgr2=0x7fa084197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08889e640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0841076b0 0x7fa084197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08889e640 1 -- 172.21.15.116:0/2642001706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa084102060 msgr2=0x7fa08419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08889e640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa084102060 0x7fa08419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08889e640 1 -- 172.21.15.116:0/2642001706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa070009560 con 0x7fa0841016a0 2024-03-24T09:28:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08889e640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa08419d900 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa07800e9f0 tx=0x7fa07800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa081ffb640 1 -- 172.21.15.116:0/2642001706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078009720 con 0x7fa0841016a0 2024-03-24T09:28:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa084198710 con 0x7fa0841016a0 2024-03-24T09:28:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa081ffb640 1 -- 172.21.15.116:0/2642001706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa07800cd80 con 0x7fa0841016a0 2024-03-24T09:28:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa081ffb640 1 -- 172.21.15.116:0/2642001706 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078010430 con 0x7fa0841016a0 2024-03-24T09:28:11.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.364+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0841a49a0 con 0x7fa0841016a0 2024-03-24T09:28:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.366+0000 7fa081ffb640 1 -- 172.21.15.116:0/2642001706 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fa078005020 con 0x7fa0841016a0 2024-03-24T09:28:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 Processor -- start 2024-03-24T09:28:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa081ffb640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0740a2230 0x7fa0740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 -- start start 2024-03-24T09:28:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08909f640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0740a2230 0x7fa0740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddb760 0x55f7c7ddbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa081ffb640 1 -- 172.21.15.116:0/2642001706 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa078014070 con 0x7fa0841016a0 2024-03-24T09:28:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7de4c60 0x55f7c7de7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7d49530 con 0x55f7c7ddc080 2024-03-24T09:28:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7d46a60 con 0x55f7c7ddb760 2024-03-24T09:28:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7d42240 con 0x55f7c7de4c60 2024-03-24T09:28:11.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.367+0000 7fa083fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa083fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37548/0 (socket says 172.21.15.116:37548) 2024-03-24T09:28:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddb760 0x55f7c7ddbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa083fff640 1 -- 172.21.15.116:0/4197922385 learned_addr learned my addr 172.21.15.116:0/4197922385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08909f640 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7de4c60 0x55f7c7de7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08909f640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0740a2230 0x7fa0740a46f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa07c005fd0 tx=0x7fa07c011810 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 -- 172.21.15.116:0/4197922385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7de4c60 msgr2=0x55f7c7de7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7de4c60 0x55f7c7de7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 -- 172.21.15.116:0/4197922385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 msgr2=0x55f7c7de4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 -- 172.21.15.116:0/4197922385 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7c7d408c0 con 0x55f7c7ddb760 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa083fff640 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa08889e640 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddb760 0x55f7c7ddbb40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa0780cbb00 tx=0x7fa078097170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4edb6af1132a24b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa06effd640 1 -- 172.21.15.116:0/4197922385 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07809e070 con 0x55f7c7ddb760 2024-03-24T09:28:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa06effd640 1 -- 172.21.15.116:0/4197922385 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa07809ada0 con 0x55f7c7ddb760 2024-03-24T09:28:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.368+0000 7fa06effd640 1 -- 172.21.15.116:0/4197922385 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0780cbf20 con 0x55f7c7ddb760 2024-03-24T09:28:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 -- 172.21.15.116:0/4197922385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddb760 msgr2=0x55f7c7ddbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddb760 0x55f7c7ddbb40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa0780cbb00 tx=0x7fa078097170 comp rx=0 tx=0).stop 2024-03-24T09:28:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 -- 172.21.15.116:0/4197922385 shutdown_connections 2024-03-24T09:28:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7de4c60 0x55f7c7de7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/4197922385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddb760 0x55f7c7ddbb40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 -- 172.21.15.116:0/4197922385 >> 172.21.15.116:0/4197922385 conn(0x55f7c7dd5e90 msgr2=0x55f7c7dd6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 -- 172.21.15.116:0/4197922385 shutdown_connections 2024-03-24T09:28:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 -- 172.21.15.116:0/4197922385 wait complete. 2024-03-24T09:28:11.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 -- start start 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.369+0000 7fa08d7c2740 1 Processor -- start 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7de1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7de2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7ce6f80 con 0x55f7c7ddc080 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7a154b0 con 0x55f7c7de4c60 2024-03-24T09:28:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7cc7ad0 con 0x55f7c7ddb760 2024-03-24T09:28:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 10 client.0 _ll_register_callbacks cb 0x55f7c7cfab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa083fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7de2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa083fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37564/0 (socket says 172.21.15.116:37564) 2024-03-24T09:28:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa083fff640 1 -- 172.21.15.116:0/2529697451 learned_addr learned my addr 172.21.15.116:0/2529697451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08889e640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7de1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 -- 172.21.15.116:0/2529697451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 msgr2=0x55f7c7de1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7de1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 -- 172.21.15.116:0/2529697451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 msgr2=0x55f7c7de21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 -- 172.21.15.116:0/2529697451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0841a49a0 con 0x55f7c7de4c60 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa083fff640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08909f640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7de2710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d60 tx=0x7fa07c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d1a1912c462652f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa063fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:28:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa063fff640 1 -- 172.21.15.116:0/2529697451 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c022070 con 0x55f7c7de4c60 2024-03-24T09:28:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f7c7d667a0 con 0x55f7c7de4c60 2024-03-24T09:28:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.370+0000 7fa063fff640 1 -- 172.21.15.116:0/2529697451 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa07c004b20 con 0x55f7c7de4c60 2024-03-24T09:28:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa063fff640 1 -- 172.21.15.116:0/2529697451 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa07c004ce0 con 0x55f7c7de4c60 2024-03-24T09:28:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 10 client.27257 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f7c7cf86d0 con 0x55f7c7de4c60 2024-03-24T09:28:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa063fff640 1 -- 172.21.15.116:0/2529697451 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f7c7d667a0 con 0x55f7c7de4c60 2024-03-24T09:28:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa063fff640 1 -- 172.21.15.116:0/2529697451 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa07c0054a0 con 0x55f7c7de4c60 2024-03-24T09:28:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 10 client.27257 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 10 client.27257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:11.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 20 client.27257 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7df4f00 0x55f7c7df7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 4 client.27257 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7c7d381f0 con 0x55f7c7df4f00 2024-03-24T09:28:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08889e640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7df4f00 0x55f7c7df7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa08889e640 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7df4f00 0x55f7c7df7320 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2796fa35b3af24e server_cookie=518e81ee7faff4b2 in_seq=0 out_seq=0 2024-03-24T09:28:11.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.371+0000 7fa063fff640 10 client.27257 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.372+0000 7fa063fff640 1 -- 172.21.15.116:0/2529697451 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f7c7d381f0 con 0x55f7c7df4f00 2024-03-24T09:28:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.372+0000 7fa063fff640 10 client.27257 handle_command_reply: tid=0 2024-03-24T09:28:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.372+0000 7fa08d7c2740 1 client.27257 shutdown 2024-03-24T09:28:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 msgr2=0x55f7c7de2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7de2710 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d60 tx=0x7fa07c004ab0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 shutdown_connections 2024-03-24T09:28:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7df4f00 0x55f7c7df7320 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7de2710 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2529697451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7de1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa063fff640 0 client.27257 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 >> 172.21.15.116:0/2529697451 conn(0x55f7c7dd5e90 msgr2=0x55f7c7dd7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 shutdown_connections 2024-03-24T09:28:11.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2529697451 wait complete. 2024-03-24T09:28:11.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.373+0000 7fa08d7c2740 20 client.27257 trim_cache size 0 max 16384 2024-03-24T09:28:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 Processor -- start 2024-03-24T09:28:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 -- start start 2024-03-24T09:28:11.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7df0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7dfcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7dd81a0 con 0x55f7c7ddc080 2024-03-24T09:28:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7dd5600 con 0x55f7c7de4c60 2024-03-24T09:28:11.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7df7980 con 0x55f7c7ddb760 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa083fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa083fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37572/0 (socket says 172.21.15.116:37572) 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa083fff640 1 -- 172.21.15.116:0/396703480 learned_addr learned my addr 172.21.15.116:0/396703480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08889e640 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7df0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.390+0000 7fa08909f640 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7dfcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa083fff640 1 -- 172.21.15.116:0/396703480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 msgr2=0x55f7c7df0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa083fff640 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7df0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa083fff640 1 -- 172.21.15.116:0/396703480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 msgr2=0x55f7c7dfcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa083fff640 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7dfcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa083fff640 1 -- 172.21.15.116:0/396703480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7c7cdcc60 con 0x55f7c7ddc080 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa08909f640 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7dfcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa083fff640 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de2530 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa070036040 tx=0x7fa070004b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4300f7dddd1f49da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa06e7fc640 1 -- 172.21.15.116:0/396703480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07002c7f0 con 0x55f7c7ddc080 2024-03-24T09:28:11.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa06e7fc640 1 -- 172.21.15.116:0/396703480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa07002de90 con 0x55f7c7ddc080 2024-03-24T09:28:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.391+0000 7fa06e7fc640 1 -- 172.21.15.116:0/396703480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700039f0 con 0x55f7c7ddc080 2024-03-24T09:28:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- 172.21.15.116:0/396703480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 msgr2=0x55f7c7de2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de2530 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa070036040 tx=0x7fa070004b10 comp rx=0 tx=0).stop 2024-03-24T09:28:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- 172.21.15.116:0/396703480 shutdown_connections 2024-03-24T09:28:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7de4c60 0x55f7c7dfcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7ddc080 0x55f7c7de2530 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/396703480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7df0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- 172.21.15.116:0/396703480 >> 172.21.15.116:0/396703480 conn(0x55f7c7dd5e90 msgr2=0x55f7c7dd4ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- 172.21.15.116:0/396703480 shutdown_connections 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- 172.21.15.116:0/396703480 wait complete. 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- start start 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 Processor -- start 2024-03-24T09:28:11.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7decb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 0x55f7c7de9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 0x55f7c7dea090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7ded340 con 0x55f7c7de4c60 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7dea5d0 con 0x55f7c7ddc080 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f7c7dea790 con 0x55f7c7ddb760 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.392+0000 7fa08d7c2740 10 client.0 _ll_register_callbacks cb 0x7fa07c01da70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:11.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 0x55f7c7dea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa083fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 0x55f7c7de9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 0x55f7c7dea090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37580/0 (socket says 172.21.15.116:37580) 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa083fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 0x55f7c7de9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59312/0 (socket says 172.21.15.116:59312) 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 -- 172.21.15.116:0/2668927930 learned_addr learned my addr 172.21.15.116:0/2668927930 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08889e640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7decb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 -- 172.21.15.116:0/2668927930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 msgr2=0x55f7c7decb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7decb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 -- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 msgr2=0x55f7c7de9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 0x55f7c7de9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 -- 172.21.15.116:0/2668927930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7c7cf86d0 con 0x55f7c7de4c60 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa083fff640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 0x55f7c7de9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08909f640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 0x55f7c7dea090 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009db0 tx=0x7fa07c027650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3912fabb35cb2a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa063fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:11.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa063fff640 1 -- 172.21.15.116:0/2668927930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c022040 con 0x55f7c7de4c60 2024-03-24T09:28:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa063fff640 1 -- 172.21.15.116:0/2668927930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fa07c00f3c0 con 0x55f7c7de4c60 2024-03-24T09:28:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa07809ada0 con 0x55f7c7de4c60 2024-03-24T09:28:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa063fff640 1 -- 172.21.15.116:0/2668927930 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa07c00fe00 con 0x55f7c7de4c60 2024-03-24T09:28:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08d7c2740 10 client.17439 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.393+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7fa070030070 con 0x55f7c7de4c60 2024-03-24T09:28:11.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa063fff640 1 -- 172.21.15.116:0/2668927930 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa07809ada0 con 0x55f7c7de4c60 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa063fff640 1 -- 172.21.15.116:0/2668927930 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa07c0338e0 con 0x55f7c7de4c60 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08d7c2740 10 client.17439 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08d7c2740 10 client.17439 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08d7c2740 20 client.17439 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7e5c5f0 0x55f7c7e5ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08d7c2740 4 client.17439 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:28:11.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f7c7a41860 con 0x55f7c7e5c5f0 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08889e640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7e5c5f0 0x55f7c7e5ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa063fff640 10 client.17439 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.394+0000 7fa08889e640 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7e5c5f0 0x55f7c7e5ea10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=182806bf77064c49 server_cookie=31a5242f5b417a74 in_seq=0 out_seq=0 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa063fff640 1 -- 172.21.15.116:0/2668927930 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f7c7a41860 con 0x55f7c7e5c5f0 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa063fff640 10 client.17439 handle_command_reply: tid=0 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 client.17439 shutdown 2024-03-24T09:28:11.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 msgr2=0x55f7c7dea090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 0x55f7c7dea090 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009db0 tx=0x7fa07c027650 comp rx=0 tx=0).stop 2024-03-24T09:28:11.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 shutdown_connections 2024-03-24T09:28:11.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f7c7de4c60 0x55f7c7dea090 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f7c7e5c5f0 0x55f7c7e5ea10 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f7c7ddc080 0x55f7c7de9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 --2- 172.21.15.116:0/2668927930 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f7c7ddb760 0x55f7c7decb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa063fff640 0 client.17439 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.395+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 >> 172.21.15.116:0/2668927930 conn(0x55f7c7dd5e90 msgr2=0x55f7c7dd7120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 shutdown_connections 2024-03-24T09:28:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08d7c2740 1 -- 172.21.15.116:0/2668927930 wait complete. 2024-03-24T09:28:11.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08d7c2740 20 client.17439 trim_cache size 0 max 16384 2024-03-24T09:28:11.414 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:28:11.415 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:28:11.415 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "3f1c457d-cfa5-4619-878c-f87991d2ee8d", 2024-03-24T09:28:11.415 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:28:11.415 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0740a2230 msgr2=0x7fa0740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0740a2230 0x7fa0740a46f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa07c005fd0 tx=0x7fa07c011810 comp rx=0 tx=0).stop 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 msgr2=0x7fa08419d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa08419d900 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa07800e9f0 tx=0x7fa07800edf0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 shutdown_connections 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa0740a2230 0x7fa0740a46f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa0841076b0 0x7fa084197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa084102060 0x7fa08419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 --2- 172.21.15.116:0/2642001706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0841016a0 0x7fa08419d900 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 >> 172.21.15.116:0/2642001706 conn(0x7fa084078b10 msgr2=0x7fa0841041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 shutdown_connections 2024-03-24T09:28:11.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.396+0000 7fa08ab02640 1 -- 172.21.15.116:0/2642001706 wait complete. 2024-03-24T09:28:11.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:28:11.419 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:28:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:11 smithi143 ceph-mon[35965]: pgmap v185: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-24T09:28:11.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 -- 172.21.15.116:0/1779449101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 msgr2=0x7f82dc107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 --2- 172.21.15.116:0/1779449101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc107a90 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009930 tx=0x7f82d802bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 -- 172.21.15.116:0/1779449101 shutdown_connections 2024-03-24T09:28:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 --2- 172.21.15.116:0/1779449101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82dc102060 0x7f82dc10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 --2- 172.21.15.116:0/1779449101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82dc1016a0 0x7f82dc101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 --2- 172.21.15.116:0/1779449101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc107a90 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.539+0000 7f82e399a640 1 -- 172.21.15.116:0/1779449101 >> 172.21.15.116:0/1779449101 conn(0x7f82dc078ad0 msgr2=0x7f82dc078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.540+0000 7f82e399a640 1 -- 172.21.15.116:0/1779449101 shutdown_connections 2024-03-24T09:28:11.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.540+0000 7f82e399a640 1 -- 172.21.15.116:0/1779449101 wait complete. 2024-03-24T09:28:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.540+0000 7f82e399a640 1 Processor -- start 2024-03-24T09:28:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.540+0000 7f82e399a640 1 -- start start 2024-03-24T09:28:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e399a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82dc1016a0 0x7f82dc10afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e399a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82dc102060 0x7f82dc10b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e399a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc1a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e399a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1a44c0 con 0x7f82dc1076b0 2024-03-24T09:28:11.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e399a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1a4630 con 0x7f82dc102060 2024-03-24T09:28:11.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e399a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f82dc1a47f0 con 0x7f82dc1016a0 2024-03-24T09:28:11.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc1a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc1a3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37598/0 (socket says 172.21.15.116:37598) 2024-03-24T09:28:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82dc102060 0x7f82dc10b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 -- 172.21.15.116:0/1674085010 learned_addr learned my addr 172.21.15.116:0/1674085010 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1736640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82dc1016a0 0x7f82dc10afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 -- 172.21.15.116:0/1674085010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82dc1016a0 msgr2=0x7f82dc10afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82dc1016a0 0x7f82dc10afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 -- 172.21.15.116:0/1674085010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82dc102060 msgr2=0x7f82dc10b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82dc102060 0x7f82dc10b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 -- 172.21.15.116:0/1674085010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d8009560 con 0x7f82dc1076b0 2024-03-24T09:28:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.541+0000 7f82e1f37640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc1a3ec0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f82d400eb10 tx=0x7f82d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.542+0000 7f82d27fc640 1 -- 172.21.15.116:0/1674085010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4004b20 con 0x7f82dc1076b0 2024-03-24T09:28:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.542+0000 7f82d27fc640 1 -- 172.21.15.116:0/1674085010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82d4004c80 con 0x7f82dc1076b0 2024-03-24T09:28:11.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.542+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82dc1a4ad0 con 0x7f82dc1076b0 2024-03-24T09:28:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.542+0000 7f82d27fc640 1 -- 172.21.15.116:0/1674085010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d4010690 con 0x7f82dc1076b0 2024-03-24T09:28:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.542+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc1ab150 con 0x7f82dc1076b0 2024-03-24T09:28:11.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.544+0000 7f82d27fc640 1 -- 172.21.15.116:0/1674085010 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f82d4005020 con 0x7f82dc1076b0 2024-03-24T09:28:11.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 Processor -- start 2024-03-24T09:28:11.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 -- start start 2024-03-24T09:28:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460fa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460fcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565546015f30 con 0x5655460fa6d0 2024-03-24T09:28:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56554605f810 con 0x5655460f11d0 2024-03-24T09:28:11.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82e657f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56554605cd40 con 0x5655460f1af0 2024-03-24T09:28:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.545+0000 7f82d27fc640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f82c80a22b0 0x7f82c80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59320/0 (socket says 172.21.15.116:59320) 2024-03-24T09:28:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 -- 172.21.15.116:0/206344190 learned_addr learned my addr 172.21.15.116:0/206344190 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460fcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e1f37640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f82c80a22b0 0x7f82c80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e1736640 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82d27fc640 1 -- 172.21.15.116:0/1674085010 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f82d4014070 con 0x7f82dc1076b0 2024-03-24T09:28:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 -- 172.21.15.116:0/206344190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 msgr2=0x5655460fa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460fa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 -- 172.21.15.116:0/206344190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 msgr2=0x5655460f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.546+0000 7f82e0f35640 1 -- 172.21.15.116:0/206344190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565546019ea0 con 0x5655460fa6d0 2024-03-24T09:28:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.547+0000 7f82e1f37640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f82c80a22b0 0x7f82c80a4770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f82d4097410 tx=0x7f82d409ad80 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.547+0000 7f82e0f35640 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460fcac0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0058c0 tx=0x7f82cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b9a3937a6182f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.547+0000 7f82c77fe640 1 -- 172.21.15.116:0/206344190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc011e30 con 0x5655460fa6d0 2024-03-24T09:28:11.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.547+0000 7f82c77fe640 1 -- 172.21.15.116:0/206344190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82cc0044e0 con 0x5655460fa6d0 2024-03-24T09:28:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.547+0000 7f82c77fe640 1 -- 172.21.15.116:0/206344190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc004a00 con 0x5655460fa6d0 2024-03-24T09:28:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 -- 172.21.15.116:0/206344190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 msgr2=0x5655460fcac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460fcac0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f82cc0058c0 tx=0x7f82cc005f90 comp rx=0 tx=0).stop 2024-03-24T09:28:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 -- 172.21.15.116:0/206344190 shutdown_connections 2024-03-24T09:28:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460fcac0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460fa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 --2- 172.21.15.116:0/206344190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 -- 172.21.15.116:0/206344190 >> 172.21.15.116:0/206344190 conn(0x5655460eb900 msgr2=0x5655460ebce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 -- 172.21.15.116:0/206344190 shutdown_connections 2024-03-24T09:28:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 -- 172.21.15.116:0/206344190 wait complete. 2024-03-24T09:28:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.548+0000 7f82e657f740 1 -- start start 2024-03-24T09:28:11.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 Processor -- start 2024-03-24T09:28:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460f8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565546058520 con 0x5655460fa6d0 2024-03-24T09:28:11.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460f8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460f8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37608/0 (socket says 172.21.15.116:37608) 2024-03-24T09:28:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e0f35640 1 -- 172.21.15.116:0/1362344781 learned_addr learned my addr 172.21.15.116:0/1362344781 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x565545d2b430 con 0x5655460f11d0 2024-03-24T09:28:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x565545ce7e00 con 0x5655460f1af0 2024-03-24T09:28:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.549+0000 7f82e1736640 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e657f740 10 client.0 _ll_register_callbacks cb 0x565545ff6d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e0f35640 1 -- 172.21.15.116:0/1362344781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 msgr2=0x5655460f7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e0f35640 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e0f35640 1 -- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 msgr2=0x5655460f77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e0f35640 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e0f35640 1 -- 172.21.15.116:0/1362344781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82cc019040 con 0x5655460fa6d0 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e0f35640 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460f8220 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009730 tx=0x7f82cc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842e1f29cf4b1b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82c4ff9640 1 -- 172.21.15.116:0/1362344781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc022070 con 0x5655460fa6d0 2024-03-24T09:28:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82c4ff9640 1 -- 172.21.15.116:0/1362344781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f82cc00f790 con 0x5655460fa6d0 2024-03-24T09:28:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x565545d577e0 con 0x5655460fa6d0 2024-03-24T09:28:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.550+0000 7f82c4ff9640 1 -- 172.21.15.116:0/1362344781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f82cc00f970 con 0x5655460fa6d0 2024-03-24T09:28:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 10 client.17451 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:11.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x565546056ba0 con 0x5655460fa6d0 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82c4ff9640 1 -- 172.21.15.116:0/1362344781 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x565545d577e0 con 0x5655460fa6d0 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82c4ff9640 1 -- 172.21.15.116:0/1362344781 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f82cc004bf0 con 0x5655460fa6d0 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 10 client.17451 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 10 client.17451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 20 client.17451 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56554611bb20 0x56554611df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 4 client.17451 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565545d94720 con 0x56554611bb20 2024-03-24T09:28:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.551+0000 7f82e1736640 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56554611bb20 0x56554611df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.552+0000 7f82e1736640 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56554611bb20 0x56554611df10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f77ea52d59493b3 server_cookie=6c768fbc05d477d8 in_seq=0 out_seq=0 2024-03-24T09:28:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.552+0000 7f82c4ff9640 10 client.17451 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.553+0000 7f82c4ff9640 1 -- 172.21.15.116:0/1362344781 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565545d94720 con 0x56554611bb20 2024-03-24T09:28:11.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.553+0000 7f82c4ff9640 10 client.17451 handle_command_reply: tid=0 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.553+0000 7f82e657f740 1 client.17451 shutdown 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.553+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 msgr2=0x5655460f8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.553+0000 7f82e657f740 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460f8220 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009730 tx=0x7f82cc005850 comp rx=0 tx=0).stop 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 shutdown_connections 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x5655460f8220 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56554611bb20 0x56554611df10 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 --2- 172.21.15.116:0/1362344781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 >> 172.21.15.116:0/1362344781 conn(0x5655460eb900 msgr2=0x5655460eab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82c4ff9640 0 client.17451 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 shutdown_connections 2024-03-24T09:28:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 1 -- 172.21.15.116:0/1362344781 wait complete. 2024-03-24T09:28:11.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.554+0000 7f82e657f740 20 client.17451 trim_cache size 0 max 16384 2024-03-24T09:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 Processor -- start 2024-03-24T09:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 -- start start 2024-03-24T09:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x565546116f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x565545fdd430 con 0x5655460fa6d0 2024-03-24T09:28:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x565545d57810 con 0x5655460f11d0 2024-03-24T09:28:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e657f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x565546118640 con 0x5655460f1af0 2024-03-24T09:28:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x565546116f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e1736640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e1736640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:45528/0 (socket says 172.21.15.116:45528) 2024-03-24T09:28:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59348/0 (socket says 172.21.15.116:59348) 2024-03-24T09:28:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.578+0000 7f82e1736640 1 -- 172.21.15.116:0/1357582821 learned_addr learned my addr 172.21.15.116:0/1357582821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e0f35640 1 -- 172.21.15.116:0/1357582821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 msgr2=0x5655460f6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e0f35640 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e0f35640 1 -- 172.21.15.116:0/1357582821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 msgr2=0x5655460f9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e0f35640 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e0f35640 1 -- 172.21.15.116:0/1357582821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565545ff2b20 con 0x5655460fa6d0 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e1736640 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82e0f35640 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x565546116f70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f82cc001eb0 tx=0x7f82cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5242616b3f0b868d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82c6ffd640 1 -- 172.21.15.116:0/1357582821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc022070 con 0x5655460fa6d0 2024-03-24T09:28:11.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82c6ffd640 1 -- 172.21.15.116:0/1357582821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82cc023040 con 0x5655460fa6d0 2024-03-24T09:28:11.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.579+0000 7f82c6ffd640 1 -- 172.21.15.116:0/1357582821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc005bb0 con 0x5655460fa6d0 2024-03-24T09:28:11.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 -- 172.21.15.116:0/1357582821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 msgr2=0x565546116f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x565546116f70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f82cc001eb0 tx=0x7f82cc005f90 comp rx=0 tx=0).stop 2024-03-24T09:28:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 -- 172.21.15.116:0/1357582821 shutdown_connections 2024-03-24T09:28:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x565546116f70 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x5655460f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 --2- 172.21.15.116:0/1357582821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655460f9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 -- 172.21.15.116:0/1357582821 >> 172.21.15.116:0/1357582821 conn(0x5655460eb900 msgr2=0x5655460ea2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 -- 172.21.15.116:0/1357582821 shutdown_connections 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.580+0000 7f82e657f740 1 -- 172.21.15.116:0/1357582821 wait complete. 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 -- start start 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 Processor -- start 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655461138a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x56554610ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x56554610f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56554610f910 con 0x5655460fa6d0 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56554610fa80 con 0x5655460f11d0 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x565546180e20 con 0x5655460f1af0 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e657f740 10 client.0 _ll_register_callbacks cb 0x7f82cc009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x56554610f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e0f35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x56554610f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37624/0 (socket says 172.21.15.116:37624) 2024-03-24T09:28:11.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e0f35640 1 -- 172.21.15.116:0/3342533777 learned_addr learned my addr 172.21.15.116:0/3342533777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e0f35640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655461138a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.581+0000 7f82e1736640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x56554610ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e0f35640 1 -- 172.21.15.116:0/3342533777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 msgr2=0x56554610ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e0f35640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x56554610ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e0f35640 1 -- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 msgr2=0x5655461138a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e0f35640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655461138a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e0f35640 1 -- 172.21.15.116:0/3342533777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82dc10ee70 con 0x5655460fa6d0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e0f35640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x56554610f1c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009800 tx=0x7f82cc013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b313fbfd7dbc54a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82c4ff9640 1 -- 172.21.15.116:0/3342533777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82cc022070 con 0x5655460fa6d0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82c4ff9640 1 -- 172.21.15.116:0/3342533777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f82cc000e40 con 0x5655460fa6d0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5655460f7340 con 0x5655460fa6d0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82c4ff9640 1 -- 172.21.15.116:0/3342533777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f82cc009ea0 con 0x5655460fa6d0 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e657f740 10 client.17457 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:11.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x565546056ba0 con 0x5655460fa6d0 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.582+0000 7f82c4ff9640 1 -- 172.21.15.116:0/3342533777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f82cc0300b0 con 0x5655460fa6d0 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82c4ff9640 1 -- 172.21.15.116:0/3342533777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f82cc026070 con 0x5655460fa6d0 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e657f740 10 client.17457 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e657f740 10 client.17457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e657f740 20 client.17457 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e657f740 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5655461821a0 0x565546184590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:11.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e657f740 4 client.17457 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5655460f7340 con 0x5655461821a0 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e1736640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5655461821a0 0x565546184590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82e1736640 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5655461821a0 0x565546184590 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4abab2012567e0 server_cookie=5c35e63fbc68b3af in_seq=0 out_seq=0 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.583+0000 7f82c4ff9640 10 client.17457 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82c4ff9640 1 -- 172.21.15.116:0/3342533777 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5655460f7340 con 0x5655461821a0 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82c4ff9640 10 client.17457 handle_command_reply: tid=0 2024-03-24T09:28:11.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 client.17457 shutdown 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 msgr2=0x56554610f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x56554610f1c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f82cc009800 tx=0x7f82cc013040 comp rx=0 tx=0).stop 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 shutdown_connections 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5655460fa6d0 0x56554610f1c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5655460f1af0 0x56554610ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5655461821a0 0x565546184590 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 --2- 172.21.15.116:0/3342533777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5655460f11d0 0x5655461138a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82c4ff9640 0 client.17457 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.584+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 >> 172.21.15.116:0/3342533777 conn(0x5655460eb900 msgr2=0x5655460eab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 shutdown_connections 2024-03-24T09:28:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e657f740 1 -- 172.21.15.116:0/3342533777 wait complete. 2024-03-24T09:28:11.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e657f740 20 client.17457 trim_cache size 0 max 16384 2024-03-24T09:28:11.600 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:28:11.601 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:28:11.601 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:28:11.601 INFO:teuthology.orchestra.run.smithi116.stdout: "3f1c457d-cfa5-4619-878c-f87991d2ee8d": { 2024-03-24T09:28:11.601 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:28:11.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "3f1c457d-cfa5-4619-878c-f87991d2ee8d", 2024-03-24T09:28:11.601 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:28:11.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:28:11.602 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:28:11.602 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:28:11.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f82c80a22b0 msgr2=0x7f82c80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e399a640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f82c80a22b0 0x7f82c80a4770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f82d4097410 tx=0x7f82d409ad80 comp rx=0 tx=0).stop 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 msgr2=0x7f82dc1a3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e399a640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc1a3ec0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f82d400eb10 tx=0x7f82d400c2d0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.585+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 shutdown_connections 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82dc1076b0 0x7f82dc1a3ec0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82dc102060 0x7f82dc10b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f82c80a22b0 0x7f82c80a4770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 --2- 172.21.15.116:0/1674085010 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f82dc1016a0 0x7f82dc10afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:11.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 >> 172.21.15.116:0/1674085010 conn(0x7f82dc078ad0 msgr2=0x7f82dc103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:11.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 shutdown_connections 2024-03-24T09:28:11.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:11.586+0000 7f82e399a640 1 -- 172.21.15.116:0/1674085010 wait complete. 2024-03-24T09:28:11.605 INFO:tasks.cephfs.filesystem:scrub status for tag:3f1c457d-cfa5-4619-878c-f87991d2ee8d - {'path': '/', 'tag': '3f1c457d-cfa5-4619-878c-f87991d2ee8d', 'options': 'recursive,force'} 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:12 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:28:12.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:12.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:12.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:12.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:12.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:12 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:28:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:12.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:12.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:12.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:12.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:12.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:12 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: pgmap v186: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:13.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:13.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:13.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:13 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: pgmap v186: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:13.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:13 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: pgmap v186: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-24T09:28:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:13 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:15 smithi117 ceph-mon[35788]: pgmap v187: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:28:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:15 smithi116 ceph-mon[32207]: pgmap v187: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:28:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:15 smithi143 ceph-mon[35965]: pgmap v187: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:28:16.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:16 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:28:16.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:16 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:28:16.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:16 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:28:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:17 smithi117 ceph-mon[35788]: pgmap v188: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:28:17.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:17 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:28:17.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:17 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:28:17.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:17 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:28:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:17 smithi116 ceph-mon[32207]: pgmap v188: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:28:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:17 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:28:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:17 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:28:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:17 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:28:17.425 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:17 smithi143 ceph-mon[35965]: pgmap v188: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-24T09:28:17.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:17 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:28:17.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:17 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:28:17.426 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:17 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:28:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:19 smithi117 ceph-mon[35788]: pgmap v189: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:28:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:19 smithi116 ceph-mon[32207]: pgmap v189: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:28:19.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:19 smithi143 ceph-mon[35965]: pgmap v189: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-24T09:28:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:21 smithi117 ceph-mon[35788]: pgmap v190: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:28:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:21 smithi116 ceph-mon[32207]: pgmap v190: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:28:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:21 smithi143 ceph-mon[35965]: pgmap v190: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T09:28:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:23 smithi117 ceph-mon[35788]: pgmap v191: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:28:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:23 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:23 smithi116 ceph-mon[32207]: pgmap v191: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:28:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:23 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:23 smithi143 ceph-mon[35965]: pgmap v191: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:28:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:23 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:24.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:24 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:24 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:24 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:24 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:24 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:24 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:24.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:24 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:24.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:24 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:24 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:24 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:24 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:24 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:25.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:25 smithi117 ceph-mon[35788]: pgmap v192: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:25 smithi116 ceph-mon[32207]: pgmap v192: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:25 smithi143 ceph-mon[35965]: pgmap v192: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:27 smithi116 ceph-mon[32207]: pgmap v193: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:27 smithi143 ceph-mon[35965]: pgmap v193: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:27.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:27 smithi117 ceph-mon[35788]: pgmap v193: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:29.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:29 smithi143 ceph-mon[35965]: pgmap v194: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:29.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:29 smithi117 ceph-mon[35788]: pgmap v194: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:29.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:29 smithi116 ceph-mon[32207]: pgmap v194: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:31.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:31 smithi143 ceph-mon[35965]: pgmap v195: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:31.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:31 smithi117 ceph-mon[35788]: pgmap v195: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:31.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:31 smithi116 ceph-mon[32207]: pgmap v195: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:33.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:33 smithi116 ceph-mon[32207]: pgmap v196: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:33 smithi143 ceph-mon[35965]: pgmap v196: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:33.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:33 smithi117 ceph-mon[35788]: pgmap v196: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:35.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:35 smithi143 ceph-mon[35965]: pgmap v197: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:35.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:35 smithi117 ceph-mon[35788]: pgmap v197: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:35.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:35 smithi116 ceph-mon[32207]: pgmap v197: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:37 smithi143 ceph-mon[35965]: pgmap v198: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:37.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:37 smithi117 ceph-mon[35788]: pgmap v198: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:37.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:37 smithi116 ceph-mon[32207]: pgmap v198: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:39.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:39 smithi143 ceph-mon[35965]: pgmap v199: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:39 smithi117 ceph-mon[35788]: pgmap v199: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:39.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:39 smithi116 ceph-mon[32207]: pgmap v199: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:41 smithi143 ceph-mon[35965]: pgmap v200: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:41 smithi117 ceph-mon[35788]: pgmap v200: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:41.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:41 smithi116 ceph-mon[32207]: pgmap v200: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:41.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:28:41.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:28:41.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 -- 172.21.15.116:0/4125619577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc8107910 msgr2=0x7f4dc8107cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 --2- 172.21.15.116:0/4125619577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc8107910 0x7f4dc8107cf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4db8009860 tx=0x7f4db802bb70 comp rx=0 tx=0).stop 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 -- 172.21.15.116:0/4125619577 shutdown_connections 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 --2- 172.21.15.116:0/4125619577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8100990 0x7f4dc810e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 --2- 172.21.15.116:0/4125619577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc80fffd0 0x7f4dc8100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 --2- 172.21.15.116:0/4125619577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc8107910 0x7f4dc8107cf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 -- 172.21.15.116:0/4125619577 >> 172.21.15.116:0/4125619577 conn(0x7f4dc8075950 msgr2=0x7f4dc8075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 -- 172.21.15.116:0/4125619577 shutdown_connections 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.721+0000 7f4dcd083640 1 -- 172.21.15.116:0/4125619577 wait complete. 2024-03-24T09:28:41.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 Processor -- start 2024-03-24T09:28:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 -- start start 2024-03-24T09:28:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc80fffd0 0x7f4dc819d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 0x7f4dc819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8107910 0x7f4dc8197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc81980e0 con 0x7f4dc8100990 2024-03-24T09:28:41.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8198250 con 0x7f4dc80fffd0 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.722+0000 7f4dcd083640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc8198410 con 0x7f4dc8107910 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 0x7f4dc819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 0x7f4dc819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49268/0 (socket says 172.21.15.116:49268) 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc6d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc80fffd0 0x7f4dc819d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 -- 172.21.15.116:0/905299986 learned_addr learned my addr 172.21.15.116:0/905299986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc759e640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8107910 0x7f4dc8197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 -- 172.21.15.116:0/905299986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8107910 msgr2=0x7f4dc8197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8107910 0x7f4dc8197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 -- 172.21.15.116:0/905299986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc80fffd0 msgr2=0x7f4dc819d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc80fffd0 0x7f4dc819d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 -- 172.21.15.116:0/905299986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dbc009560 con 0x7f4dc8100990 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc80fffd0 0x7f4dc819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc759e640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8107910 0x7f4dc8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dc659c640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 0x7f4dc819de20 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00e9c0 tx=0x7f4dbc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4daffff640 1 -- 172.21.15.116:0/905299986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc00cde0 con 0x7f4dc8100990 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db8009490 con 0x7f4dc8100990 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4daffff640 1 -- 172.21.15.116:0/905299986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4dbc0044d0 con 0x7f4dc8100990 2024-03-24T09:28:41.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4daffff640 1 -- 172.21.15.116:0/905299986 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc010430 con 0x7f4dc8100990 2024-03-24T09:28:41.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.723+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc81a47a0 con 0x7f4dc8100990 2024-03-24T09:28:41.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.725+0000 7f4daffff640 1 -- 172.21.15.116:0/905299986 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f4dbc004000 con 0x7f4dc8100990 2024-03-24T09:28:41.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 Processor -- start 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 -- start start 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4daffff640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4db00a2230 0x7f4db00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 0x5606369c6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c67a0 0x5606369d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5606368eaf30 con 0x5606369c5e80 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560636934810 con 0x5606369c67a0 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560636931d40 con 0x5606369d3ae0 2024-03-24T09:28:41.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dc759e640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4db00a2230 0x7f4db00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4daffff640 1 -- 172.21.15.116:0/905299986 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4dbc014070 con 0x7f4dc8100990 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dc6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 0x5606369c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dc6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 0x5606369c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49274/0 (socket says 172.21.15.116:49274) 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dc6d9d640 1 -- 172.21.15.116:0/919320722 learned_addr learned my addr 172.21.15.116:0/919320722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dc759e640 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.726+0000 7f4dc659c640 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c67a0 0x5606369d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc659c640 1 -- 172.21.15.116:0/919320722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 msgr2=0x5606369d5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc659c640 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc659c640 1 -- 172.21.15.116:0/919320722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 msgr2=0x5606369c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc659c640 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 0x5606369c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc659c640 1 -- 172.21.15.116:0/919320722 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606368eeea0 con 0x5606369c67a0 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 0x5606369c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc759e640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4db00a2230 0x7f4db00a46f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00099e0 tx=0x7f4dc000fef0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dc659c640 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c67a0 0x5606369d35a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0cbbc0 tx=0x7f4dbc09acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab137cbc6caa3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dad7fa640 1 -- 172.21.15.116:0/919320722 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc09b560 con 0x5606369c67a0 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dad7fa640 1 -- 172.21.15.116:0/919320722 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4dbc09ae60 con 0x5606369c67a0 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.727+0000 7f4dad7fa640 1 -- 172.21.15.116:0/919320722 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0cbfa0 con 0x5606369c67a0 2024-03-24T09:28:41.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/919320722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c67a0 msgr2=0x5606369d35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c67a0 0x5606369d35a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0cbbc0 tx=0x7f4dbc09acf0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/919320722 shutdown_connections 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c67a0 0x5606369d35a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/919320722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c5e80 0x5606369c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/919320722 >> 172.21.15.116:0/919320722 conn(0x5606369c0900 msgr2=0x5606369c0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/919320722 shutdown_connections 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/919320722 wait complete. 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- start start 2024-03-24T09:28:41.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 Processor -- start 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369d10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56063692d520 con 0x5606369c67a0 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560636600430 con 0x5606369c5e80 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5606365bce00 con 0x5606369d3ae0 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dcfc5c740 10 client.0 _ll_register_callbacks cb 0x5606368cbd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dc6d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.728+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369d10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369d10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49288/0 (socket says 172.21.15.116:49288) 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 -- 172.21.15.116:0/28563132 learned_addr learned my addr 172.21.15.116:0/28563132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc759e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 -- 172.21.15.116:0/28563132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 msgr2=0x5606369d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 -- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 msgr2=0x5606369d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 -- 172.21.15.116:0/28563132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc81a47a0 con 0x5606369c67a0 2024-03-24T09:28:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dc659c640 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369d10b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0cbc70 tx=0x7f4dbc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3fc6d6773199ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4daaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4daaff5640 1 -- 172.21.15.116:0/28563132 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc09efb0 con 0x5606369c67a0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4daaff5640 1 -- 172.21.15.116:0/28563132 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4dbc09d030 con 0x5606369c67a0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56063662c7e0 con 0x5606369c67a0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4daaff5640 1 -- 172.21.15.116:0/28563132 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4dbc0a3600 con 0x5606369c67a0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dcfc5c740 10 client.17469 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56063692bba0 con 0x5606369c67a0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.729+0000 7f4daaff5640 1 -- 172.21.15.116:0/28563132 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56063662c7e0 con 0x5606369c67a0 2024-03-24T09:28:41.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4daaff5640 1 -- 172.21.15.116:0/28563132 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4dbc01f060 con 0x5606369c67a0 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dcfc5c740 10 client.17469 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dcfc5c740 10 client.17469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dcfc5c740 20 client.17469 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5606369e3b30 0x5606369e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dcfc5c740 4 client.17469 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560636669720 con 0x5606369e3b30 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5606369e3b30 0x5606369e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4daaff5640 10 client.17469 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.730+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5606369e3b30 0x5606369e5f50 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1925e8ff2b3073ce server_cookie=ff3f443028325f7a in_seq=0 out_seq=0 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4daaff5640 1 -- 172.21.15.116:0/28563132 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560636669720 con 0x5606369e3b30 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4daaff5640 10 client.17469 handle_command_reply: tid=0 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 client.17469 shutdown 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 msgr2=0x5606369d10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369d10b0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc0cbc70 tx=0x7f4dbc003820 comp rx=0 tx=0).stop 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 shutdown_connections 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369d3ae0 0x5606369d15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369d10b0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5606369e3b30 0x5606369e5f50 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/28563132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4daaff5640 0 client.17469 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 >> 172.21.15.116:0/28563132 conn(0x5606369c0900 msgr2=0x5606369c2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 shutdown_connections 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/28563132 wait complete. 2024-03-24T09:28:41.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.731+0000 7f4dcfc5c740 20 client.17469 trim_cache size 0 max 16384 2024-03-24T09:28:41.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 Processor -- start 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 -- start start 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c5e80 0x5606369d32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369d3ae0 0x5606369df340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5606368b2430 con 0x5606369c67a0 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5606369c1a30 con 0x5606369d3ae0 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.748+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5606369b24d0 con 0x5606369c5e80 2024-03-24T09:28:41.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369c0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49292/0 (socket says 172.21.15.116:49292) 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 -- 172.21.15.116:0/2102997996 learned_addr learned my addr 172.21.15.116:0/2102997996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc6d9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c5e80 0x5606369d32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc759e640 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369d3ae0 0x5606369df340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 -- 172.21.15.116:0/2102997996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c5e80 msgr2=0x5606369d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c5e80 0x5606369d32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 -- 172.21.15.116:0/2102997996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369d3ae0 msgr2=0x5606369df340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369d3ae0 0x5606369df340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 -- 172.21.15.116:0/2102997996 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606368c7b20 con 0x5606369c67a0 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dc659c640 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369c0ce0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009530 tx=0x7f4dbc09acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49328238013d9dde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dacff9640 1 -- 172.21.15.116:0/2102997996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc09efb0 con 0x5606369c67a0 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dacff9640 1 -- 172.21.15.116:0/2102997996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4dbc0a3600 con 0x5606369c67a0 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.749+0000 7f4dacff9640 1 -- 172.21.15.116:0/2102997996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc0840d0 con 0x5606369c67a0 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2102997996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 msgr2=0x5606369c0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369c0ce0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009530 tx=0x7f4dbc09acf0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2102997996 shutdown_connections 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369d3ae0 0x5606369df340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369c67a0 0x5606369c0ce0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2102997996 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c5e80 0x5606369d32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2102997996 >> 172.21.15.116:0/2102997996 conn(0x5606369c0900 msgr2=0x5606369c1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2102997996 shutdown_connections 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2102997996 wait complete. 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- start start 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 Processor -- start 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c67a0 0x5606369ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 0x560636a48e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5606369ceab0 con 0x5606369d3ae0 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560636a493d0 con 0x5606369c5e80 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560636a49560 con 0x5606369c67a0 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dcfc5c740 10 client.0 _ll_register_callbacks cb 0x7f4dbc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.750+0000 7f4dc759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 0x560636a48e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 0x560636a48e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49306/0 (socket says 172.21.15.116:49306) 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc659c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c67a0 0x5606369ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 -- 172.21.15.116:0/2311279170 learned_addr learned my addr 172.21.15.116:0/2311279170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:41.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 -- 172.21.15.116:0/2311279170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c67a0 msgr2=0x5606369ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c67a0 0x5606369ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 -- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 msgr2=0x5606369cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 -- 172.21.15.116:0/2311279170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56063692bba0 con 0x5606369d3ae0 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc659c640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c67a0 0x5606369ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dc759e640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 0x560636a48e90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0020660 tx=0x7f4dc0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dfc03da60f0f21 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4daaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4daaff5640 1 -- 172.21.15.116:0/2311279170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0018040 con 0x5606369d3ae0 2024-03-24T09:28:41.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4daaff5640 1 -- 172.21.15.116:0/2311279170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f4dc0010640 con 0x5606369d3ae0 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dbc09ae60 con 0x5606369d3ae0 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4daaff5640 1 -- 172.21.15.116:0/2311279170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4dc0010800 con 0x5606369d3ae0 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dcfc5c740 10 client.17475 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5606368ed700 con 0x5606369d3ae0 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.751+0000 7f4daaff5640 1 -- 172.21.15.116:0/2311279170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4dbc09ae60 con 0x5606369d3ae0 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4daaff5640 1 -- 172.21.15.116:0/2311279170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4dc0029b60 con 0x5606369d3ae0 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 10 client.17475 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 10 client.17475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:41.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 20 client.17475 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560636a4ae40 0x560636a4d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 4 client.17475 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5606366b8890 con 0x560636a4ae40 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560636a4ae40 0x560636a4d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4daaff5640 10 client.17475 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dc6d9d640 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560636a4ae40 0x560636a4d260 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c83accebf82ede server_cookie=e1ecd209fdf38ec5 in_seq=0 out_seq=0 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4daaff5640 1 -- 172.21.15.116:0/2311279170 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5606366b8890 con 0x560636a4ae40 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4daaff5640 10 client.17475 handle_command_reply: tid=0 2024-03-24T09:28:41.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 client.17475 shutdown 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 msgr2=0x560636a48e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 0x560636a48e90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0020660 tx=0x7f4dc0004ab0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 shutdown_connections 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5606369d3ae0 0x560636a48e90 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5606369c67a0 0x5606369ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560636a4ae40 0x560636a4d260 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 --2- 172.21.15.116:0/2311279170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606369c5e80 0x5606369cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4daaff5640 0 client.17475 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:41.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.752+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 >> 172.21.15.116:0/2311279170 conn(0x5606369c0900 msgr2=0x5606369bfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 shutdown_connections 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcfc5c740 1 -- 172.21.15.116:0/2311279170 wait complete. 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcfc5c740 20 client.17475 trim_cache size 0 max 16384 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:28:41.763 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4db00a2230 msgr2=0x7f4db00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4db00a2230 0x7f4db00a46f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00099e0 tx=0x7f4dc000fef0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 msgr2=0x7f4dc819de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 0x7f4dc819de20 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00e9c0 tx=0x7f4dbc00ee80 comp rx=0 tx=0).stop 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 shutdown_connections 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4db00a2230 0x7f4db00a46f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4dc8107910 0x7f4dc8197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc8100990 0x7f4dc819de20 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 --2- 172.21.15.116:0/905299986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4dc80fffd0 0x7f4dc819d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 >> 172.21.15.116:0/905299986 conn(0x7f4dc8075950 msgr2=0x7f4dc80febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 shutdown_connections 2024-03-24T09:28:41.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.753+0000 7f4dcd083640 1 -- 172.21.15.116:0/905299986 wait complete. 2024-03-24T09:28:41.766 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:28:41.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 -- 172.21.15.116:0/2412549402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b81032a0 msgr2=0x7fd1b810b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 --2- 172.21.15.116:0/2412549402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b81032a0 0x7fd1b810b780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009900 tx=0x7fd1a802bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 -- 172.21.15.116:0/2412549402 shutdown_connections 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 --2- 172.21.15.116:0/2412549402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b810bd50 0x7fd1b810e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 --2- 172.21.15.116:0/2412549402 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b81032a0 0x7fd1b810b780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 --2- 172.21.15.116:0/2412549402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1b8102980 0x7fd1b8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 -- 172.21.15.116:0/2412549402 >> 172.21.15.116:0/2412549402 conn(0x7fd1b8077710 msgr2=0x7fd1b8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:41.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 -- 172.21.15.116:0/2412549402 shutdown_connections 2024-03-24T09:28:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 -- 172.21.15.116:0/2412549402 wait complete. 2024-03-24T09:28:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 Processor -- start 2024-03-24T09:28:41.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.868+0000 7fd1be484640 1 -- start start 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 0x7fd1b819d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1b81032a0 0x7fd1b819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b810bd50 0x7fd1b8197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8198130 con 0x7fd1b8102980 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b81982a0 con 0x7fd1b81032a0 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8198460 con 0x7fd1b810bd50 2024-03-24T09:28:41.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 0x7fd1b819d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 0x7fd1b819d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49326/0 (socket says 172.21.15.116:49326) 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 -- 172.21.15.116:0/1495182293 learned_addr learned my addr 172.21.15.116:0/1495182293 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b77fe640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1b81032a0 0x7fd1b819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1bca21640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b810bd50 0x7fd1b8197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 -- 172.21.15.116:0/1495182293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b810bd50 msgr2=0x7fd1b8197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b810bd50 0x7fd1b8197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 -- 172.21.15.116:0/1495182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1b81032a0 msgr2=0x7fd1b819de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1b81032a0 0x7fd1b819de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 -- 172.21.15.116:0/1495182293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a8009560 con 0x7fd1b8102980 2024-03-24T09:28:41.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b7fff640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 0x7fd1b819d930 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e9f0 tx=0x7fd1a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0009700 con 0x7fd1b8102980 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd1a000cd80 con 0x7fd1b8102980 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b8198740 con 0x7fd1b8102980 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0018840 con 0x7fd1b8102980 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.869+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b81a49d0 con 0x7fd1b8102980 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.871+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b8068500 con 0x7fd1b8102980 2024-03-24T09:28:41.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.871+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd1a00189a0 con 0x7fd1b8102980 2024-03-24T09:28:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.872+0000 7fd1b57fa640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1a40a2230 0x7fd1a40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.872+0000 7fd1b77fe640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1a40a2230 0x7fd1a40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.872+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd1a0014070 con 0x7fd1b8102980 2024-03-24T09:28:41.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.872+0000 7fd1b77fe640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1a40a2230 0x7fd1a40a46f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8006010 tx=0x7fd1a8005df0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:41.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:41.877+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd1a0053280 con 0x7fd1b8102980 2024-03-24T09:28:42.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.088+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1b8106e40 con 0x7fd1b8102980 2024-03-24T09:28:42.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.089+0000 7fd1b57fa640 1 -- 172.21.15.116:0/1495182293 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7fd1a0097120 con 0x7fd1b8102980 2024-03-24T09:28:42.089 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:28:42.089 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:28:42.090 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:28:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1a40a2230 msgr2=0x7fd1a40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1a40a2230 0x7fd1a40a46f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8006010 tx=0x7fd1a8005df0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 msgr2=0x7fd1b819d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 0x7fd1b819d930 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000e9f0 tx=0x7fd1a000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 shutdown_connections 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1a40a2230 0x7fd1a40a46f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd1b810bd50 0x7fd1b8197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1b81032a0 0x7fd1b819de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 --2- 172.21.15.116:0/1495182293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1b8102980 0x7fd1b819d930 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 >> 172.21.15.116:0/1495182293 conn(0x7fd1b8077710 msgr2=0x7fd1b80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 shutdown_connections 2024-03-24T09:28:42.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.091+0000 7fd1be484640 1 -- 172.21.15.116:0/1495182293 wait complete. 2024-03-24T09:28:42.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:28:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 -- 172.21.15.116:0/680232141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c106660 msgr2=0x7f867c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 --2- 172.21.15.116:0/680232141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c106660 0x7f867c106a40 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f866c009930 tx=0x7f866c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 -- 172.21.15.116:0/680232141 shutdown_connections 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 --2- 172.21.15.116:0/680232141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f867c100f80 0x7f867c10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 --2- 172.21.15.116:0/680232141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867c1005c0 0x7f867c100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 --2- 172.21.15.116:0/680232141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c106660 0x7f867c106a40 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 -- 172.21.15.116:0/680232141 >> 172.21.15.116:0/680232141 conn(0x7f867c076990 msgr2=0x7f867c076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 -- 172.21.15.116:0/680232141 shutdown_connections 2024-03-24T09:28:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 -- 172.21.15.116:0/680232141 wait complete. 2024-03-24T09:28:42.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 Processor -- start 2024-03-24T09:28:42.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.207+0000 7f86827f4640 1 -- start start 2024-03-24T09:28:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f86827f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f867c1005c0 0x7f867c198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f86827f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 0x7f867c198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f86827f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867c106660 0x7f867c1a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f86827f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f867c1997e0 con 0x7f867c100f80 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f86827f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f867c1a4000 con 0x7f867c106660 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f86827f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f867c1a4140 con 0x7f867c1005c0 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 0x7f867c198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f8680d91640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867c106660 0x7f867c1a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 0x7f867c198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49330/0 (socket says 172.21.15.116:49330) 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 -- 172.21.15.116:0/3342583281 learned_addr learned my addr 172.21.15.116:0/3342583281 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867bfff640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f867c1005c0 0x7f867c198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 -- 172.21.15.116:0/3342583281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f867c1005c0 msgr2=0x7f867c198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f867c1005c0 0x7f867c198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 -- 172.21.15.116:0/3342583281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867c106660 msgr2=0x7f867c1a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867c106660 0x7f867c1a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 -- 172.21.15.116:0/3342583281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8670009490 con 0x7f867c100f80 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.208+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 0x7f867c198f80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f867000ea60 tx=0x7f867000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.209+0000 7f86797fa640 1 -- 172.21.15.116:0/3342583281 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867000cde0 con 0x7f867c100f80 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.209+0000 7f86797fa640 1 -- 172.21.15.116:0/3342583281 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86700044d0 con 0x7f867c100f80 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.209+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866c009560 con 0x7f867c100f80 2024-03-24T09:28:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.209+0000 7f86797fa640 1 -- 172.21.15.116:0/3342583281 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670010690 con 0x7f867c100f80 2024-03-24T09:28:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.209+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f867c1a4720 con 0x7f867c100f80 2024-03-24T09:28:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86797fa640 1 -- 172.21.15.116:0/3342583281 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f8670003e40 con 0x7f867c100f80 2024-03-24T09:28:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 Processor -- start 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 -- start start 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86797fa640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86640a2230 0x7f86640a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e91e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e9273b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e929ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f76e88c840 con 0x55f76e9278f0 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f76e889d70 con 0x55f76e91ed10 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.211+0000 7f86853cd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f76e885550 con 0x55f76e91e3f0 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86640a2230 0x7f86640a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e929ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f86797fa640 1 -- 172.21.15.116:0/3342583281 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f8670014070 con 0x7f867c100f80 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f867bfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e91e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e929ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49344/0 (socket says 172.21.15.116:49344) 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 -- 172.21.15.116:0/3454503430 learned_addr learned my addr 172.21.15.116:0/3454503430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e9273b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 -- 172.21.15.116:0/3454503430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 msgr2=0x55f76e91e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e91e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 -- 172.21.15.116:0/3454503430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 msgr2=0x55f76e9273b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e9273b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 -- 172.21.15.116:0/3454503430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f76e81ade0 con 0x55f76e9278f0 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f867bfff640 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e91e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e929ce0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8674010dd0 tx=0x7f867400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b88a1edc05cab3a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f8680d91640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86640a2230 0x7f86640a46f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f867400fc90 tx=0x7f8674018000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f86627fc640 1 -- 172.21.15.116:0/3454503430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674009e20 con 0x55f76e9278f0 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f86627fc640 1 -- 172.21.15.116:0/3454503430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86740106b0 con 0x55f76e9278f0 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.212+0000 7f86627fc640 1 -- 172.21.15.116:0/3454503430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86740056c0 con 0x55f76e9278f0 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 -- 172.21.15.116:0/3454503430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 msgr2=0x55f76e929ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e929ce0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f8674010dd0 tx=0x7f867400f790 comp rx=0 tx=0).stop 2024-03-24T09:28:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 -- 172.21.15.116:0/3454503430 shutdown_connections 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e929ce0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e9273b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 --2- 172.21.15.116:0/3454503430 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e91e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 -- 172.21.15.116:0/3454503430 >> 172.21.15.116:0/3454503430 conn(0x55f76e918b20 msgr2=0x55f76e918f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 -- 172.21.15.116:0/3454503430 shutdown_connections 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 -- 172.21.15.116:0/3454503430 wait complete. 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.213+0000 7f86853cd740 1 -- start start 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 Processor -- start 2024-03-24T09:28:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91e3f0 0x55f76e924970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e924eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e9253f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f76e558430 con 0x55f76e9278f0 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f76e514e00 con 0x55f76e91e3f0 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f76e80a590 con 0x55f76e91ed10 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f86853cd740 10 client.0 _ll_register_callbacks cb 0x55f76e879fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e9253f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f867bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91e3f0 0x55f76e924970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e9253f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49354/0 (socket says 172.21.15.116:49354) 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 -- 172.21.15.116:0/1439936968 learned_addr learned my addr 172.21.15.116:0/1439936968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f867b7fe640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e924eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 -- 172.21.15.116:0/1439936968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 msgr2=0x55f76e924eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e924eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 -- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91e3f0 msgr2=0x55f76e924970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91e3f0 0x55f76e924970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 -- 172.21.15.116:0/1439936968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86740109b0 con 0x55f76e9278f0 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f867bfff640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91e3f0 0x55f76e924970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f8680d91640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e9253f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8674006010 tx=0x7f8674004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acb6509737fc071 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.214+0000 7f865b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f865b7fe640 1 -- 172.21.15.116:0/1439936968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867402a040 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f865b7fe640 1 -- 172.21.15.116:0/1439936968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f86740055c0 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f76e5847e0 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f865b7fe640 1 -- 172.21.15.116:0/1439936968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8674009e20 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 10 client.17493 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f76e883bd0 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f865b7fe640 1 -- 172.21.15.116:0/1439936968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f76e5847e0 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f865b7fe640 1 -- 172.21.15.116:0/1439936968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8674024070 con 0x55f76e9278f0 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 10 client.17493 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 10 client.17493 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 20 client.17493 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e9379b0 0x55f76e939dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 4 client.17493 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f76e5c1720 con 0x55f76e9379b0 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.215+0000 7f867bfff640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e9379b0 0x55f76e939dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f865b7fe640 10 client.17493 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f867bfff640 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e9379b0 0x55f76e939dd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8c361f1b39922a2b server_cookie=769c216f2e3f7aa4 in_seq=0 out_seq=0 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f865b7fe640 1 -- 172.21.15.116:0/1439936968 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f76e5c1720 con 0x55f76e9379b0 2024-03-24T09:28:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f865b7fe640 10 client.17493 handle_command_reply: tid=0 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f86853cd740 1 client.17493 shutdown 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 msgr2=0x55f76e9253f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f86853cd740 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e9253f0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f8674006010 tx=0x7f8674004ff0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 shutdown_connections 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.216+0000 7f86853cd740 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e9379b0 0x55f76e939dd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e9253f0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e924eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f865b7fe640 0 client.17493 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 1 --2- 172.21.15.116:0/1439936968 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91e3f0 0x55f76e924970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 >> 172.21.15.116:0/1439936968 conn(0x55f76e918b20 msgr2=0x55f76e91a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 shutdown_connections 2024-03-24T09:28:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 1 -- 172.21.15.116:0/1439936968 wait complete. 2024-03-24T09:28:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.217+0000 7f86853cd740 20 client.17493 trim_cache size 0 max 16384 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.229+0000 7f86853cd740 1 Processor -- start 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 -- start start 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e9270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e933120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e91a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f76e5c1720 con 0x55f76e9278f0 2024-03-24T09:28:42.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f76e5847e0 con 0x55f76e91ed10 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f86853cd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f76e934460 con 0x55f76e91e3f0 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e91a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e91a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49358/0 (socket says 172.21.15.116:49358) 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 -- 172.21.15.116:0/289842886 learned_addr learned my addr 172.21.15.116:0/289842886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f867b7fe640 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e933120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f867bfff640 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e9270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 -- 172.21.15.116:0/289842886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 msgr2=0x55f76e9270a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e9270a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 -- 172.21.15.116:0/289842886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 msgr2=0x55f76e933120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e933120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 -- 172.21.15.116:0/289842886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f76e81fc80 con 0x55f76e9278f0 2024-03-24T09:28:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8680d91640 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e91a660 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8674010dd0 tx=0x7f8674014060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708fa9f90d8348ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8661ffb640 1 -- 172.21.15.116:0/289842886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f867402a070 con 0x55f76e9278f0 2024-03-24T09:28:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8661ffb640 1 -- 172.21.15.116:0/289842886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f867400fcc0 con 0x55f76e9278f0 2024-03-24T09:28:42.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.230+0000 7f8661ffb640 1 -- 172.21.15.116:0/289842886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674005590 con 0x55f76e9278f0 2024-03-24T09:28:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 -- 172.21.15.116:0/289842886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 msgr2=0x55f76e91a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e91a660 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f8674010dd0 tx=0x7f8674014060 comp rx=0 tx=0).stop 2024-03-24T09:28:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 -- 172.21.15.116:0/289842886 shutdown_connections 2024-03-24T09:28:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e9278f0 0x55f76e91a660 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e91ed10 0x55f76e933120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 --2- 172.21.15.116:0/289842886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91e3f0 0x55f76e9270a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 -- 172.21.15.116:0/289842886 >> 172.21.15.116:0/289842886 conn(0x55f76e918b20 msgr2=0x55f76e919f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 -- 172.21.15.116:0/289842886 shutdown_connections 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 -- 172.21.15.116:0/289842886 wait complete. 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.231+0000 7f86853cd740 1 -- start start 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 Processor -- start 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 0x55f76e921600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e921b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e9278f0 0x55f76e922080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f76e99ccd0 con 0x55f76e91e3f0 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f76e99ce10 con 0x55f76e9278f0 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f76e99cfd0 con 0x55f76e91ed10 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 0x55f76e921600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 10 client.0 _ll_register_callbacks cb 0x7f867000cf40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 0x55f76e921600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49368/0 (socket says 172.21.15.116:49368) 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 -- 172.21.15.116:0/3719046997 learned_addr learned my addr 172.21.15.116:0/3719046997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f8680d91640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e9278f0 0x55f76e922080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e921b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 -- 172.21.15.116:0/3719046997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 msgr2=0x55f76e921b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e921b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 -- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e9278f0 msgr2=0x55f76e922080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e9278f0 0x55f76e922080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 -- 172.21.15.116:0/3719046997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f76e883bd0 con 0x55f76e91e3f0 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f867bfff640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 0x55f76e921600 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f866c009340 tx=0x7f866c04ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55c93c90c65fe72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f865b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f865b7fe640 1 -- 172.21.15.116:0/3719046997 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866c045040 con 0x55f76e91e3f0 2024-03-24T09:28:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f865b7fe640 1 -- 172.21.15.116:0/3719046997 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f866c02de40 con 0x55f76e91e3f0 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.232+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f76e610890 con 0x55f76e91e3f0 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f865b7fe640 1 -- 172.21.15.116:0/3719046997 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f866c04de20 con 0x55f76e91e3f0 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 10 client.17499 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f76e845920 con 0x55f76e91e3f0 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f865b7fe640 1 -- 172.21.15.116:0/3719046997 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f76e610890 con 0x55f76e91e3f0 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f865b7fe640 1 -- 172.21.15.116:0/3719046997 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f866c03c040 con 0x55f76e91e3f0 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 10 client.17499 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 10 client.17499 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:28:42.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 20 client.17499 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e99e870 0x55f76e9a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 4 client.17499 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f76e530700 con 0x55f76e99e870 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e99e870 0x55f76e9a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f867b7fe640 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e99e870 0x55f76e9a0c90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e3ba3c0aa0927643 server_cookie=f2ee3e13a2a5b775 in_seq=0 out_seq=0 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.233+0000 7f865b7fe640 10 client.17499 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f865b7fe640 1 -- 172.21.15.116:0/3719046997 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f76e530700 con 0x55f76e99e870 2024-03-24T09:28:42.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f865b7fe640 10 client.17499 handle_command_reply: tid=0 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 client.17499 shutdown 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 msgr2=0x55f76e921600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 0x55f76e921600 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f866c009340 tx=0x7f866c04ddb0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 shutdown_connections 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f76e9278f0 0x55f76e922080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f76e91ed10 0x55f76e921b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f76e99e870 0x55f76e9a0c90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 --2- 172.21.15.116:0/3719046997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f76e91e3f0 0x55f76e921600 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f865b7fe640 0 client.17499 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:28:42.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 >> 172.21.15.116:0/3719046997 conn(0x55f76e918b20 msgr2=0x55f76e91a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 shutdown_connections 2024-03-24T09:28:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 1 -- 172.21.15.116:0/3719046997 wait complete. 2024-03-24T09:28:42.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86853cd740 20 client.17499 trim_cache size 0 max 16384 2024-03-24T09:28:42.244 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86640a2230 msgr2=0x7f86640a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86827f4640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86640a2230 0x7f86640a46f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f867400fc90 tx=0x7f8674018000 comp rx=0 tx=0).stop 2024-03-24T09:28:42.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 msgr2=0x7f867c198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.234+0000 7f86827f4640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 0x7f867c198f80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f867000ea60 tx=0x7f867000ef20 comp rx=0 tx=0).stop 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 shutdown_connections 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867c106660 0x7f867c1a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f867c100f80 0x7f867c198f80 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86640a2230 0x7f86640a46f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 --2- 172.21.15.116:0/3342583281 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f867c1005c0 0x7f867c198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 >> 172.21.15.116:0/3342583281 conn(0x7f867c076990 msgr2=0x7f867c0fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 shutdown_connections 2024-03-24T09:28:42.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.235+0000 7f86827f4640 1 -- 172.21.15.116:0/3342583281 wait complete. 2024-03-24T09:28:42.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 -- 172.21.15.116:0/3554153781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 msgr2=0x7fd724073ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 --2- 172.21.15.116:0/3554153781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724073ea0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd714009900 tx=0x7fd71402bcd0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 -- 172.21.15.116:0/3554153781 shutdown_connections 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 --2- 172.21.15.116:0/3554153781 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd72410df40 0x7fd724110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 --2- 172.21.15.116:0/3554153781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7240743e0 0x7fd72410da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 --2- 172.21.15.116:0/3554153781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724073ea0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 -- 172.21.15.116:0/3554153781 >> 172.21.15.116:0/3554153781 conn(0x7fd7240fd170 msgr2=0x7fd7240ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 -- 172.21.15.116:0/3554153781 shutdown_connections 2024-03-24T09:28:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 -- 172.21.15.116:0/3554153781 wait complete. 2024-03-24T09:28:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 Processor -- start 2024-03-24T09:28:42.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.350+0000 7fd72a76c640 1 -- start start 2024-03-24T09:28:42.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd72a76c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd72a76c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7240743e0 0x7fd72419a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd72a76c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd72410df40 0x7fd72419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd72a76c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd72419ed10 con 0x7fd724073ac0 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd72a76c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd72419ee80 con 0x7fd72410df40 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd72a76c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd72419f040 con 0x7fd7240743e0 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49398/0 (socket says 172.21.15.116:49398) 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 -- 172.21.15.116:0/2102919062 learned_addr learned my addr 172.21.15.116:0/2102919062 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd728d09640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd72410df40 0x7fd72419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd7237fe640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7240743e0 0x7fd72419a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 -- 172.21.15.116:0/2102919062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7240743e0 msgr2=0x7fd72419a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7240743e0 0x7fd72419a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 -- 172.21.15.116:0/2102919062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd72410df40 msgr2=0x7fd72419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd72410df40 0x7fd72419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 -- 172.21.15.116:0/2102919062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd718009490 con 0x7fd724073ac0 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.351+0000 7fd723fff640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724199e50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd714006010 tx=0x7fd714005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.352+0000 7fd7217fa640 1 -- 172.21.15.116:0/2102919062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7140043d0 con 0x7fd724073ac0 2024-03-24T09:28:42.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.352+0000 7fd7217fa640 1 -- 172.21.15.116:0/2102919062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd714003aa0 con 0x7fd724073ac0 2024-03-24T09:28:42.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.352+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd714009560 con 0x7fd724073ac0 2024-03-24T09:28:42.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.352+0000 7fd7217fa640 1 -- 172.21.15.116:0/2102919062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd714005350 con 0x7fd724073ac0 2024-03-24T09:28:42.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.352+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7241a69c0 con 0x7fd724073ac0 2024-03-24T09:28:42.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd7217fa640 1 -- 172.21.15.116:0/2102919062 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd714003c10 con 0x7fd724073ac0 2024-03-24T09:28:42.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 Processor -- start 2024-03-24T09:28:42.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 -- start start 2024-03-24T09:28:42.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065e57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ee3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 0x555f065f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f06553840 con 0x555f065e53f0 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f06550d70 con 0x555f065ee8f0 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd72d428740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555f0654c550 con 0x555f065e5d10 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd7217fa640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd70c0a2230 0x7fd70c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd728d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065e57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd7237fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 0x555f065f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd723fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ee3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd728d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065e57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49412/0 (socket says 172.21.15.116:49412) 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd7237fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 0x555f065f0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46348/0 (socket says 172.21.15.116:46348) 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd728d09640 1 -- 172.21.15.116:0/67167683 learned_addr learned my addr 172.21.15.116:0/67167683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.354+0000 7fd728d09640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd70c0a2230 0x7fd70c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7217fa640 1 -- 172.21.15.116:0/2102919062 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd7140f0ac0 con 0x7fd724073ac0 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7237fe640 1 -- 172.21.15.116:0/67167683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 msgr2=0x555f065ee3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7237fe640 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ee3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7237fe640 1 -- 172.21.15.116:0/67167683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 msgr2=0x555f065e57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7237fe640 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065e57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7237fe640 1 -- 172.21.15.116:0/67167683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f064e1de0 con 0x555f065ee8f0 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd728d09640 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065e57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd7237fe640 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 0x555f065f0ce0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd718009d80 tx=0x7fd718011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=439ad8abc7a4f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd728d09640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd70c0a2230 0x7fd70c0a46f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0099e0 tx=0x7fd71c009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd70a7fc640 1 -- 172.21.15.116:0/67167683 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718019410 con 0x555f065ee8f0 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd70a7fc640 1 -- 172.21.15.116:0/67167683 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd718019a20 con 0x555f065ee8f0 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.355+0000 7fd70a7fc640 1 -- 172.21.15.116:0/67167683 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718018a80 con 0x555f065ee8f0 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 -- 172.21.15.116:0/67167683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 msgr2=0x555f065f0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 0x555f065f0ce0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd718009d80 tx=0x7fd718011810 comp rx=0 tx=0).stop 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 -- 172.21.15.116:0/67167683 shutdown_connections 2024-03-24T09:28:42.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065ee8f0 0x555f065f0ce0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ee3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 --2- 172.21.15.116:0/67167683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065e57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 -- 172.21.15.116:0/67167683 >> 172.21.15.116:0/67167683 conn(0x555f065dfb20 msgr2=0x555f065dff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 -- 172.21.15.116:0/67167683 shutdown_connections 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 -- 172.21.15.116:0/67167683 wait complete. 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 -- start start 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.356+0000 7fd72d428740 1 Processor -- start 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065eb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ebec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f065ec400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f0621f430 con 0x555f065ee8f0 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f061dbe00 con 0x555f065e53f0 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555f064d1590 con 0x555f065e5d10 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 10 client.0 _ll_register_callbacks cb 0x555f06540fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f065ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd723fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065eb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd7237fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f065ec400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49428/0 (socket says 172.21.15.116:49428) 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd723fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065eb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46350/0 (socket says 172.21.15.116:46350) 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 -- 172.21.15.116:0/2669533091 learned_addr learned my addr 172.21.15.116:0/2669533091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 -- 172.21.15.116:0/2669533091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 msgr2=0x555f065ebec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ebec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 -- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 msgr2=0x555f065eb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065eb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 -- 172.21.15.116:0/2669533091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7180099e0 con 0x555f065ee8f0 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd7237fe640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd723fff640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd728d09640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f065ec400 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fd71c011a70 tx=0x7fd71c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d3075ddfc59543 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd7037fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd7037fe640 1 -- 172.21.15.116:0/2669533091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0047d0 con 0x555f065ee8f0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd7037fe640 1 -- 172.21.15.116:0/2669533091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd71c004b20 con 0x555f065ee8f0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555f0624b7e0 con 0x555f065ee8f0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd7037fe640 1 -- 172.21.15.116:0/2669533091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd71c004ce0 con 0x555f065ee8f0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 10 client.17511 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.357+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555f0654abd0 con 0x555f065ee8f0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd7037fe640 1 -- 172.21.15.116:0/2669533091 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x555f0624b7e0 con 0x555f065ee8f0 2024-03-24T09:28:42.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd7037fe640 1 -- 172.21.15.116:0/2669533091 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd71c005800 con 0x555f065ee8f0 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd72d428740 10 client.17511 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd72d428740 10 client.17511 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd72d428740 20 client.17511 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd72d428740 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f065feb30 0x555f06600f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd72d428740 4 client.17511 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f06288720 con 0x555f065feb30 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd723fff640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f065feb30 0x555f06600f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd7037fe640 10 client.17511 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.358+0000 7fd723fff640 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f065feb30 0x555f06600f50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b29a8a51357632ce server_cookie=d68765b9cc35024f in_seq=0 out_seq=0 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd7037fe640 1 -- 172.21.15.116:0/2669533091 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555f06288720 con 0x555f065feb30 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd7037fe640 10 client.17511 handle_command_reply: tid=0 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 client.17511 shutdown 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 msgr2=0x555f065ec400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f065ec400 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fd71c011a70 tx=0x7fd71c004a40 comp rx=0 tx=0).stop 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 shutdown_connections 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f065feb30 0x555f06600f50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f065ec400 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd7037fe640 0 client.17511 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 --2- 172.21.15.116:0/2669533091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065eb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 >> 172.21.15.116:0/2669533091 conn(0x555f065dfb20 msgr2=0x555f065e1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 shutdown_connections 2024-03-24T09:28:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.359+0000 7fd72d428740 1 -- 172.21.15.116:0/2669533091 wait complete. 2024-03-24T09:28:42.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.360+0000 7fd72d428740 20 client.17511 trim_cache size 0 max 16384 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.372+0000 7fd72d428740 1 Processor -- start 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.372+0000 7fd72d428740 1 -- start start 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd72d428740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065fa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd72d428740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ec1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd72d428740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f06606860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd72d428740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f065e9ed0 con 0x555f065ee8f0 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd72d428740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f065e4770 con 0x555f065e53f0 2024-03-24T09:28:42.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd72d428740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555f065e7520 con 0x555f065e5d10 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f06606860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f06606860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49430/0 (socket says 172.21.15.116:49430) 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 -- 172.21.15.116:0/3821027831 learned_addr learned my addr 172.21.15.116:0/3821027831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd723fff640 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065fa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd7237fe640 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ec1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 -- 172.21.15.116:0/3821027831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 msgr2=0x555f065ec1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ec1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 -- 172.21.15.116:0/3821027831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 msgr2=0x555f065fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065fa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 -- 172.21.15.116:0/3821027831 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f064e6c80 con 0x555f065ee8f0 2024-03-24T09:28:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd728d09640 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f06606860 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fd71c011fd0 tx=0x7fd71c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1892ab0ca7ed739e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd709ffb640 1 -- 172.21.15.116:0/3821027831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c01f920 con 0x555f065ee8f0 2024-03-24T09:28:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd709ffb640 1 -- 172.21.15.116:0/3821027831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd71c004ab0 con 0x555f065ee8f0 2024-03-24T09:28:42.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.373+0000 7fd709ffb640 1 -- 172.21.15.116:0/3821027831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c005020 con 0x555f065ee8f0 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 -- 172.21.15.116:0/3821027831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 msgr2=0x555f06606860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f06606860 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fd71c011fd0 tx=0x7fd71c009dc0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 -- 172.21.15.116:0/3821027831 shutdown_connections 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065ee8f0 0x555f06606860 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065e5d10 0x555f065ec1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 --2- 172.21.15.116:0/3821027831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e53f0 0x555f065fa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 -- 172.21.15.116:0/3821027831 >> 172.21.15.116:0/3821027831 conn(0x555f065dfb20 msgr2=0x555f065e1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 -- 172.21.15.116:0/3821027831 shutdown_connections 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 -- 172.21.15.116:0/3821027831 wait complete. 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 -- start start 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.374+0000 7fd72d428740 1 Processor -- start 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e5d10 0x555f065f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065ee8f0 0x555f065f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555f06663fb0 con 0x555f065e53f0 2024-03-24T09:28:42.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555f06664120 con 0x555f065e5d10 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x555f066642e0 con 0x555f065ee8f0 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 10 client.0 _ll_register_callbacks cb 0x7fd71c011930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49444/0 (socket says 172.21.15.116:49444) 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 -- 172.21.15.116:0/437391239 learned_addr learned my addr 172.21.15.116:0/437391239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd7237fe640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e5d10 0x555f065f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd728d09640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065ee8f0 0x555f065f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 -- 172.21.15.116:0/437391239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065ee8f0 msgr2=0x555f065f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065ee8f0 0x555f065f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 -- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e5d10 msgr2=0x555f065f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e5d10 0x555f065f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 -- 172.21.15.116:0/437391239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0654abd0 con 0x555f065e53f0 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd723fff640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065f22b0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd714003ec0 tx=0x7fd7140f0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d9f77e86ab1255 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd7037fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd7037fe640 1 -- 172.21.15.116:0/437391239 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7140c1160 con 0x555f065e53f0 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd7037fe640 1 -- 172.21.15.116:0/437391239 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fd7140bd910 con 0x555f065e53f0 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.375+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd718019a20 con 0x555f065e53f0 2024-03-24T09:28:42.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd7037fe640 1 -- 172.21.15.116:0/437391239 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd7140bd210 con 0x555f065e53f0 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 10 client.17517 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x555f0650c920 con 0x555f065e53f0 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd7037fe640 1 -- 172.21.15.116:0/437391239 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd718019a20 con 0x555f065e53f0 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd7037fe640 1 -- 172.21.15.116:0/437391239 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd71403c070 con 0x555f065e53f0 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 10 client.17517 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 10 client.17517 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 20 client.17517 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f06665860 0x555f06667c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 4 client.17517 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f062d7890 con 0x555f06665860 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd7237fe640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f06665860 0x555f06667c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd7037fe640 10 client.17517 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.376+0000 7fd7237fe640 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f06665860 0x555f06667c50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=13590a80281cde7f server_cookie=a257756bae60a749 in_seq=0 out_seq=0 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd7037fe640 1 -- 172.21.15.116:0/437391239 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f062d7890 con 0x555f06665860 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd7037fe640 10 client.17517 handle_command_reply: tid=0 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 client.17517 shutdown 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 msgr2=0x555f065f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065f22b0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fd714003ec0 tx=0x7fd7140f0040 comp rx=0 tx=0).stop 2024-03-24T09:28:42.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 shutdown_connections 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x555f06665860 0x555f06667c50 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x555f065ee8f0 0x555f065f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555f065e5d10 0x555f065f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd7037fe640 0 client.17517 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 --2- 172.21.15.116:0/437391239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555f065e53f0 0x555f065f22b0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 >> 172.21.15.116:0/437391239 conn(0x555f065dfb20 msgr2=0x555f065de430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 shutdown_connections 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 1 -- 172.21.15.116:0/437391239 wait complete. 2024-03-24T09:28:42.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72d428740 20 client.17517 trim_cache size 0 max 16384 2024-03-24T09:28:42.387 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd70c0a2230 msgr2=0x7fd70c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd70c0a2230 0x7fd70c0a46f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0099e0 tx=0x7fd71c009500 comp rx=0 tx=0).stop 2024-03-24T09:28:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 msgr2=0x7fd724199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724199e50 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fd714006010 tx=0x7fd714005e10 comp rx=0 tx=0).stop 2024-03-24T09:28:42.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 shutdown_connections 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd72410df40 0x7fd72419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd70c0a2230 0x7fd70c0a46f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7240743e0 0x7fd72419a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 --2- 172.21.15.116:0/2102919062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd724073ac0 0x7fd724199e50 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.377+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 >> 172.21.15.116:0/2102919062 conn(0x7fd7240fd170 msgr2=0x7fd7240fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.378+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 shutdown_connections 2024-03-24T09:28:42.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.378+0000 7fd72a76c640 1 -- 172.21.15.116:0/2102919062 wait complete. 2024-03-24T09:28:42.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:28:42.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 -- 172.21.15.116:0/673281866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 msgr2=0x7f57c010d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 --2- 172.21.15.116:0/673281866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c010d500 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009cd0 tx=0x7f57b802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 -- 172.21.15.116:0/673281866 shutdown_connections 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 --2- 172.21.15.116:0/673281866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c010d500 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 --2- 172.21.15.116:0/673281866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 0x7f57c0100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 --2- 172.21.15.116:0/673281866 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c0106640 0x7f57c0106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 -- 172.21.15.116:0/673281866 >> 172.21.15.116:0/673281866 conn(0x7f57c0077770 msgr2=0x7f57c0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 -- 172.21.15.116:0/673281866 shutdown_connections 2024-03-24T09:28:42.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 -- 172.21.15.116:0/673281866 wait complete. 2024-03-24T09:28:42.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.492+0000 7f57c6840640 1 Processor -- start 2024-03-24T09:28:42.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 -- start start 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 0x7f57c019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c0106640 0x7f57c0197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57c0198170 con 0x7f57c0100ff0 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c01982e0 con 0x7f57c0100630 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c6840640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f57c01984a0 con 0x7f57c0106640 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c019de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49452/0 (socket says 172.21.15.116:49452) 2024-03-24T09:28:42.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 -- 172.21.15.116:0/3813485449 learned_addr learned my addr 172.21.15.116:0/3813485449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bffff640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 0x7f57c019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57c4ddd640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c0106640 0x7f57c0197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 -- 172.21.15.116:0/3813485449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c0106640 msgr2=0x7f57c0197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c0106640 0x7f57c0197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 -- 172.21.15.116:0/3813485449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 msgr2=0x7f57c019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 0x7f57c019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 -- 172.21.15.116:0/3813485449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b8009900 con 0x7f57c0100ff0 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bffff640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 0x7f57c019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.493+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c019de60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f57b000e9f0 tx=0x7f57b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.494+0000 7f57bd7fa640 1 -- 172.21.15.116:0/3813485449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0009700 con 0x7f57c0100ff0 2024-03-24T09:28:42.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.494+0000 7f57bd7fa640 1 -- 172.21.15.116:0/3813485449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b000cd80 con 0x7f57c0100ff0 2024-03-24T09:28:42.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.494+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c0198780 con 0x7f57c0100ff0 2024-03-24T09:28:42.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.494+0000 7f57bd7fa640 1 -- 172.21.15.116:0/3813485449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0018840 con 0x7f57c0100ff0 2024-03-24T09:28:42.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.494+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c01a49c0 con 0x7f57c0100ff0 2024-03-24T09:28:42.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57bd7fa640 1 -- 172.21.15.116:0/3813485449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f57b00189a0 con 0x7f57c0100ff0 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 Processor -- start 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 -- start start 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f1b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f243b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f26ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f53e89840 con 0x563f53f1b3f0 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f53e86d70 con 0x563f53f248f0 2024-03-24T09:28:42.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57c941e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f53e82550 con 0x563f53f1bd10 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.496+0000 7f57bd7fa640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f57a80a22b0 0x7f57a80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57c4ddd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f1b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f26ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57c4ddd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f1b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49460/0 (socket says 172.21.15.116:49460) 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f243b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:39716/0 (socket says 172.21.15.116:39716) 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57c4ddd640 1 -- 172.21.15.116:0/2680826168 learned_addr learned my addr 172.21.15.116:0/2680826168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57c4ddd640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f57a80a22b0 0x7f57a80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bd7fa640 1 -- 172.21.15.116:0/3813485449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f57b0014070 con 0x7f57c0100ff0 2024-03-24T09:28:42.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 -- 172.21.15.116:0/2680826168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 msgr2=0x563f53f243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f243b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 -- 172.21.15.116:0/2680826168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 msgr2=0x563f53f1b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f1b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 -- 172.21.15.116:0/2680826168 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f53e17de0 con 0x563f53f248f0 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57c4ddd640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f57a80a22b0 0x7f57a80a4770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57b8006010 tx=0x7f57b8005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f26ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57b00cba80 tx=0x7f57b009aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9403dbacc2b0235 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57a67fc640 1 -- 172.21.15.116:0/2680826168 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b009add0 con 0x563f53f248f0 2024-03-24T09:28:42.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57a67fc640 1 -- 172.21.15.116:0/2680826168 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b009ee70 con 0x563f53f248f0 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.497+0000 7f57a67fc640 1 -- 172.21.15.116:0/2680826168 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b0097440 con 0x563f53f248f0 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 -- 172.21.15.116:0/2680826168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 msgr2=0x563f53f26ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f26ce0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f57b00cba80 tx=0x7f57b009aba0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 -- 172.21.15.116:0/2680826168 shutdown_connections 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f26ce0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f243b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 --2- 172.21.15.116:0/2680826168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f1b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 -- 172.21.15.116:0/2680826168 >> 172.21.15.116:0/2680826168 conn(0x563f53f15b20 msgr2=0x563f53f15f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 -- 172.21.15.116:0/2680826168 shutdown_connections 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 -- 172.21.15.116:0/2680826168 wait complete. 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.498+0000 7f57c941e740 1 -- start start 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 Processor -- start 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1b3f0 0x563f53f21970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 0x563f53f21eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f223f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f53b55430 con 0x563f53f1bd10 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f53b11e00 con 0x563f53f1b3f0 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f53e07590 con 0x563f53f248f0 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 10 client.0 _ll_register_callbacks cb 0x563f53e76fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 0x563f53f21eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1b3f0 0x563f53f21970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 0x563f53f21eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49472/0 (socket says 172.21.15.116:49472) 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 -- 172.21.15.116:0/4100202630 learned_addr learned my addr 172.21.15.116:0/4100202630 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c4ddd640 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f223f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 -- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 msgr2=0x563f53f223f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f223f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 -- 172.21.15.116:0/4100202630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1b3f0 msgr2=0x563f53f21970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1b3f0 0x563f53f21970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 -- 172.21.15.116:0/4100202630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c01a49c0 con 0x563f53f1bd10 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 0x563f53f21eb0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f57b009aad0 tx=0x7f57b0097650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d37791fd506d00f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f579f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f579f7fe640 1 -- 172.21.15.116:0/4100202630 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b00a34c0 con 0x563f53f1bd10 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f579f7fe640 1 -- 172.21.15.116:0/4100202630 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f57b009add0 con 0x563f53f1bd10 2024-03-24T09:28:42.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.499+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563f53b817e0 con 0x563f53f1bd10 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f579f7fe640 1 -- 172.21.15.116:0/4100202630 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f57b00ce4e0 con 0x563f53f1bd10 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 10 client.17529 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563f53e80bd0 con 0x563f53f1bd10 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f579f7fe640 1 -- 172.21.15.116:0/4100202630 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563f53b817e0 con 0x563f53f1bd10 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f579f7fe640 1 -- 172.21.15.116:0/4100202630 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f57b0099c90 con 0x563f53f1bd10 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 10 client.17529 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 10 client.17529 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:28:42.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 20 client.17529 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f34710 0x563f53f36b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 4 client.17529 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f53bbe720 con 0x563f53f34710 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.500+0000 7f57bffff640 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f34710 0x563f53f36b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57bffff640 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f34710 0x563f53f36b50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bc428be078d31eb1 server_cookie=2f0fc849ae0e3551 in_seq=0 out_seq=0 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f579f7fe640 10 client.17529 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f579f7fe640 1 -- 172.21.15.116:0/4100202630 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563f53bbe720 con 0x563f53f34710 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f579f7fe640 10 client.17529 handle_command_reply: tid=0 2024-03-24T09:28:42.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 client.17529 shutdown 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 msgr2=0x563f53f21eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 0x563f53f21eb0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f57b009aad0 tx=0x7f57b0097650 comp rx=0 tx=0).stop 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 shutdown_connections 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f223f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1bd10 0x563f53f21eb0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f34710 0x563f53f36b50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 --2- 172.21.15.116:0/4100202630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1b3f0 0x563f53f21970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f579f7fe640 0 client.17529 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:42.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 >> 172.21.15.116:0/4100202630 conn(0x563f53f15b20 msgr2=0x563f53f17870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 shutdown_connections 2024-03-24T09:28:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.501+0000 7f57c941e740 1 -- 172.21.15.116:0/4100202630 wait complete. 2024-03-24T09:28:42.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.502+0000 7f57c941e740 20 client.17529 trim_cache size 0 max 16384 2024-03-24T09:28:42.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.514+0000 7f57c941e740 1 Processor -- start 2024-03-24T09:28:42.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.514+0000 7f57c941e740 1 -- start start 2024-03-24T09:28:42.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c941e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f30180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c941e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f17610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c941e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f3c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c941e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f53f314c0 con 0x563f53f1b3f0 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c941e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f53f22e00 con 0x563f53f248f0 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c941e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f53f243d0 con 0x563f53f1bd10 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f30180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f30180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49488/0 (socket says 172.21.15.116:49488) 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 -- 172.21.15.116:0/332368242 learned_addr learned my addr 172.21.15.116:0/332368242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57c4ddd640 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f3c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f17610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 -- 172.21.15.116:0/332368242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 msgr2=0x563f53f17610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f17610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 -- 172.21.15.116:0/332368242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 msgr2=0x563f53f3c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f3c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 -- 172.21.15.116:0/332368242 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f53e1cc80 con 0x563f53f1b3f0 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57bffff640 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f30180 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x563f53f37390 tx=0x7f57b4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db183030b5b1a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57a5ffb640 1 -- 172.21.15.116:0/332368242 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4017030 con 0x563f53f1b3f0 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57a5ffb640 1 -- 172.21.15.116:0/332368242 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b4018040 con 0x563f53f1b3f0 2024-03-24T09:28:42.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.515+0000 7f57a5ffb640 1 -- 172.21.15.116:0/332368242 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4010b80 con 0x563f53f1b3f0 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 -- 172.21.15.116:0/332368242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 msgr2=0x563f53f30180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f30180 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x563f53f37390 tx=0x7f57b4005f90 comp rx=0 tx=0).stop 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 -- 172.21.15.116:0/332368242 shutdown_connections 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f248f0 0x563f53f3c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f1bd10 0x563f53f17610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 --2- 172.21.15.116:0/332368242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f30180 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 -- 172.21.15.116:0/332368242 >> 172.21.15.116:0/332368242 conn(0x563f53f15b20 msgr2=0x563f53f16f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 -- 172.21.15.116:0/332368242 shutdown_connections 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 -- 172.21.15.116:0/332368242 wait complete. 2024-03-24T09:28:42.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 -- start start 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.516+0000 7f57c941e740 1 Processor -- start 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f2cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1bd10 0x563f53f1e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f1ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563f53f1f190 con 0x563f53f1b3f0 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f53f9de60 con 0x563f53f1bd10 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563f53f9dfa0 con 0x563f53f248f0 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 10 client.0 _ll_register_callbacks cb 0x7f57b0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f2cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f2cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49502/0 (socket says 172.21.15.116:49502) 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 -- 172.21.15.116:0/2255233310 learned_addr learned my addr 172.21.15.116:0/2255233310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1bd10 0x563f53f1e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c4ddd640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f1ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 -- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 msgr2=0x563f53f1ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f1ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 -- 172.21.15.116:0/2255233310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1bd10 msgr2=0x563f53f1e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1bd10 0x563f53f1e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 -- 172.21.15.116:0/2255233310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b4012040 con 0x563f53f1b3f0 2024-03-24T09:28:42.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1bd10 0x563f53f1e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57bffff640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f2cbe0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009ef0 tx=0x7f57b4009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68590a31b9f5fa16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f579f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f579f7fe640 1 -- 172.21.15.116:0/2255233310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4017050 con 0x563f53f1b3f0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f579f7fe640 1 -- 172.21.15.116:0/2255233310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f57b4010530 con 0x563f53f1b3f0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b009ee70 con 0x563f53f1b3f0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.517+0000 7f579f7fe640 1 -- 172.21.15.116:0/2255233310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f57b4010cd0 con 0x563f53f1b3f0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 10 client.17535 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563f53e42920 con 0x563f53f1b3f0 2024-03-24T09:28:42.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f579f7fe640 1 -- 172.21.15.116:0/2255233310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f57b403b080 con 0x563f53f1b3f0 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f579f7fe640 1 -- 172.21.15.116:0/2255233310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f57b4028710 con 0x563f53f1b3f0 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 10 client.17535 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 10 client.17535 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 20 client.17535 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f9f750 0x563f53fa1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 4 client.17535 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f53c0d890 con 0x563f53f9f750 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f9f750 0x563f53fa1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f579f7fe640 10 client.17535 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:42.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.518+0000 7f57bf7fe640 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f9f750 0x563f53fa1b70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a940af8686874b2e server_cookie=2786213e8fc94368 in_seq=0 out_seq=0 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f579f7fe640 1 -- 172.21.15.116:0/2255233310 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f53c0d890 con 0x563f53f9f750 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f579f7fe640 10 client.17535 handle_command_reply: tid=0 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 client.17535 shutdown 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 msgr2=0x563f53f2cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f2cbe0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009ef0 tx=0x7f57b4009f20 comp rx=0 tx=0).stop 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 shutdown_connections 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563f53f248f0 0x563f53f1ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x563f53f9f750 0x563f53fa1b70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f53f1bd10 0x563f53f1e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 --2- 172.21.15.116:0/2255233310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563f53f1b3f0 0x563f53f2cbe0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f579f7fe640 0 client.17535 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:28:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 >> 172.21.15.116:0/2255233310 conn(0x563f53f15b20 msgr2=0x563f53f206e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 shutdown_connections 2024-03-24T09:28:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 1 -- 172.21.15.116:0/2255233310 wait complete. 2024-03-24T09:28:42.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c941e740 20 client.17535 trim_cache size 0 max 16384 2024-03-24T09:28:42.529 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f57a80a22b0 msgr2=0x7f57a80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f57a80a22b0 0x7f57a80a4770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f57b8006010 tx=0x7f57b8005c50 comp rx=0 tx=0).stop 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 msgr2=0x7f57c019de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c019de60 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f57b000e9f0 tx=0x7f57b000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 shutdown_connections 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f57a80a22b0 0x7f57a80a4770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57c0106640 0x7f57c0197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57c0100ff0 0x7f57c019de60 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 --2- 172.21.15.116:0/3813485449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0100630 0x7f57c019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.519+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 >> 172.21.15.116:0/3813485449 conn(0x7f57c0077770 msgr2=0x7f57c0103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.520+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 shutdown_connections 2024-03-24T09:28:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.520+0000 7f57c6840640 1 -- 172.21.15.116:0/3813485449 wait complete. 2024-03-24T09:28:42.532 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:28:42.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 -- 172.21.15.116:0/447306605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc100f80 msgr2=0x7efdfc10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 --2- 172.21.15.116:0/447306605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc100f80 0x7efdfc10d490 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009cd0 tx=0x7efdf402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 -- 172.21.15.116:0/447306605 shutdown_connections 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 --2- 172.21.15.116:0/447306605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc100f80 0x7efdfc10d490 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 --2- 172.21.15.116:0/447306605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdfc1005c0 0x7efdfc100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 --2- 172.21.15.116:0/447306605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc106660 0x7efdfc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 -- 172.21.15.116:0/447306605 >> 172.21.15.116:0/447306605 conn(0x7efdfc076990 msgr2=0x7efdfc076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 -- 172.21.15.116:0/447306605 shutdown_connections 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.635+0000 7efe0435c640 1 -- 172.21.15.116:0/447306605 wait complete. 2024-03-24T09:28:42.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 Processor -- start 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 -- start start 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 0x7efdfc19c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc100f80 0x7efdfc19cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdfc106660 0x7efdfc196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc196f70 con 0x7efdfc1005c0 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc1970e0 con 0x7efdfc100f80 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe0435c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc1972a0 con 0x7efdfc106660 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 0x7efdfc19c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 0x7efdfc19c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49526/0 (socket says 172.21.15.116:49526) 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 -- 172.21.15.116:0/1889314918 learned_addr learned my addr 172.21.15.116:0/1889314918 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe018f7640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc100f80 0x7efdfc19cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe028f9640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdfc106660 0x7efdfc196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 -- 172.21.15.116:0/1889314918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdfc106660 msgr2=0x7efdfc196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdfc106660 0x7efdfc196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 -- 172.21.15.116:0/1889314918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc100f80 msgr2=0x7efdfc19cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.636+0000 7efe020f8640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc100f80 0x7efdfc19cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efe020f8640 1 -- 172.21.15.116:0/1889314918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf8009490 con 0x7efdfc1005c0 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efe018f7640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc100f80 0x7efdfc19cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efe020f8640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 0x7efdfc19c720 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7efdf800e9c0 tx=0x7efdf800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efdf37fe640 1 -- 172.21.15.116:0/1889314918 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf800cde0 con 0x7efdfc1005c0 2024-03-24T09:28:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf4009900 con 0x7efdfc1005c0 2024-03-24T09:28:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efdf37fe640 1 -- 172.21.15.116:0/1889314918 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf80108e0 con 0x7efdfc1005c0 2024-03-24T09:28:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efdf37fe640 1 -- 172.21.15.116:0/1889314918 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf80188e0 con 0x7efdfc1005c0 2024-03-24T09:28:42.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.637+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdfc1a35e0 con 0x7efdfc1005c0 2024-03-24T09:28:42.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.639+0000 7efdf37fe640 1 -- 172.21.15.116:0/1889314918 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7efdf8010a50 con 0x7efdfc1005c0 2024-03-24T09:28:42.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.639+0000 7efe0700f740 1 Processor -- start 2024-03-24T09:28:42.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 -- start start 2024-03-24T09:28:42.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b147d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b1d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b1fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561938a82840 con 0x561938b143f0 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561938a7fd70 con 0x561938b14d10 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe0700f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561938a7b550 con 0x561938b1d8f0 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efdf37fe640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7efde80a2230 0x7efde80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b1d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe028f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b147d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe018f7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b1fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b1d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46460/0 (socket says 172.21.15.116:46460) 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe028f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b147d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49542/0 (socket says 172.21.15.116:49542) 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 -- 172.21.15.116:0/44206688 learned_addr learned my addr 172.21.15.116:0/44206688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe028f9640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7efde80a2230 0x7efde80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efdf37fe640 1 -- 172.21.15.116:0/1889314918 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7efdf8014070 con 0x7efdfc1005c0 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 -- 172.21.15.116:0/44206688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 msgr2=0x561938b1fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b1fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 -- 172.21.15.116:0/44206688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 msgr2=0x561938b147d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b147d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 -- 172.21.15.116:0/44206688 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561938a10de0 con 0x561938b14d10 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe028f9640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7efde80a2230 0x7efde80a46f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efdf4005cc0 tx=0x7efdf4005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.640+0000 7efe020f8640 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b1d3b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efdf806e1b0 tx=0x7efdf806e1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0b612143fe58079 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efdf0bfe640 1 -- 172.21.15.116:0/44206688 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf809b170 con 0x561938b14d10 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efdf0bfe640 1 -- 172.21.15.116:0/44206688 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf809f2c0 con 0x561938b14d10 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efdf0bfe640 1 -- 172.21.15.116:0/44206688 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf80662e0 con 0x561938b14d10 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 -- 172.21.15.116:0/44206688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 msgr2=0x561938b1d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b1d3b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efdf806e1b0 tx=0x7efdf806e1e0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 -- 172.21.15.116:0/44206688 shutdown_connections 2024-03-24T09:28:42.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b1fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b1d3b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 --2- 172.21.15.116:0/44206688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b147d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.641+0000 7efe0700f740 1 -- 172.21.15.116:0/44206688 >> 172.21.15.116:0/44206688 conn(0x561938b0eb20 msgr2=0x561938b0ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 -- 172.21.15.116:0/44206688 shutdown_connections 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 -- 172.21.15.116:0/44206688 wait complete. 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 -- start start 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 Processor -- start 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b1b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56193874e430 con 0x561938b14d10 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56193870ae00 con 0x561938b1d8f0 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561938a00590 con 0x561938b143f0 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe0700f740 10 client.0 _ll_register_callbacks cb 0x561938a6ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe018f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe018f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49552/0 (socket says 172.21.15.116:49552) 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe018f7640 1 -- 172.21.15.116:0/2519775905 learned_addr learned my addr 172.21.15.116:0/2519775905 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe028f9640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b1b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe020f8640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe028f9640 1 -- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 msgr2=0x561938b1a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe028f9640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe028f9640 1 -- 172.21.15.116:0/2519775905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 msgr2=0x561938b1aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe028f9640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.642+0000 7efe028f9640 1 -- 172.21.15.116:0/2519775905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdfc1a35e0 con 0x561938b1d8f0 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe018f7640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe028f9640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b1b410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efdf402c630 tx=0x7efdf40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7c6095b360bf4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efde5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efde5ffb640 1 -- 172.21.15.116:0/2519775905 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf4034790 con 0x561938b1d8f0 2024-03-24T09:28:42.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56193877a7e0 con 0x561938b1d8f0 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efde5ffb640 1 -- 172.21.15.116:0/2519775905 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7efdf4003b70 con 0x561938b1d8f0 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efde5ffb640 1 -- 172.21.15.116:0/2519775905 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdf4034e60 con 0x561938b1d8f0 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 10 client.27371 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561938a79bd0 con 0x561938b1d8f0 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efde5ffb640 1 -- 172.21.15.116:0/2519775905 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56193877a7e0 con 0x561938b1d8f0 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efde5ffb640 1 -- 172.21.15.116:0/2519775905 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efdf4042540 con 0x561938b1d8f0 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 10 client.27371 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 10 client.27371 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 20 client.27371 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b2da50 0x561938b2fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 4 client.27371 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.643+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5619387b7720 con 0x561938b2da50 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.644+0000 7efe020f8640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b2da50 0x561938b2fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.644+0000 7efe020f8640 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b2da50 0x561938b2fe70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4bf3ec7b4686a10f server_cookie=223694ac51dba41c in_seq=0 out_seq=0 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.644+0000 7efde5ffb640 10 client.27371 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.644+0000 7efde5ffb640 1 -- 172.21.15.116:0/2519775905 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5619387b7720 con 0x561938b2da50 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.644+0000 7efde5ffb640 10 client.27371 handle_command_reply: tid=0 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.644+0000 7efe0700f740 1 client.27371 shutdown 2024-03-24T09:28:42.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 msgr2=0x561938b1b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b1b410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7efdf402c630 tx=0x7efdf40095c0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 shutdown_connections 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b2da50 0x561938b2fe70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b1b410 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efde5ffb640 0 client.27371 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 --2- 172.21.15.116:0/2519775905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 >> 172.21.15.116:0/2519775905 conn(0x561938b0eb20 msgr2=0x561938b10870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 shutdown_connections 2024-03-24T09:28:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 1 -- 172.21.15.116:0/2519775905 wait complete. 2024-03-24T09:28:42.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.645+0000 7efe0700f740 20 client.27371 trim_cache size 0 max 16384 2024-03-24T09:28:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 Processor -- start 2024-03-24T09:28:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 -- start start 2024-03-24T09:28:42.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b35740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561938b13e80 con 0x561938b14d10 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561938b13820 con 0x561938b1d8f0 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe0700f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561938b0fb20 con 0x561938b143f0 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49564/0 (socket says 172.21.15.116:49564) 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 -- 172.21.15.116:0/526451100 learned_addr learned my addr 172.21.15.116:0/526451100 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe028f9640 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b35740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe020f8640 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 -- 172.21.15.116:0/526451100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 msgr2=0x561938b1d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 -- 172.21.15.116:0/526451100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 msgr2=0x561938b35740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b35740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 -- 172.21.15.116:0/526451100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561938a15c80 con 0x561938b14d10 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.658+0000 7efe018f7640 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1b230 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7efdec012ab0 tx=0x7efdec005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47631e7edb42dbe3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efde7fff640 1 -- 172.21.15.116:0/526451100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec017070 con 0x561938b14d10 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efde7fff640 1 -- 172.21.15.116:0/526451100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdec018040 con 0x561938b14d10 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efde7fff640 1 -- 172.21.15.116:0/526451100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec0038a0 con 0x561938b14d10 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 -- 172.21.15.116:0/526451100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 msgr2=0x561938b1b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1b230 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7efdec012ab0 tx=0x7efdec005f90 comp rx=0 tx=0).stop 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 -- 172.21.15.116:0/526451100 shutdown_connections 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b1d8f0 0x561938b35740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b14d10 0x561938b1b230 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 --2- 172.21.15.116:0/526451100 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b143f0 0x561938b1d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 -- 172.21.15.116:0/526451100 >> 172.21.15.116:0/526451100 conn(0x561938b0eb20 msgr2=0x561938b0fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 -- 172.21.15.116:0/526451100 shutdown_connections 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.659+0000 7efe0700f740 1 -- 172.21.15.116:0/526451100 wait complete. 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 -- start start 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 Processor -- start 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b257b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b22780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b22cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561938b25cf0 con 0x561938b143f0 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561938b25e60 con 0x561938b14d10 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561938b26020 con 0x561938b1d8f0 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe0700f740 10 client.0 _ll_register_callbacks cb 0x7efdf4034d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b257b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49568/0 (socket says 172.21.15.116:49568) 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 -- 172.21.15.116:0/1029453057 learned_addr learned my addr 172.21.15.116:0/1029453057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe028f9640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b22cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe018f7640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b22780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 -- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 msgr2=0x561938b22cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b22cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 -- 172.21.15.116:0/1029453057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 msgr2=0x561938b22780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b22780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 -- 172.21.15.116:0/1029453057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdec012760 con 0x561938b143f0 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efe020f8640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b257b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efdf800c6a0 tx=0x7efdf800efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58107b2be403e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.660+0000 7efde5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efde5ffb640 1 -- 172.21.15.116:0/1029453057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf80b3ea0 con 0x561938b143f0 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efde5ffb640 1 -- 172.21.15.116:0/1029453057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7efdf80977e0 con 0x561938b143f0 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdf809f2c0 con 0x561938b143f0 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efde5ffb640 1 -- 172.21.15.116:0/1029453057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdf80979a0 con 0x561938b143f0 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 10 client.17553 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561938a3b920 con 0x561938b143f0 2024-03-24T09:28:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efde5ffb640 1 -- 172.21.15.116:0/1029453057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7efdf809f2c0 con 0x561938b143f0 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efde5ffb640 1 -- 172.21.15.116:0/1029453057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7efdf809a760 con 0x561938b143f0 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 10 client.17553 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 10 client.17553 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 20 client.17553 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b94a10 0x561938b96e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 4 client.17553 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561938806890 con 0x561938b94a10 2024-03-24T09:28:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.661+0000 7efe018f7640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b94a10 0x561938b96e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efde5ffb640 10 client.17553 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:42.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe018f7640 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b94a10 0x561938b96e00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74d39d618eb622c3 server_cookie=edad4a5b612816e2 in_seq=0 out_seq=0 2024-03-24T09:28:42.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efde5ffb640 1 -- 172.21.15.116:0/1029453057 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561938806890 con 0x561938b94a10 2024-03-24T09:28:42.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efde5ffb640 10 client.17553 handle_command_reply: tid=0 2024-03-24T09:28:42.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 client.17553 shutdown 2024-03-24T09:28:42.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 msgr2=0x561938b257b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b257b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7efdf800c6a0 tx=0x7efdf800efb0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 shutdown_connections 2024-03-24T09:28:42.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x561938b94a10 0x561938b96e00 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561938b1d8f0 0x561938b22cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561938b14d10 0x561938b22780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efde5ffb640 0 client.17553 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:28:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 --2- 172.21.15.116:0/1029453057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561938b143f0 0x561938b257b0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 >> 172.21.15.116:0/1029453057 conn(0x561938b0eb20 msgr2=0x561938b0fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 shutdown_connections 2024-03-24T09:28:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 1 -- 172.21.15.116:0/1029453057 wait complete. 2024-03-24T09:28:42.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0700f740 20 client.17553 trim_cache size 0 max 16384 2024-03-24T09:28:42.675 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7efde80a2230 msgr2=0x7efde80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7efde80a2230 0x7efde80a46f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efdf4005cc0 tx=0x7efdf4005c50 comp rx=0 tx=0).stop 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 msgr2=0x7efdfc19c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 0x7efdfc19c720 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7efdf800e9c0 tx=0x7efdf800ee80 comp rx=0 tx=0).stop 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 shutdown_connections 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7efde80a2230 0x7efde80a46f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efdfc106660 0x7efdfc196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdfc100f80 0x7efdfc19cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 --2- 172.21.15.116:0/1889314918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdfc1005c0 0x7efdfc19c720 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.662+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 >> 172.21.15.116:0/1889314918 conn(0x7efdfc076990 msgr2=0x7efdfc0fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.663+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 shutdown_connections 2024-03-24T09:28:42.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.663+0000 7efe0435c640 1 -- 172.21.15.116:0/1889314918 wait complete. 2024-03-24T09:28:42.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:28:42.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 -- 172.21.15.116:0/232782247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc500104d40 msgr2=0x7fc500105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 --2- 172.21.15.116:0/232782247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc500104d40 0x7fc500105120 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc009900 tx=0x7fc4fc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 -- 172.21.15.116:0/232782247 shutdown_connections 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 --2- 172.21.15.116:0/232782247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc500100970 0x7fc50010cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 --2- 172.21.15.116:0/232782247 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5000fffd0 0x7fc500100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 --2- 172.21.15.116:0/232782247 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc500104d40 0x7fc500105120 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 -- 172.21.15.116:0/232782247 >> 172.21.15.116:0/232782247 conn(0x7fc500075950 msgr2=0x7fc500075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 -- 172.21.15.116:0/232782247 shutdown_connections 2024-03-24T09:28:42.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.784+0000 7fc5086af640 1 -- 172.21.15.116:0/232782247 wait complete. 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 Processor -- start 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 -- start start 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 0x7fc50019b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc500100970 0x7fc50019bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc500104d40 0x7fc500195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc500195f10 con 0x7fc5000fffd0 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc500196080 con 0x7fc500104d40 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc5086af640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc500196240 con 0x7fc500100970 2024-03-24T09:28:42.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 0x7fc50019b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 0x7fc50019b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49588/0 (socket says 172.21.15.116:49588) 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 -- 172.21.15.116:0/552783837 learned_addr learned my addr 172.21.15.116:0/552783837 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc505c4a640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc500100970 0x7fc50019bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc506c4c640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc500104d40 0x7fc500195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 -- 172.21.15.116:0/552783837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc500100970 msgr2=0x7fc50019bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc500100970 0x7fc50019bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 -- 172.21.15.116:0/552783837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc500104d40 msgr2=0x7fc500195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc500104d40 0x7fc500195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 -- 172.21.15.116:0/552783837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f800a950 con 0x7fc5000fffd0 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.785+0000 7fc50644b640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 0x7fc50019b6b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc005940 tx=0x7fc4fc005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.786+0000 7fc4f77fe640 1 -- 172.21.15.116:0/552783837 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc004350 con 0x7fc5000fffd0 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.786+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4fc009560 con 0x7fc5000fffd0 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.786+0000 7fc4f77fe640 1 -- 172.21.15.116:0/552783837 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4fc0044b0 con 0x7fc5000fffd0 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.786+0000 7fc4f77fe640 1 -- 172.21.15.116:0/552783837 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0052b0 con 0x7fc5000fffd0 2024-03-24T09:28:42.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.786+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5001a2500 con 0x7fc5000fffd0 2024-03-24T09:28:42.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.788+0000 7fc4f77fe640 1 -- 172.21.15.116:0/552783837 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc4fc003cc0 con 0x7fc5000fffd0 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.788+0000 7fc50b369740 1 Processor -- start 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.788+0000 7fc50b369740 1 -- start start 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50b369740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c29839a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50b369740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50b369740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50b369740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c298308840 con 0x55c298399dd0 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50b369740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c298305d70 con 0x55c29839a780 2024-03-24T09:28:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50b369740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c298301550 con 0x55c2983a5b00 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc4f77fe640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4ec0a2230 0x7fc4ec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc506c4c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c29839a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46500/0 (socket says 172.21.15.116:46500) 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 -- 172.21.15.116:0/942088371 learned_addr learned my addr 172.21.15.116:0/942088371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc505c4a640 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc506c4c640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4ec0a2230 0x7fc4ec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc4f77fe640 1 -- 172.21.15.116:0/552783837 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4fc0f0470 con 0x7fc5000fffd0 2024-03-24T09:28:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 -- 172.21.15.116:0/942088371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 msgr2=0x55c2983a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 -- 172.21.15.116:0/942088371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 msgr2=0x55c29839a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c29839a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 -- 172.21.15.116:0/942088371 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c298296de0 con 0x55c29839a780 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc505c4a640 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc50644b640 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0c4e30 tx=0x7fc4fc02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263a4f3b6366c290 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc506c4c640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4ec0a2230 0x7fc4ec0a46f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800ac90 tx=0x7fc4f8004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.789+0000 7fc4f4ff9640 1 -- 172.21.15.116:0/942088371 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0c30c0 con 0x55c29839a780 2024-03-24T09:28:42.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc4f4ff9640 1 -- 172.21.15.116:0/942088371 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4fc0c0da0 con 0x55c29839a780 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc4f4ff9640 1 -- 172.21.15.116:0/942088371 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0c1000 con 0x55c29839a780 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 -- 172.21.15.116:0/942088371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 msgr2=0x55c2983a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0c4e30 tx=0x7fc4fc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 -- 172.21.15.116:0/942088371 shutdown_connections 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a55c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 --2- 172.21.15.116:0/942088371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c29839a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 -- 172.21.15.116:0/942088371 >> 172.21.15.116:0/942088371 conn(0x55c298394b20 msgr2=0x55c298394f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 -- 172.21.15.116:0/942088371 shutdown_connections 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.790+0000 7fc50b369740 1 -- 172.21.15.116:0/942088371 wait complete. 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 -- start start 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 Processor -- start 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c2983a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c297fd4430 con 0x55c298399dd0 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c297f90e00 con 0x55c29839a780 2024-03-24T09:28:42.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c298286590 con 0x55c2983a5b00 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50b369740 10 client.0 _ll_register_callbacks cb 0x55c2982f5fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c2983a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c2983a2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49602/0 (socket says 172.21.15.116:49602) 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 -- 172.21.15.116:0/1329716156 learned_addr learned my addr 172.21.15.116:0/1329716156 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc505c4a640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc506c4c640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 -- 172.21.15.116:0/1329716156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 msgr2=0x55c2983a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 -- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 msgr2=0x55c2983a3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 -- 172.21.15.116:0/1329716156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5001a2500 con 0x55c298399dd0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc50644b640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c2983a2b10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0ede10 tx=0x7fc4fc0bd0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d434e0f6667e53e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.791+0000 7fc4e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/1329716156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc0bcc50 con 0x55c298399dd0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/1329716156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4fc0a7040 con 0x55c298399dd0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c2980007e0 con 0x55c298399dd0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/1329716156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4fc0c3050 con 0x55c298399dd0 2024-03-24T09:28:42.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 10 client.17565 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c2982ffbd0 con 0x55c298399dd0 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/1329716156 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c2980007e0 con 0x55c298399dd0 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/1329716156 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc4fc03c070 con 0x55c298399dd0 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 10 client.17565 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 10 client.17565 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 20 client.17565 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c2983b54f0 0x55c2983b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 4 client.17565 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:42.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c29803d720 con 0x55c2983b54f0 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.792+0000 7fc505c4a640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c2983b54f0 0x55c2983b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc4e9ffb640 10 client.17565 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc505c4a640 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c2983b54f0 0x55c2983b78e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60ba20f234ab076a server_cookie=2aac859a9e3fac2 in_seq=0 out_seq=0 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/1329716156 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c29803d720 con 0x55c2983b54f0 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc4e9ffb640 10 client.17565 handle_command_reply: tid=0 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 client.17565 shutdown 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 msgr2=0x55c2983a2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c2983a2b10 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc0ede10 tx=0x7fc4fc0bd0c0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 shutdown_connections 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c2983b54f0 0x55c2983b78e0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c29839a780 0x55c2983a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 --2- 172.21.15.116:0/1329716156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c298399dd0 0x55c2983a2b10 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc4e9ffb640 0 client.17565 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 >> 172.21.15.116:0/1329716156 conn(0x55c298394b20 msgr2=0x55c298396870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.793+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 shutdown_connections 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.794+0000 7fc50b369740 1 -- 172.21.15.116:0/1329716156 wait complete. 2024-03-24T09:28:42.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.794+0000 7fc50b369740 20 client.17565 trim_cache size 0 max 16384 2024-03-24T09:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.806+0000 7fc50b369740 1 Processor -- start 2024-03-24T09:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.806+0000 7fc50b369740 1 -- start start 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50b369740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c298399dd0 0x55c2983a5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50b369740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50b369740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2983a5b00 0x55c2983bd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50b369740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c2980007e0 con 0x55c29839a780 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50b369740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c29839b1d0 con 0x55c2983a5b00 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50b369740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c2983b7f10 con 0x55c298399dd0 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc505c4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2983a5b00 0x55c2983bd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc505c4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49614/0 (socket says 172.21.15.116:49614) 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc505c4a640 1 -- 172.21.15.116:0/2513978623 learned_addr learned my addr 172.21.15.116:0/2513978623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc50644b640 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c298399dd0 0x55c2983a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 -- 172.21.15.116:0/2513978623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c298399dd0 msgr2=0x55c2983a5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c298399dd0 0x55c2983a5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 -- 172.21.15.116:0/2513978623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 msgr2=0x55c2983b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 -- 172.21.15.116:0/2513978623 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c29829bc80 con 0x55c2983a5b00 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc505c4a640 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc506c4c640 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2983a5b00 0x55c2983bd160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800efd0 tx=0x7fc4f800ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede548f818a1c5d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc4ebfff640 1 -- 172.21.15.116:0/2513978623 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f800dd30 con 0x55c2983a5b00 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc4ebfff640 1 -- 172.21.15.116:0/2513978623 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4f8015040 con 0x55c2983a5b00 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.807+0000 7fc4ebfff640 1 -- 172.21.15.116:0/2513978623 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f80054e0 con 0x55c2983a5b00 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- 172.21.15.116:0/2513978623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2983a5b00 msgr2=0x55c2983bd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2983a5b00 0x55c2983bd160 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800efd0 tx=0x7fc4f800ac20 comp rx=0 tx=0).stop 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- 172.21.15.116:0/2513978623 shutdown_connections 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2983a5b00 0x55c2983bd160 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- 172.21.15.116:0/2513978623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c298399dd0 0x55c2983a5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- 172.21.15.116:0/2513978623 >> 172.21.15.116:0/2513978623 conn(0x55c298394b20 msgr2=0x55c298396800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- 172.21.15.116:0/2513978623 shutdown_connections 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- 172.21.15.116:0/2513978623 wait complete. 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- start start 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 Processor -- start 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c298399dd0 0x55c2983a92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983a9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c29841a8b0 con 0x55c29839a780 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c29841aa20 con 0x55c298399dd0 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.808+0000 7fc50b369740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c29841abe0 con 0x55c2983a5b00 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc50b369740 10 client.0 _ll_register_callbacks cb 0x7fc4f800a1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983a9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983a9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49624/0 (socket says 172.21.15.116:49624) 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc50644b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c298399dd0 0x55c2983a92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 -- 172.21.15.116:0/336134886 learned_addr learned my addr 172.21.15.116:0/336134886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc506c4c640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 -- 172.21.15.116:0/336134886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 msgr2=0x55c2983a9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 -- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c298399dd0 msgr2=0x55c2983a92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c298399dd0 0x55c2983a92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 -- 172.21.15.116:0/336134886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f0009b00 con 0x55c29839a780 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc50644b640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c298399dd0 0x55c2983a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc505c4a640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983a9800 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x55c298395f40 tx=0x7fc4f0020410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98dd6ae97de150d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc4e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/336134886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0017040 con 0x55c29839a780 2024-03-24T09:28:42.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/336134886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7fc4f0003a00 con 0x55c29839a780 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4fc0c0da0 con 0x55c29839a780 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/336134886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4f0003bc0 con 0x55c29839a780 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc50b369740 10 client.17571 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.809+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7fc4f800eb30 con 0x55c29839a780 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/336134886 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc4fc0c0da0 con 0x55c29839a780 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/336134886 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc4f0029420 con 0x55c29839a780 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 10 client.17571 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 10 client.17571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 20 client.17571 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:42.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c29841c4c0 0x55c29841e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 4 client.17571 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c29808c890 con 0x55c29841c4c0 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50644b640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c29841c4c0 0x55c29841e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc4e9ffb640 10 client.17571 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50644b640 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c29841c4c0 0x55c29841e8e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bee98db8b30b737f server_cookie=7bba364e8d9fa559 in_seq=0 out_seq=0 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc4e9ffb640 1 -- 172.21.15.116:0/336134886 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c29808c890 con 0x55c29841c4c0 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc4e9ffb640 10 client.17571 handle_command_reply: tid=0 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 client.17571 shutdown 2024-03-24T09:28:42.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 msgr2=0x55c2983a9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983a9800 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x55c298395f40 tx=0x7fc4f0020410 comp rx=0 tx=0).stop 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 shutdown_connections 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c2983a5b00 0x55c2983a9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c29839a780 0x55c2983a9800 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55c29841c4c0 0x55c29841e8e0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.810+0000 7fc50b369740 1 --2- 172.21.15.116:0/336134886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c298399dd0 0x55c2983a92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc4e9ffb640 0 client.17571 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 >> 172.21.15.116:0/336134886 conn(0x55c298394b20 msgr2=0x55c298396800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 shutdown_connections 2024-03-24T09:28:42.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc50b369740 1 -- 172.21.15.116:0/336134886 wait complete. 2024-03-24T09:28:42.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc50b369740 20 client.17571 trim_cache size 0 max 16384 2024-03-24T09:28:42.822 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:28:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4ec0a2230 msgr2=0x7fc4ec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4ec0a2230 0x7fc4ec0a46f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800ac90 tx=0x7fc4f8004ab0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 msgr2=0x7fc50019b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 0x7fc50019b6b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc005940 tx=0x7fc4fc005df0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 shutdown_connections 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc500104d40 0x7fc500195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc4ec0a2230 0x7fc4ec0a46f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc500100970 0x7fc50019bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 --2- 172.21.15.116:0/552783837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5000fffd0 0x7fc50019b6b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 >> 172.21.15.116:0/552783837 conn(0x7fc500075950 msgr2=0x7fc5000fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 shutdown_connections 2024-03-24T09:28:42.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:42.811+0000 7fc5086af640 1 -- 172.21.15.116:0/552783837 wait complete. 2024-03-24T09:28:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:43 smithi143 ceph-mon[35965]: pgmap v201: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:43 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1495182293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:28:43.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:43 smithi117 ceph-mon[35788]: pgmap v201: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:43.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:43 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1495182293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:28:43.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:43 smithi116 ceph-mon[32207]: pgmap v201: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:28:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:43 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1495182293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:28:43.825 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:28:43.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:28:43.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1513000410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 msgr2=0x7f0ef0100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1513000410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef0100ab0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009900 tx=0x7f0ee402bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1513000410 shutdown_connections 2024-03-24T09:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1513000410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ef0100ff0 0x7f0ef010d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1513000410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef0100ab0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1513000410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 0x7f0ef0106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1513000410 >> 172.21.15.116:0/1513000410 conn(0x7f0ef0077770 msgr2=0x7f0ef0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1513000410 shutdown_connections 2024-03-24T09:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.941+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1513000410 wait complete. 2024-03-24T09:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 Processor -- start 2024-03-24T09:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 -- start start 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef019d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ef0100ff0 0x7f0ef019de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 0x7f0ef0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef0198140 con 0x7f0ef0100630 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef01982b0 con 0x7f0ef0100ff0 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef85e2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef0198470 con 0x7f0ef0106640 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef019d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef019d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49642/0 (socket says 172.21.15.116:49642) 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 -- 172.21.15.116:0/1504941214 learned_addr learned my addr 172.21.15.116:0/1504941214 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ef0100ff0 0x7f0ef019de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 0x7f0ef0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 -- 172.21.15.116:0/1504941214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 msgr2=0x7f0ef0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 0x7f0ef0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 -- 172.21.15.116:0/1504941214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ef0100ff0 msgr2=0x7f0ef019de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ef0100ff0 0x7f0ef019de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef637e640 1 -- 172.21.15.116:0/1504941214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee4009560 con 0x7f0ef0100630 2024-03-24T09:28:43.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.942+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 0x7f0ef0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.943+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef019d940 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000eb90 tx=0x7f0ee000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.943+0000 7f0eef7fe640 1 -- 172.21.15.116:0/1504941214 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee00099b0 con 0x7f0ef0100630 2024-03-24T09:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.943+0000 7f0eef7fe640 1 -- 172.21.15.116:0/1504941214 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ee000ee90 con 0x7f0ef0100630 2024-03-24T09:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.943+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef0198750 con 0x7f0ef0100630 2024-03-24T09:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.943+0000 7f0eef7fe640 1 -- 172.21.15.116:0/1504941214 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0010690 con 0x7f0ef0100630 2024-03-24T09:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.943+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef01a49e0 con 0x7f0ef0100630 2024-03-24T09:28:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.945+0000 7f0eef7fe640 1 -- 172.21.15.116:0/1504941214 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f0ee0005020 con 0x7f0ef0100630 2024-03-24T09:28:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 Processor -- start 2024-03-24T09:28:43.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 -- start start 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0eef7fe640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ed80a2230 0x7f0ed80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4aa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4ba460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56477c418530 con 0x56477c4aa140 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56477c415a60 con 0x56477c4aaaf0 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0efb1be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56477c411240 con 0x56477c4b8070 2024-03-24T09:28:43.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ed80a2230 0x7f0ed80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0ef5b7d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0ef6b7f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4ba460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49658/0 (socket says 172.21.15.116:49658) 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0ef637e640 1 -- 172.21.15.116:0/4218406455 learned_addr learned my addr 172.21.15.116:0/4218406455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.946+0000 7f0eef7fe640 1 -- 172.21.15.116:0/1504941214 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0ee0014070 con 0x7f0ef0100630 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/4218406455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 msgr2=0x56477c4ba460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4ba460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/4218406455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 msgr2=0x56477c4aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/4218406455 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56477c40f8c0 con 0x56477c4aaaf0 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef637e640 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ed80a2230 0x7f0ed80a46f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80128e0 tx=0x7f0ee8009a80 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b7b30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0ee402c170 tx=0x7f0ee4005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=802691ff47ddffb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0eecbfe640 1 -- 172.21.15.116:0/4218406455 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4004880 con 0x56477c4aaaf0 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0eecbfe640 1 -- 172.21.15.116:0/4218406455 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ee402cc20 con 0x56477c4aaaf0 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.947+0000 7f0eecbfe640 1 -- 172.21.15.116:0/4218406455 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4003850 con 0x56477c4aaaf0 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 -- 172.21.15.116:0/4218406455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 msgr2=0x56477c4b7b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b7b30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0ee402c170 tx=0x7f0ee4005e10 comp rx=0 tx=0).stop 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 -- 172.21.15.116:0/4218406455 shutdown_connections 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4ba460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b7b30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 --2- 172.21.15.116:0/4218406455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 -- 172.21.15.116:0/4218406455 >> 172.21.15.116:0/4218406455 conn(0x56477c4a4e90 msgr2=0x56477c4a5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 -- 172.21.15.116:0/4218406455 shutdown_connections 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 -- 172.21.15.116:0/4218406455 wait complete. 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 -- start start 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.948+0000 7f0efb1be740 1 Processor -- start 2024-03-24T09:28:43.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56477c3b5f80 con 0x56477c4aa140 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56477c0e44b0 con 0x56477c4aaaf0 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56477c396ad0 con 0x56477c4b8070 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 10 client.0 _ll_register_callbacks cb 0x56477c3c9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef5b7d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef5b7d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46576/0 (socket says 172.21.15.116:46576) 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49670/0 (socket says 172.21.15.116:49670) 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/1052691722 learned_addr learned my addr 172.21.15.116:0/1052691722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 -- 172.21.15.116:0/1052691722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 msgr2=0x56477c4b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 -- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 msgr2=0x56477c4b55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 -- 172.21.15.116:0/1052691722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef006a0a0 con 0x56477c4aa140 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:43.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4b50a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0ee009b1e0 tx=0x7f0ee009ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28be5f5231cf30cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ed5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/1052691722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0078460 con 0x56477c4aa140 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/1052691722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0ee009b250 con 0x56477c4aa140 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56477c4357a0 con 0x56477c4aa140 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/1052691722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0ee009f3a0 con 0x56477c4aa140 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 10 client.17583 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.949+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56477c3c76d0 con 0x56477c4aa140 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/1052691722 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56477c4357a0 con 0x56477c4aa140 2024-03-24T09:28:43.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/1052691722 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0ee001f060 con 0x56477c4aa140 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0efb1be740 10 client.17583 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0efb1be740 10 client.17583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0efb1be740 20 client.17583 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c4c8180 0x56477c4ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0efb1be740 4 client.17583 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56477c4071f0 con 0x56477c4c8180 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c4c8180 0x56477c4ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c4c8180 0x56477c4ca5b0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4967fbd56dee9fd server_cookie=cebaaaad474f8332 in_seq=0 out_seq=0 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.950+0000 7f0ed5ffb640 10 client.17583 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:43.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.951+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/1052691722 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56477c4071f0 con 0x56477c4c8180 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.951+0000 7f0ed5ffb640 10 client.17583 handle_command_reply: tid=0 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.951+0000 7f0efb1be740 1 client.17583 shutdown 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 msgr2=0x56477c4b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4b50a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f0ee009b1e0 tx=0x7f0ee009ace0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 shutdown_connections 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4b5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c4c8180 0x56477c4ca5b0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0ed5ffb640 0 client.17583 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1052691722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4b50a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 >> 172.21.15.116:0/1052691722 conn(0x56477c4a4e90 msgr2=0x56477c4a6be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 shutdown_connections 2024-03-24T09:28:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 1 -- 172.21.15.116:0/1052691722 wait complete. 2024-03-24T09:28:43.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.952+0000 7f0efb1be740 20 client.17583 trim_cache size 0 max 16384 2024-03-24T09:28:43.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 Processor -- start 2024-03-24T09:28:43.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 -- start start 2024-03-24T09:28:43.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56477c4ab540 con 0x56477c4aa140 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56477c4a3350 con 0x56477c4aaaf0 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0efb1be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56477c4ba460 con 0x56477c4b8070 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4c38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49674/0 (socket says 172.21.15.116:49674) 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 -- 172.21.15.116:0/1040707292 learned_addr learned my addr 172.21.15.116:0/1040707292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:43.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 -- 172.21.15.116:0/1040707292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 msgr2=0x56477c4cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 -- 172.21.15.116:0/1040707292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 msgr2=0x56477c4b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.969+0000 7f0ef637e640 1 -- 172.21.15.116:0/1040707292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56477c3abc60 con 0x56477c4aa140 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0ef637e640 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4c38b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000cba0 tx=0x7f0ee0009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac4fa3be90069d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0ed7fff640 1 -- 172.21.15.116:0/1040707292 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0077070 con 0x56477c4aa140 2024-03-24T09:28:43.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0ed7fff640 1 -- 172.21.15.116:0/1040707292 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0ee00052d0 con 0x56477c4aa140 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0ed7fff640 1 -- 172.21.15.116:0/1040707292 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee00978c0 con 0x56477c4aa140 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 -- 172.21.15.116:0/1040707292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 msgr2=0x56477c4c38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4c38b0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000cba0 tx=0x7f0ee0009200 comp rx=0 tx=0).stop 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 -- 172.21.15.116:0/1040707292 shutdown_connections 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4b8070 0x56477c4cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4aaaf0 0x56477c4b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 --2- 172.21.15.116:0/1040707292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aa140 0x56477c4c38b0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 -- 172.21.15.116:0/1040707292 >> 172.21.15.116:0/1040707292 conn(0x56477c4a4e90 msgr2=0x56477c4a3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 -- 172.21.15.116:0/1040707292 shutdown_connections 2024-03-24T09:28:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.970+0000 7f0efb1be740 1 -- 172.21.15.116:0/1040707292 wait complete. 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 -- start start 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 Processor -- start 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4aa140 0x56477c4bbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 0x56477c4bc090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4b8070 0x56477c52d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56477c4bc860 con 0x56477c4aaaf0 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56477c4bc9d0 con 0x56477c4b8070 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56477c52da60 con 0x56477c4aa140 2024-03-24T09:28:43.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0efb1be740 10 client.0 _ll_register_callbacks cb 0x7f0ee009ae40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 0x56477c4bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4aa140 0x56477c4bbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef6b7f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4b8070 0x56477c52d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef637e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4aa140 0x56477c4bbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:39894/0 (socket says 172.21.15.116:39894) 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 0x56477c4bc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49686/0 (socket says 172.21.15.116:49686) 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef637e640 1 -- 172.21.15.116:0/2943603054 learned_addr learned my addr 172.21.15.116:0/2943603054 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/2943603054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4aa140 msgr2=0x56477c4bbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4aa140 0x56477c4bbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4b8070 msgr2=0x56477c52d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4b8070 0x56477c52d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 -- 172.21.15.116:0/2943603054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56477c3c76d0 con 0x56477c4aaaf0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef6b7f640 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4b8070 0x56477c52d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ef5b7d640 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 0x56477c4bc090 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4005ec0 tx=0x7f0ee4004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bfc8b9b72aa089 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.971+0000 7f0ed5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/2943603054 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee402c770 con 0x56477c4aaaf0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/2943603054 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f0ee404f040 con 0x56477c4aaaf0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56477c4a5810 con 0x56477c4aaaf0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/2943603054 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0ee402c8d0 con 0x56477c4aaaf0 2024-03-24T09:28:43.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 10 client.17589 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56477c3d1970 con 0x56477c4aaaf0 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/2943603054 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56477c4a5810 con 0x56477c4aaaf0 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/2943603054 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0ee403c050 con 0x56477c4aaaf0 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 10 client.17589 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 10 client.17589 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 20 client.17589 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c52ef90 0x56477c5313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 4 client.17589 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f0ee402cc20 con 0x56477c52ef90 2024-03-24T09:28:43.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.972+0000 7f0ef637e640 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c52ef90 0x56477c5313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.973+0000 7f0ef637e640 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c52ef90 0x56477c5313b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ceacf51e3f93fcb server_cookie=b8b8c5597268a30a in_seq=0 out_seq=0 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.973+0000 7f0ed5ffb640 10 client.17589 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ed5ffb640 1 -- 172.21.15.116:0/2943603054 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f0ee402cc20 con 0x56477c52ef90 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ed5ffb640 10 client.17589 handle_command_reply: tid=0 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 client.17589 shutdown 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 msgr2=0x56477c4bc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 0x56477c4bc090 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4005ec0 tx=0x7f0ee4004500 comp rx=0 tx=0).stop 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 shutdown_connections 2024-03-24T09:28:43.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56477c52ef90 0x56477c5313b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56477c4b8070 0x56477c52d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56477c4aaaf0 0x56477c4bc090 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ed5ffb640 0 client.17589 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 --2- 172.21.15.116:0/2943603054 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56477c4aa140 0x56477c4bbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 >> 172.21.15.116:0/2943603054 conn(0x56477c4a4e90 msgr2=0x56477c4a62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 shutdown_connections 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 1 -- 172.21.15.116:0/2943603054 wait complete. 2024-03-24T09:28:43.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0efb1be740 20 client.17589 trim_cache size 0 max 16384 2024-03-24T09:28:43.984 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:28:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:28:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "bd2a6e11-ea15-4686-a950-39ee1a7093a5", 2024-03-24T09:28:43.984 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:28:43.984 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ed80a2230 msgr2=0x7f0ed80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ed80a2230 0x7f0ed80a46f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80128e0 tx=0x7f0ee8009a80 comp rx=0 tx=0).stop 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 msgr2=0x7f0ef019d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef019d940 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000eb90 tx=0x7f0ee000c6a0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 shutdown_connections 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f0ed80a2230 0x7f0ed80a46f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0ef0106640 0x7f0ef0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ef0100ff0 0x7f0ef019de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 --2- 172.21.15.116:0/1504941214 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0100630 0x7f0ef019d940 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.974+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 >> 172.21.15.116:0/1504941214 conn(0x7f0ef0077770 msgr2=0x7f0ef0103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.975+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 shutdown_connections 2024-03-24T09:28:43.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:43.975+0000 7f0ef85e2640 1 -- 172.21.15.116:0/1504941214 wait complete. 2024-03-24T09:28:43.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:28:43.987 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:28:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 -- 172.21.15.116:0/9526629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f70068910 msgr2=0x7f5f70068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 --2- 172.21.15.116:0/9526629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f70068910 0x7f5f70068d90 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009930 tx=0x7f5f6402bc30 comp rx=0 tx=0).stop 2024-03-24T09:28:44.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 -- 172.21.15.116:0/9526629 shutdown_connections 2024-03-24T09:28:44.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 --2- 172.21.15.116:0/9526629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f700692d0 0x7f5f7010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 --2- 172.21.15.116:0/9526629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f70068910 0x7f5f70068d90 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 --2- 172.21.15.116:0/9526629 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f70107760 0x7f5f70107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.106+0000 7f5f75223640 1 -- 172.21.15.116:0/9526629 >> 172.21.15.116:0/9526629 conn(0x7f5f70077710 msgr2=0x7f5f70077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:44.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.107+0000 7f5f75223640 1 -- 172.21.15.116:0/9526629 shutdown_connections 2024-03-24T09:28:44.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.107+0000 7f5f75223640 1 -- 172.21.15.116:0/9526629 wait complete. 2024-03-24T09:28:44.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.107+0000 7f5f75223640 1 Processor -- start 2024-03-24T09:28:44.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 -- start start 2024-03-24T09:28:44.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f70068910 0x7f5f70199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 0x7f5f7019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f70107760 0x7f5f7019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7019ecc0 con 0x7f5f700692d0 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7019ee30 con 0x7f5f70107760 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f75223640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7019eff0 con 0x7f5f70068910 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 0x7f5f7019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 0x7f5f7019a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49710/0 (socket says 172.21.15.116:49710) 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 -- 172.21.15.116:0/604330729 learned_addr learned my addr 172.21.15.116:0/604330729 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f70068910 0x7f5f70199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f70107760 0x7f5f7019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 -- 172.21.15.116:0/604330729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f70068910 msgr2=0x7f5f70199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f70068910 0x7f5f70199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 -- 172.21.15.116:0/604330729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f70107760 msgr2=0x7f5f7019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f70107760 0x7f5f7019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6e59c640 1 -- 172.21.15.116:0/604330729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f64009560 con 0x7f5f700692d0 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.108+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f70107760 0x7f5f7019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.109+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 0x7f5f7019a340 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009230 tx=0x7f5f64004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.109+0000 7f5f57fff640 1 -- 172.21.15.116:0/604330729 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64004440 con 0x7f5f700692d0 2024-03-24T09:28:44.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.109+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7019f270 con 0x7f5f700692d0 2024-03-24T09:28:44.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.109+0000 7f5f57fff640 1 -- 172.21.15.116:0/604330729 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f64003aa0 con 0x7f5f700692d0 2024-03-24T09:28:44.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.109+0000 7f5f57fff640 1 -- 172.21.15.116:0/604330729 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f64005510 con 0x7f5f700692d0 2024-03-24T09:28:44.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.109+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f701a6b50 con 0x7f5f700692d0 2024-03-24T09:28:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.112+0000 7f5f57fff640 1 -- 172.21.15.116:0/604330729 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f5f64003c10 con 0x7f5f700692d0 2024-03-24T09:28:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f77e03740 1 Processor -- start 2024-03-24T09:28:44.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f77e03740 1 -- start start 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f57fff640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5f580a22b0 0x7f5f580a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 0x55779c8165b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c81f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c81f6d0 0x55779c821ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f57fff640 1 -- 172.21.15.116:0/604330729 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5f640f0450 con 0x7f5f700692d0 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5f580a22b0 0x7f5f580a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f6f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c81f6d0 0x55779c821ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 0x55779c8165b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f6f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c81f6d0 0x55779c821ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49714/0 (socket says 172.21.15.116:49714) 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.113+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 0x55779c8165b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46644/0 (socket says 172.21.15.116:46644) 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6f59e640 1 -- 172.21.15.116:0/1783337351 learned_addr learned my addr 172.21.15.116:0/1783337351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55779c73af30 con 0x55779c81f6d0 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55779c784810 con 0x55779c8161d0 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55779c781d40 con 0x55779c816af0 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c81f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1783337351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 msgr2=0x55779c81f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c81f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1783337351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c81f6d0 msgr2=0x55779c821ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c81f6d0 0x55779c821ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1783337351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55779c73eea0 con 0x55779c8161d0 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5f580a22b0 0x7f5f580a4770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f68013c80 tx=0x7f5f68007f00 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.114+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 0x55779c8165b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5f60017fd0 tx=0x7f5f60009c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56b52db246c2eca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f557fa640 1 -- 172.21.15.116:0/1783337351 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6001c070 con 0x55779c8161d0 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f557fa640 1 -- 172.21.15.116:0/1783337351 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f60004500 con 0x55779c8161d0 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f557fa640 1 -- 172.21.15.116:0/1783337351 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f60003910 con 0x55779c8161d0 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 msgr2=0x55779c8165b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 0x55779c8165b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5f60017fd0 tx=0x7f5f60009c10 comp rx=0 tx=0).stop 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 shutdown_connections 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c81f6d0 0x55779c821ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c81f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1783337351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c8161d0 0x55779c8165b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.115+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 >> 172.21.15.116:0/1783337351 conn(0x55779c810900 msgr2=0x55779c810ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.116+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 shutdown_connections 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.116+0000 7f5f77e03740 1 -- 172.21.15.116:0/1783337351 wait complete. 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.116+0000 7f5f77e03740 1 -- start start 2024-03-24T09:28:44.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.116+0000 7f5f77e03740 1 Processor -- start 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c81c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c81d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55779c77d520 con 0x55779c8161d0 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55779c450430 con 0x55779c816af0 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55779c40ce00 con 0x55779c81f6d0 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f77e03740 10 client.0 _ll_register_callbacks cb 0x55779c71bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c81c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c81c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49728/0 (socket says 172.21.15.116:49728) 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/3866912557 learned_addr learned my addr 172.21.15.116:0/3866912557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c81d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/3866912557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 msgr2=0x55779c81d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c81d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 msgr2=0x55779c81cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/3866912557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f60017c80 con 0x55779c8161d0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.117+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c81c6a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5f600096e0 tx=0x7f5f6000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b213344f4dd405 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f52ff5640 1 -- 172.21.15.116:0/3866912557 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6001c030 con 0x55779c8161d0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f52ff5640 1 -- 172.21.15.116:0/3866912557 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5f600038f0 con 0x55779c8161d0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55779c4b9720 con 0x55779c8161d0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f52ff5640 1 -- 172.21.15.116:0/3866912557 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5f60009ec0 con 0x55779c8161d0 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f77e03740 10 client.17601 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:44.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55779c77bba0 con 0x55779c8161d0 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.118+0000 7f5f52ff5640 1 -- 172.21.15.116:0/3866912557 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55779c4b9720 con 0x55779c8161d0 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f52ff5640 1 -- 172.21.15.116:0/3866912557 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5f60021bd0 con 0x55779c8161d0 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f77e03740 10 client.17601 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f77e03740 10 client.17601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f77e03740 20 client.17601 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f77e03740 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c82fb10 0x55779c831f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f77e03740 4 client.17601 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55779c508890 con 0x55779c82fb10 2024-03-24T09:28:44.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c82fb10 0x55779c831f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f52ff5640 10 client.17601 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.119+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c82fb10 0x55779c831f50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a65fadf90db16cd3 server_cookie=eaa443dce23977b4 in_seq=0 out_seq=0 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.120+0000 7f5f52ff5640 1 -- 172.21.15.116:0/3866912557 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55779c508890 con 0x55779c82fb10 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.120+0000 7f5f52ff5640 10 client.17601 handle_command_reply: tid=0 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 client.17601 shutdown 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 msgr2=0x55779c81c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c81c6a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f5f600096e0 tx=0x7f5f6000ec40 comp rx=0 tx=0).stop 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 shutdown_connections 2024-03-24T09:28:44.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c81d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c82fb10 0x55779c831f50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 --2- 172.21.15.116:0/3866912557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c81c6a0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f52ff5640 0 client.17601 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 >> 172.21.15.116:0/3866912557 conn(0x55779c810900 msgr2=0x55779c812650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 shutdown_connections 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 1 -- 172.21.15.116:0/3866912557 wait complete. 2024-03-24T09:28:44.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.121+0000 7f5f77e03740 20 client.17601 trim_cache size 0 max 16384 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.145+0000 7f5f77e03740 1 Processor -- start 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.145+0000 7f5f77e03740 1 -- start start 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c82af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c837820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f77e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55779c702430 con 0x55779c8161d0 2024-03-24T09:28:44.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f77e03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55779c82c580 con 0x55779c816af0 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f77e03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55779c82b4d0 con 0x55779c81f6d0 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c82af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c82af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49730/0 (socket says 172.21.15.116:49730) 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1094322248 learned_addr learned my addr 172.21.15.116:0/1094322248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c837820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1094322248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 msgr2=0x55779c837820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c837820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1094322248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 msgr2=0x55779c81cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 -- 172.21.15.116:0/1094322248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55779c717b20 con 0x55779c8161d0 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c82af60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5f60017c50 tx=0x7f5f6000e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27312a16fcc09b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f54ff9640 1 -- 172.21.15.116:0/1094322248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6003f070 con 0x55779c8161d0 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f54ff9640 1 -- 172.21.15.116:0/1094322248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f6001d040 con 0x55779c8161d0 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.146+0000 7f5f54ff9640 1 -- 172.21.15.116:0/1094322248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6001c070 con 0x55779c8161d0 2024-03-24T09:28:44.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 -- 172.21.15.116:0/1094322248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 msgr2=0x55779c82af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c82af60 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f5f60017c50 tx=0x7f5f6000e8d0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 -- 172.21.15.116:0/1094322248 shutdown_connections 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c81f6d0 0x55779c837820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c816af0 0x55779c81cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 --2- 172.21.15.116:0/1094322248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c82af60 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 -- 172.21.15.116:0/1094322248 >> 172.21.15.116:0/1094322248 conn(0x55779c810900 msgr2=0x55779c80fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 -- 172.21.15.116:0/1094322248 shutdown_connections 2024-03-24T09:28:44.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.147+0000 7f5f77e03740 1 -- 172.21.15.116:0/1094322248 wait complete. 2024-03-24T09:28:44.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 -- start start 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 Processor -- start 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c827940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c822c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 0x55779c823220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55779c823920 con 0x55779c8161d0 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55779c823a90 con 0x55779c81f6d0 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55779c894e50 con 0x55779c816af0 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f77e03740 10 client.0 _ll_register_callbacks cb 0x7f5f68007da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:28:44.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c827940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 0x55779c823220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 0x55779c823220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46674/0 (socket says 172.21.15.116:46674) 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 -- 172.21.15.116:0/2591494243 learned_addr learned my addr 172.21.15.116:0/2591494243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.148+0000 7f5f6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c827940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49736/0 (socket says 172.21.15.116:49736) 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6e59c640 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c822c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 -- 172.21.15.116:0/2591494243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 msgr2=0x55779c822c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c822c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 -- 172.21.15.116:0/2591494243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 msgr2=0x55779c827940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c827940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 -- 172.21.15.116:0/2591494243 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55779c77bba0 con 0x55779c81f6d0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f6f59e640 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 0x55779c823220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5f68003bf0 tx=0x7f5f680072a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95e95c853fcf7468 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.149+0000 7f5f52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.150+0000 7f5f52ff5640 1 -- 172.21.15.116:0/2591494243 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f680047f0 con 0x55779c81f6d0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.150+0000 7f5f52ff5640 1 -- 172.21.15.116:0/2591494243 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9466+0+0 (secure 0 0 0) 0x7f5f68004b20 con 0x55779c81f6d0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.150+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55779c47c7e0 con 0x55779c81f6d0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.150+0000 7f5f52ff5640 1 -- 172.21.15.116:0/2591494243 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5f68004ce0 con 0x55779c81f6d0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.150+0000 7f5f77e03740 10 client.27428 fetch_fsmap learned FSMap version 35 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.150+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55779c73d700 con 0x55779c81f6d0 2024-03-24T09:28:44.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f52ff5640 1 -- 172.21.15.116:0/2591494243 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5f68024070 con 0x55779c81f6d0 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f52ff5640 1 -- 172.21.15.116:0/2591494243 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f5f68005020 con 0x55779c81f6d0 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f77e03740 10 client.27428 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f77e03740 10 client.27428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f77e03740 20 client.27428 populate_metadata read hostname 'smithi116' 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f77e03740 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c8966c0 0x55779c898ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f77e03740 4 client.27428 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55779c47c7e0 con 0x55779c8966c0 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.151+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c8966c0 0x55779c898ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f52ff5640 10 client.27428 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:44.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f6ed9d640 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c8966c0 0x55779c898ae0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d273f6a32cbe1c7a server_cookie=ab0f1f982f8ca697 in_seq=0 out_seq=0 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f52ff5640 1 -- 172.21.15.116:0/2591494243 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55779c47c7e0 con 0x55779c8966c0 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f52ff5640 10 client.27428 handle_command_reply: tid=0 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 client.27428 shutdown 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 msgr2=0x55779c823220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 0x55779c823220 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5f68003bf0 tx=0x7f5f680072a0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 shutdown_connections 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55779c8966c0 0x55779c898ae0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55779c81f6d0 0x55779c823220 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55779c816af0 0x55779c822c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 --2- 172.21.15.116:0/2591494243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55779c8161d0 0x55779c827940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 >> 172.21.15.116:0/2591494243 conn(0x55779c810900 msgr2=0x55779c811ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f52ff5640 0 client.27428 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 shutdown_connections 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 1 -- 172.21.15.116:0/2591494243 wait complete. 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.152+0000 7f5f77e03740 20 client.27428 trim_cache size 0 max 16384 2024-03-24T09:28:44.161 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: "bd2a6e11-ea15-4686-a950-39ee1a7093a5": { 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "bd2a6e11-ea15-4686-a950-39ee1a7093a5", 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:28:44.162 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:28:44.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5f580a22b0 msgr2=0x7f5f580a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5f580a22b0 0x7f5f580a4770 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5f68013c80 tx=0x7f5f68007f00 comp rx=0 tx=0).stop 2024-03-24T09:28:44.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 msgr2=0x7f5f7019a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:28:44.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 0x7f5f7019a340 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009230 tx=0x7f5f64004040 comp rx=0 tx=0).stop 2024-03-24T09:28:44.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 shutdown_connections 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f70107760 0x7f5f7019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f700692d0 0x7f5f7019a340 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f5f580a22b0 0x7f5f580a4770 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 --2- 172.21.15.116:0/604330729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5f70068910 0x7f5f70199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 >> 172.21.15.116:0/604330729 conn(0x7f5f70077710 msgr2=0x7f5f7010ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 shutdown_connections 2024-03-24T09:28:44.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:28:44.153+0000 7f5f75223640 1 -- 172.21.15.116:0/604330729 wait complete. 2024-03-24T09:28:44.165 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:44 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:28:44.165 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:44 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:44.165 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:44 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:44.165 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:44 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:44.165 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:44 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:44.166 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:44 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:44.166 INFO:tasks.cephfs.filesystem:scrub status for tag:bd2a6e11-ea15-4686-a950-39ee1a7093a5 - {'path': '/', 'tag': 'bd2a6e11-ea15-4686-a950-39ee1a7093a5', 'options': 'recursive,force'} 2024-03-24T09:28:44.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:44 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:28:44.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:44 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:44.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:44 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:44.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:44 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:44.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:44 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:44.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:44 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:44.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:44 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:28:44.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:44 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:44.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:44 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:44.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:44 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:44.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:44 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:44.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:44 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:45.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: pgmap v202: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:45 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: pgmap v202: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:45 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: pgmap v202: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:45.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:45 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:28:47.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:47 smithi143 ceph-mon[35965]: pgmap v203: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:47.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:47 smithi117 ceph-mon[35788]: pgmap v203: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:47.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:47 smithi116 ceph-mon[32207]: pgmap v203: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:28:49.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:49 smithi143 ceph-mon[35965]: pgmap v204: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:49.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:49 smithi117 ceph-mon[35788]: pgmap v204: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:49.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:49 smithi116 ceph-mon[32207]: pgmap v204: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:51.367 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:51 smithi117 ceph-mon[35788]: pgmap v205: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:51 smithi143 ceph-mon[35965]: pgmap v205: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:51.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:51 smithi116 ceph-mon[32207]: pgmap v205: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:53 smithi143 ceph-mon[35965]: pgmap v206: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:53.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:53 smithi117 ceph-mon[35788]: pgmap v206: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:53.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:53 smithi116 ceph-mon[32207]: pgmap v206: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:55 smithi143 ceph-mon[35965]: pgmap v207: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:55.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:55 smithi117 ceph-mon[35788]: pgmap v207: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:55.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:55 smithi116 ceph-mon[32207]: pgmap v207: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:57 smithi143 ceph-mon[35965]: pgmap v208: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:57.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:57 smithi117 ceph-mon[35788]: pgmap v208: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:57 smithi116 ceph-mon[32207]: pgmap v208: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:28:58.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:58.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:58 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:58.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:58.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:58 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:58.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:58.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:58 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:59.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:59 smithi143 ceph-mon[35965]: pgmap v209: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:59.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:59 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:59.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:59 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:59.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:28:59 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:28:59.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:59 smithi117 ceph-mon[35788]: pgmap v209: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:59.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:59 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:59.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:59 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:59.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:28:59 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:28:59.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:59 smithi116 ceph-mon[32207]: pgmap v209: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:28:59.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:59 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:59.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:59 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:28:59.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:28:59 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:29:01.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:01 smithi143 ceph-mon[35965]: pgmap v210: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:01 smithi117 ceph-mon[35788]: pgmap v210: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:01 smithi116 ceph-mon[32207]: pgmap v210: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:03.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:03 smithi143 ceph-mon[35965]: pgmap v211: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:29:03.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:03 smithi117 ceph-mon[35788]: pgmap v211: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:29:03.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:03 smithi116 ceph-mon[32207]: pgmap v211: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:29:05.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:05 smithi143 ceph-mon[35965]: pgmap v212: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:29:05.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:05 smithi117 ceph-mon[35788]: pgmap v212: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:29:05.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:05 smithi116 ceph-mon[32207]: pgmap v212: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:29:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:07 smithi143 ceph-mon[35965]: pgmap v213: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:29:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:07 smithi117 ceph-mon[35788]: pgmap v213: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:29:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:07 smithi116 ceph-mon[32207]: pgmap v213: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:29:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:09 smithi143 ceph-mon[35965]: pgmap v214: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:29:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:09 smithi143 ceph-mon[35965]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T09:29:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:09 smithi117 ceph-mon[35788]: pgmap v214: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:29:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:09 smithi117 ceph-mon[35788]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T09:29:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:09 smithi116 ceph-mon[32207]: pgmap v214: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:29:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:09 smithi116 ceph-mon[32207]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T09:29:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:11 smithi143 ceph-mon[35965]: pgmap v216: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:29:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:11 smithi117 ceph-mon[35788]: pgmap v216: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:29:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:11 smithi116 ceph-mon[32207]: pgmap v216: 129 pgs: 129 active+clean; 152 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:29:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:13 smithi143 ceph-mon[35965]: pgmap v217: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:13 smithi117 ceph-mon[35788]: pgmap v217: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:13 smithi116 ceph-mon[32207]: pgmap v217: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:14.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:29:14.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:29:14.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 -- 172.21.15.116:0/1478546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100ff0 msgr2=0x7f716c10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 --2- 172.21.15.116:0/1478546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100ff0 0x7f716c10d500 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f7164009cd0 tx=0x7f716402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 -- 172.21.15.116:0/1478546316 shutdown_connections 2024-03-24T09:29:14.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 --2- 172.21.15.116:0/1478546316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100ff0 0x7f716c10d500 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 --2- 172.21.15.116:0/1478546316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716c100630 0x7f716c100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 --2- 172.21.15.116:0/1478546316 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f716c106640 0x7f716c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 -- 172.21.15.116:0/1478546316 >> 172.21.15.116:0/1478546316 conn(0x7f716c0777a0 msgr2=0x7f716c077bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 -- 172.21.15.116:0/1478546316 shutdown_connections 2024-03-24T09:29:14.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 -- 172.21.15.116:0/1478546316 wait complete. 2024-03-24T09:29:14.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.282+0000 7f717428f640 1 Processor -- start 2024-03-24T09:29:14.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 -- start start 2024-03-24T09:29:14.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 0x7f716c198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f716c100ff0 0x7f716c199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716c106640 0x7f716c1a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f716c199890 con 0x7f716c100630 2024-03-24T09:29:14.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1a40b0 con 0x7f716c106640 2024-03-24T09:29:14.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1a41f0 con 0x7f716c100ff0 2024-03-24T09:29:14.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 0x7f716c198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 0x7f716c198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58562/0 (socket says 172.21.15.116:58562) 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 -- 172.21.15.116:0/3658094091 learned_addr learned my addr 172.21.15.116:0/3658094091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717182a640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f716c100ff0 0x7f716c199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717282c640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716c106640 0x7f716c1a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 -- 172.21.15.116:0/3658094091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f716c100ff0 msgr2=0x7f716c199030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f716c100ff0 0x7f716c199030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 -- 172.21.15.116:0/3658094091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716c106640 msgr2=0x7f716c1a3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716c106640 0x7f716c1a3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 -- 172.21.15.116:0/3658094091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7164009900 con 0x7f716c100630 2024-03-24T09:29:14.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717202b640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 0x7f716c198af0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f716800e9f0 tx=0x7f716800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f71637fe640 1 -- 172.21.15.116:0/3658094091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168009720 con 0x7f716c100630 2024-03-24T09:29:14.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c1a4450 con 0x7f716c100630 2024-03-24T09:29:14.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f71637fe640 1 -- 172.21.15.116:0/3658094091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f716800cd80 con 0x7f716c100630 2024-03-24T09:29:14.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.283+0000 7f71637fe640 1 -- 172.21.15.116:0/3658094091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7168010430 con 0x7f716c100630 2024-03-24T09:29:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.284+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c1a4960 con 0x7f716c100630 2024-03-24T09:29:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.285+0000 7f71637fe640 1 -- 172.21.15.116:0/3658094091 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7168005020 con 0x7f716c100630 2024-03-24T09:29:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 Processor -- start 2024-03-24T09:29:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 -- start start 2024-03-24T09:29:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec9235b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec92c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f71637fe640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f71580a2230 0x7f71580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aaec847f30 con 0x55aaec923af0 2024-03-24T09:29:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aaec891810 con 0x55aaec92c6d0 2024-03-24T09:29:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.286+0000 7f7176e72740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55aaec88ed40 con 0x55aaec9231d0 2024-03-24T09:29:14.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec9235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717202b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec92c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717182a640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f71580a2230 0x7f71580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec9235b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:38066/0 (socket says 172.21.15.116:38066) 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 -- 172.21.15.116:0/3142585747 learned_addr learned my addr 172.21.15.116:0/3142585747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f71637fe640 1 -- 172.21.15.116:0/3658094091 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f7168014070 con 0x7f716c100630 2024-03-24T09:29:14.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 -- 172.21.15.116:0/3142585747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 msgr2=0x55aaec92eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 -- 172.21.15.116:0/3142585747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 msgr2=0x55aaec92c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec92c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 -- 172.21.15.116:0/3142585747 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaec84bea0 con 0x55aaec9231d0 2024-03-24T09:29:14.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717202b640 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec92c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717182a640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f71580a2230 0x7f71580a46f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f716c06a7f0 tx=0x7f715c005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f717282c640 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec9235b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7164004af0 tx=0x7f716402e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df28dd4aeb10ce13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f7160bfe640 1 -- 172.21.15.116:0/3142585747 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640346d0 con 0x55aaec9231d0 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f7160bfe640 1 -- 172.21.15.116:0/3142585747 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71640325d0 con 0x55aaec9231d0 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.287+0000 7f7160bfe640 1 -- 172.21.15.116:0/3142585747 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164003b90 con 0x55aaec9231d0 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 -- 172.21.15.116:0/3142585747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 msgr2=0x55aaec9235b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec9235b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7164004af0 tx=0x7f716402e040 comp rx=0 tx=0).stop 2024-03-24T09:29:14.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 -- 172.21.15.116:0/3142585747 shutdown_connections 2024-03-24T09:29:14.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec92c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 --2- 172.21.15.116:0/3142585747 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec9235b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 -- 172.21.15.116:0/3142585747 >> 172.21.15.116:0/3142585747 conn(0x55aaec91d900 msgr2=0x55aaec91dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 -- 172.21.15.116:0/3142585747 shutdown_connections 2024-03-24T09:29:14.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 -- 172.21.15.116:0/3142585747 wait complete. 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.288+0000 7f7176e72740 1 -- start start 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 Processor -- start 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec929700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aaec88a520 con 0x55aaec923af0 2024-03-24T09:29:14.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aaec55d430 con 0x55aaec92c6d0 2024-03-24T09:29:14.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55aaec519e00 con 0x55aaec9231d0 2024-03-24T09:29:14.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7176e72740 10 client.0 _ll_register_callbacks cb 0x55aaec828d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:14.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58582/0 (socket says 172.21.15.116:58582) 2024-03-24T09:29:14.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717202b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec929700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 -- 172.21.15.116:0/1284661057 learned_addr learned my addr 172.21.15.116:0/1284661057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717282c640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 -- 172.21.15.116:0/1284661057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 msgr2=0x55aaec929700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec929700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 -- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 msgr2=0x55aaec92a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 -- 172.21.15.116:0/1284661057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c10de00 con 0x55aaec923af0 2024-03-24T09:29:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717282c640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f717182a640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929c40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f715c00ed00 tx=0x7f715c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5ae62ef411ee95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7155ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.289+0000 7f7155ffb640 1 -- 172.21.15.116:0/1284661057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c01ccb0 con 0x55aaec923af0 2024-03-24T09:29:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7155ffb640 1 -- 172.21.15.116:0/1284661057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f715c023070 con 0x55aaec923af0 2024-03-24T09:29:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aaec5897e0 con 0x55aaec923af0 2024-03-24T09:29:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7155ffb640 1 -- 172.21.15.116:0/1284661057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f715c01ce10 con 0x55aaec923af0 2024-03-24T09:29:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 10 client.17619 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aaec888ba0 con 0x55aaec923af0 2024-03-24T09:29:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7155ffb640 1 -- 172.21.15.116:0/1284661057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55aaec5897e0 con 0x55aaec923af0 2024-03-24T09:29:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7155ffb640 1 -- 172.21.15.116:0/1284661057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f715c01d4a0 con 0x55aaec923af0 2024-03-24T09:29:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 10 client.17619 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 10 client.17619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 20 client.17619 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec93cbc0 0x55aaec93efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 4 client.17619 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aaec5c6720 con 0x55aaec93cbc0 2024-03-24T09:29:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.290+0000 7f717202b640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec93cbc0 0x55aaec93efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.291+0000 7f7155ffb640 10 client.17619 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.291+0000 7f717202b640 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec93cbc0 0x55aaec93efe0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44b5dc2c6c3d75cb server_cookie=5450b0ada9bd797 in_seq=0 out_seq=0 2024-03-24T09:29:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.291+0000 7f7155ffb640 1 -- 172.21.15.116:0/1284661057 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aaec5c6720 con 0x55aaec93cbc0 2024-03-24T09:29:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.291+0000 7f7155ffb640 10 client.17619 handle_command_reply: tid=0 2024-03-24T09:29:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.291+0000 7f7176e72740 1 client.17619 shutdown 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 msgr2=0x55aaec929c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929c40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f715c00ed00 tx=0x7f715c004730 comp rx=0 tx=0).stop 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 shutdown_connections 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec93cbc0 0x55aaec93efe0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec92a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929c40 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7155ffb640 0 client.17619 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 --2- 172.21.15.116:0/1284661057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec929700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 >> 172.21.15.116:0/1284661057 conn(0x55aaec91d900 msgr2=0x55aaec91f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 shutdown_connections 2024-03-24T09:29:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 1 -- 172.21.15.116:0/1284661057 wait complete. 2024-03-24T09:29:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.292+0000 7f7176e72740 20 client.17619 trim_cache size 0 max 16384 2024-03-24T09:29:14.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 Processor -- start 2024-03-24T09:29:14.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 -- start start 2024-03-24T09:29:14.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec937f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec9448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aaec80f430 con 0x55aaec923af0 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aaec919550 con 0x55aaec92c6d0 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f7176e72740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55aaec917f70 con 0x55aaec9231d0 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.309+0000 7f717182a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717202b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec937f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58596/0 (socket says 172.21.15.116:58596) 2024-03-24T09:29:14.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 -- 172.21.15.116:0/3428800582 learned_addr learned my addr 172.21.15.116:0/3428800582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717202b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec937f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:38074/0 (socket says 172.21.15.116:38074) 2024-03-24T09:29:14.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717282c640 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec9448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 -- 172.21.15.116:0/3428800582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 msgr2=0x55aaec937f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec937f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 -- 172.21.15.116:0/3428800582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 msgr2=0x55aaec9448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec9448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 -- 172.21.15.116:0/3428800582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aaec824b20 con 0x55aaec923af0 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f717182a640 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929fa0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f715c009730 tx=0x7f715c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398fb286abe56e5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f7157fff640 1 -- 172.21.15.116:0/3428800582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c005b90 con 0x55aaec923af0 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f7157fff640 1 -- 172.21.15.116:0/3428800582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f715c021070 con 0x55aaec923af0 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.310+0000 7f7157fff640 1 -- 172.21.15.116:0/3428800582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c0276b0 con 0x55aaec923af0 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- 172.21.15.116:0/3428800582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 msgr2=0x55aaec929fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929fa0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f715c009730 tx=0x7f715c004600 comp rx=0 tx=0).stop 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- 172.21.15.116:0/3428800582 shutdown_connections 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec92c6d0 0x55aaec9448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec929fa0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- 172.21.15.116:0/3428800582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec9231d0 0x55aaec937f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- 172.21.15.116:0/3428800582 >> 172.21.15.116:0/3428800582 conn(0x55aaec91d900 msgr2=0x55aaec91cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- 172.21.15.116:0/3428800582 shutdown_connections 2024-03-24T09:29:14.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- 172.21.15.116:0/3428800582 wait complete. 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- start start 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 Processor -- start 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec9231d0 0x55aaec930570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec930ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec92c6d0 0x55aaec9a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aaec931250 con 0x55aaec923af0 2024-03-24T09:29:14.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aaec9a2470 con 0x55aaec9231d0 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55aaec9a2630 con 0x55aaec92c6d0 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f7176e72740 10 client.0 _ll_register_callbacks cb 0x7f715c009680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f717182a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec930ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f717182a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec930ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58612/0 (socket says 172.21.15.116:58612) 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f717182a640 1 -- 172.21.15.116:0/4252033382 learned_addr learned my addr 172.21.15.116:0/4252033382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f717282c640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec92c6d0 0x55aaec9a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.311+0000 7f717202b640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec9231d0 0x55aaec930570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f717182a640 1 -- 172.21.15.116:0/4252033382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec92c6d0 msgr2=0x55aaec9a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f717182a640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec92c6d0 0x55aaec9a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f717182a640 1 -- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec9231d0 msgr2=0x55aaec930570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f717182a640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec9231d0 0x55aaec930570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f717182a640 1 -- 172.21.15.116:0/4252033382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f715c01de70 con 0x55aaec923af0 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f717182a640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec930ab0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f715c00efd0 tx=0x7f715c00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b215ce67a2ef6447 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7155ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7155ffb640 1 -- 172.21.15.116:0/4252033382 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c005b90 con 0x55aaec923af0 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7155ffb640 1 -- 172.21.15.116:0/4252033382 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f715c023070 con 0x55aaec923af0 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71640325d0 con 0x55aaec923af0 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7155ffb640 1 -- 172.21.15.116:0/4252033382 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f715c01da50 con 0x55aaec923af0 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 10 client.17625 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55aaec84a700 con 0x55aaec923af0 2024-03-24T09:29:14.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7155ffb640 1 -- 172.21.15.116:0/4252033382 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f71640325d0 con 0x55aaec923af0 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7155ffb640 1 -- 172.21.15.116:0/4252033382 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f715c027cf0 con 0x55aaec923af0 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 10 client.17625 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 10 client.17625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 20 client.17625 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec9a3b90 0x55aaec9a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 4 client.17625 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:29:14.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.312+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aaec615890 con 0x55aaec9a3b90 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f717202b640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec9a3b90 0x55aaec9a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7155ffb640 10 client.17625 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f717202b640 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec9a3b90 0x55aaec9a5fb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c15bcc91de3aed62 server_cookie=23ca186dc2ebc029 in_seq=0 out_seq=0 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7155ffb640 1 -- 172.21.15.116:0/4252033382 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55aaec615890 con 0x55aaec9a3b90 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7155ffb640 10 client.17625 handle_command_reply: tid=0 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 client.17625 shutdown 2024-03-24T09:29:14.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 msgr2=0x55aaec930ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec930ab0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f715c00efd0 tx=0x7f715c00e9e0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 shutdown_connections 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55aaec92c6d0 0x55aaec9a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aaec923af0 0x55aaec930ab0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55aaec9a3b90 0x55aaec9a5fb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 --2- 172.21.15.116:0/4252033382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aaec9231d0 0x55aaec930570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7155ffb640 0 client.17625 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:14.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 >> 172.21.15.116:0/4252033382 conn(0x55aaec91d900 msgr2=0x55aaec91eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 shutdown_connections 2024-03-24T09:29:14.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 1 -- 172.21.15.116:0/4252033382 wait complete. 2024-03-24T09:29:14.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.313+0000 7f7176e72740 20 client.17625 trim_cache size 0 max 16384 2024-03-24T09:29:14.335 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:29:14.335 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:29:14.335 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:29:14.335 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f71580a2230 msgr2=0x7f71580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f71580a2230 0x7f71580a46f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f716c06a7f0 tx=0x7f715c005f90 comp rx=0 tx=0).stop 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 msgr2=0x7f716c198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 0x7f716c198af0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f716800e9f0 tx=0x7f716800edf0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 shutdown_connections 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f716c106640 0x7f716c1a3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f71580a2230 0x7f71580a46f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f716c100ff0 0x7f716c199030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 --2- 172.21.15.116:0/3658094091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f716c100630 0x7f716c198af0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 >> 172.21.15.116:0/3658094091 conn(0x7f716c0777a0 msgr2=0x7f716c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 shutdown_connections 2024-03-24T09:29:14.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.314+0000 7f717428f640 1 -- 172.21.15.116:0/3658094091 wait complete. 2024-03-24T09:29:14.338 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:29:14.339 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:29:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.439+0000 7f83bcd93640 1 -- 172.21.15.116:0/4106882217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b81066b0 msgr2=0x7f83b8106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.439+0000 7f83bcd93640 1 --2- 172.21.15.116:0/4106882217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b81066b0 0x7f83b8106a90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f83b0009ca0 tx=0x7f83b002c180 comp rx=0 tx=0).stop 2024-03-24T09:29:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 -- 172.21.15.116:0/4106882217 shutdown_connections 2024-03-24T09:29:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 --2- 172.21.15.116:0/4106882217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8100d40 0x7f83b810d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 --2- 172.21.15.116:0/4106882217 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b8106fd0 0x7f83b8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 --2- 172.21.15.116:0/4106882217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b81066b0 0x7f83b8106a90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 -- 172.21.15.116:0/4106882217 >> 172.21.15.116:0/4106882217 conn(0x7f83b8077710 msgr2=0x7f83b8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 -- 172.21.15.116:0/4106882217 shutdown_connections 2024-03-24T09:29:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 -- 172.21.15.116:0/4106882217 wait complete. 2024-03-24T09:29:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 Processor -- start 2024-03-24T09:29:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 -- start start 2024-03-24T09:29:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.440+0000 7f83bcd93640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b8100d40 0x7f83b819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 0x7f83b819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b8106fd0 0x7f83b81969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83b8197020 con 0x7f83b81066b0 2024-03-24T09:29:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f83b8197190 con 0x7f83b8100d40 2024-03-24T09:29:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f83b8197350 con 0x7f83b8106fd0 2024-03-24T09:29:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 0x7f83b819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b8100d40 0x7f83b819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 0x7f83b819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58618/0 (socket says 172.21.15.116:58618) 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 -- 172.21.15.116:0/3754863973 learned_addr learned my addr 172.21.15.116:0/3754863973 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b6d9d640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b8106fd0 0x7f83b81969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 -- 172.21.15.116:0/3754863973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b8106fd0 msgr2=0x7f83b81969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b8106fd0 0x7f83b81969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 -- 172.21.15.116:0/3754863973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b8100d40 msgr2=0x7f83b819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b8100d40 0x7f83b819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 -- 172.21.15.116:0/3754863973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b0009900 con 0x7f83b81066b0 2024-03-24T09:29:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b659c640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b8100d40 0x7f83b819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83b5d9b640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 0x7f83b819cd60 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f83a000e9c0 tx=0x7f83a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a00096c0 con 0x7f83b81066b0 2024-03-24T09:29:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b8197630 con 0x7f83b81066b0 2024-03-24T09:29:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83a00054d0 con 0x7f83b81066b0 2024-03-24T09:29:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a0010430 con 0x7f83b81066b0 2024-03-24T09:29:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.441+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b81a38c0 con 0x7f83b81066b0 2024-03-24T09:29:14.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.442+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b8068500 con 0x7f83b81066b0 2024-03-24T09:29:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.443+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f83a0004f90 con 0x7f83b81066b0 2024-03-24T09:29:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.444+0000 7f83af7fe640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83a40a2230 0x7f83a40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.444+0000 7f83b659c640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83a40a2230 0x7f83a40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.444+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f83a0014070 con 0x7f83b81066b0 2024-03-24T09:29:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.444+0000 7f83b659c640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83a40a2230 0x7f83a40a46f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f83b0005cc0 tx=0x7f83b002e040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.448+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f83a0097090 con 0x7f83b81066b0 2024-03-24T09:29:14.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.658+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83b8197f00 con 0x7f83b81066b0 2024-03-24T09:29:14.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.659+0000 7f83af7fe640 1 -- 172.21.15.116:0/3754863973 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f83a009a750 con 0x7f83b81066b0 2024-03-24T09:29:14.660 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:29:14.660 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:29:14.661 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:29:14.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83a40a2230 msgr2=0x7f83a40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83a40a2230 0x7f83a40a46f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f83b0005cc0 tx=0x7f83b002e040 comp rx=0 tx=0).stop 2024-03-24T09:29:14.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 msgr2=0x7f83b819cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 0x7f83b819cd60 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f83a000e9c0 tx=0x7f83a000ee80 comp rx=0 tx=0).stop 2024-03-24T09:29:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 shutdown_connections 2024-03-24T09:29:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83a40a2230 0x7f83a40a46f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83b8106fd0 0x7f83b81969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b81066b0 0x7f83b819cd60 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 --2- 172.21.15.116:0/3754863973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83b8100d40 0x7f83b819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 >> 172.21.15.116:0/3754863973 conn(0x7f83b8077710 msgr2=0x7f83b80ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 shutdown_connections 2024-03-24T09:29:14.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.661+0000 7f83bcd93640 1 -- 172.21.15.116:0/3754863973 wait complete. 2024-03-24T09:29:14.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:29:14.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 -- 172.21.15.116:0/1515216145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 msgr2=0x7f78e0101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 --2- 172.21.15.116:0/1515216145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e0101b20 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009930 tx=0x7f78d002bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:14.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 -- 172.21.15.116:0/1515216145 shutdown_connections 2024-03-24T09:29:14.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 --2- 172.21.15.116:0/1515216145 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e0102060 0x7f78e010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 --2- 172.21.15.116:0/1515216145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e0101b20 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 --2- 172.21.15.116:0/1515216145 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78e01076b0 0x7f78e0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 -- 172.21.15.116:0/1515216145 >> 172.21.15.116:0/1515216145 conn(0x7f78e0078b10 msgr2=0x7f78e0078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 -- 172.21.15.116:0/1515216145 shutdown_connections 2024-03-24T09:29:14.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.777+0000 7f78e79ac640 1 -- 172.21.15.116:0/1515216145 wait complete. 2024-03-24T09:29:14.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 Processor -- start 2024-03-24T09:29:14.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 -- start start 2024-03-24T09:29:14.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78e0102060 0x7f78e019de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e01076b0 0x7f78e0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78e0198120 con 0x7f78e01016a0 2024-03-24T09:29:14.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f78e0198290 con 0x7f78e0102060 2024-03-24T09:29:14.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e79ac640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f78e0198450 con 0x7f78e01076b0 2024-03-24T09:29:14.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e019d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58638/0 (socket says 172.21.15.116:58638) 2024-03-24T09:29:14.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e5748640 1 -- 172.21.15.116:0/4041108892 learned_addr learned my addr 172.21.15.116:0/4041108892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e5f49640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e01076b0 0x7f78e0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.778+0000 7f78e4f47640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78e0102060 0x7f78e019de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e5748640 1 -- 172.21.15.116:0/4041108892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e01076b0 msgr2=0x7f78e0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e5748640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e01076b0 0x7f78e0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e5748640 1 -- 172.21.15.116:0/4041108892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78e0102060 msgr2=0x7f78e019de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e5748640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78e0102060 0x7f78e019de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e5748640 1 -- 172.21.15.116:0/4041108892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d0009560 con 0x7f78e01016a0 2024-03-24T09:29:14.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e5748640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e019d920 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00e9f0 tx=0x7f78dc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78d67fc640 1 -- 172.21.15.116:0/4041108892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc009700 con 0x7f78e01016a0 2024-03-24T09:29:14.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78d67fc640 1 -- 172.21.15.116:0/4041108892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78dc00cd80 con 0x7f78e01016a0 2024-03-24T09:29:14.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e0198730 con 0x7f78e01016a0 2024-03-24T09:29:14.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78d67fc640 1 -- 172.21.15.116:0/4041108892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc018840 con 0x7f78e01016a0 2024-03-24T09:29:14.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.779+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e01a49c0 con 0x7f78e01016a0 2024-03-24T09:29:14.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.781+0000 7f78d67fc640 1 -- 172.21.15.116:0/4041108892 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f78dc0189a0 con 0x7f78e01016a0 2024-03-24T09:29:14.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 Processor -- start 2024-03-24T09:29:14.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 -- start start 2024-03-24T09:29:14.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78d67fc640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f78cc0a2230 0x7f78cc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674f3f0 0x55c39674f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 0x55c3967583b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c39675ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c3966bd840 con 0x55c39674fd10 2024-03-24T09:29:14.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c3966bad70 con 0x55c39674f3f0 2024-03-24T09:29:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78ea588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c3966b6550 con 0x55c3967588f0 2024-03-24T09:29:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5f49640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f78cc0a2230 0x7f78cc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78d67fc640 1 -- 172.21.15.116:0/4041108892 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f78dc014070 con 0x7f78e01016a0 2024-03-24T09:29:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e4f47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 0x55c3967583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5748640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674f3f0 0x55c39674f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e4f47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 0x55c3967583b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58654/0 (socket says 172.21.15.116:58654) 2024-03-24T09:29:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e4f47640 1 -- 172.21.15.116:0/3747472195 learned_addr learned my addr 172.21.15.116:0/3747472195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5748640 1 -- 172.21.15.116:0/3747472195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 msgr2=0x55c39675ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5f49640 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c39675ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5748640 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c39675ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5748640 1 -- 172.21.15.116:0/3747472195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 msgr2=0x55c3967583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5748640 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 0x55c3967583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5748640 1 -- 172.21.15.116:0/3747472195 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c39664bde0 con 0x55c39674f3f0 2024-03-24T09:29:14.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e4f47640 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 0x55c3967583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.782+0000 7f78e5f49640 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c39675ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:29:14.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78e5f49640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f78cc0a2230 0x7f78cc0a46f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f78d8005fd0 tx=0x7f78d8009f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78e5748640 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674f3f0 0x55c39674f7d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0cba80 tx=0x7f78dc0971a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621dde7b257ced71 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78cb7fe640 1 -- 172.21.15.116:0/3747472195 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc020550 con 0x55c39674f3f0 2024-03-24T09:29:14.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78cb7fe640 1 -- 172.21.15.116:0/3747472195 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78dc020360 con 0x55c39674f3f0 2024-03-24T09:29:14.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78cb7fe640 1 -- 172.21.15.116:0/3747472195 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0990d0 con 0x55c39674f3f0 2024-03-24T09:29:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78ea588740 1 -- 172.21.15.116:0/3747472195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674f3f0 msgr2=0x55c39674f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.783+0000 7f78ea588740 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674f3f0 0x55c39674f7d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0cba80 tx=0x7f78dc0971a0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- 172.21.15.116:0/3747472195 shutdown_connections 2024-03-24T09:29:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c39675ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674fd10 0x55c3967583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 --2- 172.21.15.116:0/3747472195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674f3f0 0x55c39674f7d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- 172.21.15.116:0/3747472195 >> 172.21.15.116:0/3747472195 conn(0x55c396749b20 msgr2=0x55c396749f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- 172.21.15.116:0/3747472195 shutdown_connections 2024-03-24T09:29:14.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- 172.21.15.116:0/3747472195 wait complete. 2024-03-24T09:29:14.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- start start 2024-03-24T09:29:14.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 Processor -- start 2024-03-24T09:29:14.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396755a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c396755f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396756480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c396389430 con 0x55c39674f3f0 2024-03-24T09:29:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c396345e00 con 0x55c3967588f0 2024-03-24T09:29:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c39663b590 con 0x55c39674fd10 2024-03-24T09:29:14.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78ea588740 10 client.0 _ll_register_callbacks cb 0x55c3966aafc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:14.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.784+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396755a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396755a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58660/0 (socket says 172.21.15.116:58660) 2024-03-24T09:29:14.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 -- 172.21.15.116:0/2641873544 learned_addr learned my addr 172.21.15.116:0/2641873544 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e4f47640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c396755f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5f49640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396756480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 -- 172.21.15.116:0/2641873544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 msgr2=0x55c396755f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c396755f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 -- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 msgr2=0x55c396756480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396756480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 -- 172.21.15.116:0/2641873544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e01a49c0 con 0x55c39674f3f0 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78e5748640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396755a00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f78dc099080 tx=0x7f78dc09add0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d397acf5c7714b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:14.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78c8ff9640 1 -- 172.21.15.116:0/2641873544 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc020550 con 0x55c39674f3f0 2024-03-24T09:29:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78c8ff9640 1 -- 172.21.15.116:0/2641873544 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f78dc09ee70 con 0x55c39674f3f0 2024-03-24T09:29:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c3963b57e0 con 0x55c39674f3f0 2024-03-24T09:29:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78c8ff9640 1 -- 172.21.15.116:0/2641873544 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f78dc0b9250 con 0x55c39674f3f0 2024-03-24T09:29:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78ea588740 10 client.17643 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c3966b4bd0 con 0x55c39674f3f0 2024-03-24T09:29:14.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78c8ff9640 1 -- 172.21.15.116:0/2641873544 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55c3963b57e0 con 0x55c39674f3f0 2024-03-24T09:29:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.785+0000 7f78c8ff9640 1 -- 172.21.15.116:0/2641873544 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f78dc017b00 con 0x55c39674f3f0 2024-03-24T09:29:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78ea588740 10 client.17643 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78ea588740 10 client.17643 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:29:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78ea588740 20 client.17643 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78ea588740 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967685b0 0x55c39676a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78ea588740 4 client.17643 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3963f2720 con 0x55c3967685b0 2024-03-24T09:29:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78e4f47640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967685b0 0x55c39676a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78c8ff9640 10 client.17643 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.786+0000 7f78e4f47640 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967685b0 0x55c39676a9a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5733177a59c9d8f4 server_cookie=3919b66c29c8febc in_seq=0 out_seq=0 2024-03-24T09:29:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78c8ff9640 1 -- 172.21.15.116:0/2641873544 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c3963f2720 con 0x55c3967685b0 2024-03-24T09:29:14.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78c8ff9640 10 client.17643 handle_command_reply: tid=0 2024-03-24T09:29:14.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 client.17643 shutdown 2024-03-24T09:29:14.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 msgr2=0x55c396755a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396755a00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f78dc099080 tx=0x7f78dc09add0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 shutdown_connections 2024-03-24T09:29:14.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396756480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c396755f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967685b0 0x55c39676a9a0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 --2- 172.21.15.116:0/2641873544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396755a00 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78c8ff9640 0 client.17643 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:14.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 >> 172.21.15.116:0/2641873544 conn(0x55c396749b20 msgr2=0x55c39674b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 shutdown_connections 2024-03-24T09:29:14.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 1 -- 172.21.15.116:0/2641873544 wait complete. 2024-03-24T09:29:14.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.787+0000 7f78ea588740 20 client.17643 trim_cache size 0 max 16384 2024-03-24T09:29:14.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 Processor -- start 2024-03-24T09:29:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 -- start start 2024-03-24T09:29:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396758130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c3967640c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396770230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c3967569d0 con 0x55c39674f3f0 2024-03-24T09:29:14.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c39674eb90 con 0x55c3967588f0 2024-03-24T09:29:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78ea588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c39674ed40 con 0x55c39674fd10 2024-03-24T09:29:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396758130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396758130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58672/0 (socket says 172.21.15.116:58672) 2024-03-24T09:29:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78e5748640 1 -- 172.21.15.116:0/216886725 learned_addr learned my addr 172.21.15.116:0/216886725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.800+0000 7f78e4f47640 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c3967640c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5f49640 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396770230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5748640 1 -- 172.21.15.116:0/216886725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 msgr2=0x55c3967640c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5748640 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c3967640c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5748640 1 -- 172.21.15.116:0/216886725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 msgr2=0x55c396770230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5748640 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396770230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5748640 1 -- 172.21.15.116:0/216886725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c396650c80 con 0x55c39674f3f0 2024-03-24T09:29:14.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78e5748640 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396758130 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0977a0 tx=0x7f78dc0992e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d0eb9220f07951 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78caffd640 1 -- 172.21.15.116:0/216886725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0b9250 con 0x55c39674f3f0 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78caffd640 1 -- 172.21.15.116:0/216886725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78dc09ee70 con 0x55c39674f3f0 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78caffd640 1 -- 172.21.15.116:0/216886725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc09a480 con 0x55c39674f3f0 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78ea588740 1 -- 172.21.15.116:0/216886725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 msgr2=0x55c396758130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.801+0000 7f78ea588740 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396758130 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0977a0 tx=0x7f78dc0992e0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- 172.21.15.116:0/216886725 shutdown_connections 2024-03-24T09:29:14.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c3967588f0 0x55c396770230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c39674fd10 0x55c3967640c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 --2- 172.21.15.116:0/216886725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c396758130 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- 172.21.15.116:0/216886725 >> 172.21.15.116:0/216886725 conn(0x55c396749b20 msgr2=0x55c396748cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- 172.21.15.116:0/216886725 shutdown_connections 2024-03-24T09:29:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- 172.21.15.116:0/216886725 wait complete. 2024-03-24T09:29:14.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- start start 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 Processor -- start 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c39675c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674fd10 0x55c39675c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c3967cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c39675d1f0 con 0x55c39674f3f0 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c3967cdda0 con 0x55c39674fd10 2024-03-24T09:29:14.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55c3967cdee0 con 0x55c3967588f0 2024-03-24T09:29:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78ea588740 10 client.0 _ll_register_callbacks cb 0x7f78d80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c39675c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78e5748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c39675c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58688/0 (socket says 172.21.15.116:58688) 2024-03-24T09:29:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78e5748640 1 -- 172.21.15.116:0/359483069 learned_addr learned my addr 172.21.15.116:0/359483069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.802+0000 7f78e4f47640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674fd10 0x55c39675c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5f49640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c3967cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5748640 1 -- 172.21.15.116:0/359483069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 msgr2=0x55c3967cd860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5748640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c3967cd860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5748640 1 -- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674fd10 msgr2=0x55c39675c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5748640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674fd10 0x55c39675c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5748640 1 -- 172.21.15.116:0/359483069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3966b4bd0 con 0x55c39674f3f0 2024-03-24T09:29:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78e5748640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c39675c3e0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0204d0 tx=0x7f78dc09a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc50082ea358f0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78c8ff9640 1 -- 172.21.15.116:0/359483069 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0b9250 con 0x55c39674f3f0 2024-03-24T09:29:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78c8ff9640 1 -- 172.21.15.116:0/359483069 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f78dc097440 con 0x55c39674f3f0 2024-03-24T09:29:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78dc020360 con 0x55c39674f3f0 2024-03-24T09:29:14.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78c8ff9640 1 -- 172.21.15.116:0/359483069 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f78dc096b20 con 0x55c39674f3f0 2024-03-24T09:29:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 10 client.17649 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c396676920 con 0x55c39674f3f0 2024-03-24T09:29:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78c8ff9640 1 -- 172.21.15.116:0/359483069 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f78dc020360 con 0x55c39674f3f0 2024-03-24T09:29:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78c8ff9640 1 -- 172.21.15.116:0/359483069 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f78dc09feb0 con 0x55c39674f3f0 2024-03-24T09:29:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 10 client.17649 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:14.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 10 client.17649 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:29:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 20 client.17649 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967cf1e0 0x55c3967d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 4 client.17649 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.803+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c396441890 con 0x55c3967cf1e0 2024-03-24T09:29:14.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78e4f47640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967cf1e0 0x55c3967d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78c8ff9640 10 client.17649 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:14.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78e4f47640 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967cf1e0 0x55c3967d15d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8d188009d29c7d2a server_cookie=54b5d8a88a7037b2 in_seq=0 out_seq=0 2024-03-24T09:29:14.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78c8ff9640 1 -- 172.21.15.116:0/359483069 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c396441890 con 0x55c3967cf1e0 2024-03-24T09:29:14.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78c8ff9640 10 client.17649 handle_command_reply: tid=0 2024-03-24T09:29:14.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 client.17649 shutdown 2024-03-24T09:29:14.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 msgr2=0x55c39675c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c39675c3e0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0204d0 tx=0x7f78dc09a450 comp rx=0 tx=0).stop 2024-03-24T09:29:14.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 shutdown_connections 2024-03-24T09:29:14.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55c3967588f0 0x55c3967cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c39674fd10 0x55c39675c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55c3967cf1e0 0x55c3967d15d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 --2- 172.21.15.116:0/359483069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c39674f3f0 0x55c39675c3e0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78c8ff9640 0 client.17649 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:14.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 >> 172.21.15.116:0/359483069 conn(0x55c396749b20 msgr2=0x55c3967559a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 shutdown_connections 2024-03-24T09:29:14.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 1 -- 172.21.15.116:0/359483069 wait complete. 2024-03-24T09:29:14.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.804+0000 7f78ea588740 20 client.17649 trim_cache size 0 max 16384 2024-03-24T09:29:14.835 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:14.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f78cc0a2230 msgr2=0x7f78cc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f78cc0a2230 0x7f78cc0a46f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f78d8005fd0 tx=0x7f78d8009f90 comp rx=0 tx=0).stop 2024-03-24T09:29:14.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 msgr2=0x7f78e019d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e019d920 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00e9f0 tx=0x7f78dc00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 shutdown_connections 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f78cc0a2230 0x7f78cc0a46f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f78e01076b0 0x7f78e0197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78e0102060 0x7f78e019de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 --2- 172.21.15.116:0/4041108892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78e01016a0 0x7f78e019d920 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 >> 172.21.15.116:0/4041108892 conn(0x7f78e0078b10 msgr2=0x7f78e01041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 shutdown_connections 2024-03-24T09:29:14.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.805+0000 7f78e79ac640 1 -- 172.21.15.116:0/4041108892 wait complete. 2024-03-24T09:29:14.839 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:29:14.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4288411025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 msgr2=0x7f22e410d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4288411025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 0x7f22e410d420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f22dc009cd0 tx=0x7f22dc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4288411025 shutdown_connections 2024-03-24T09:29:14.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4288411025 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 0x7f22e410d420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4288411025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e4100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4288411025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4288411025 >> 172.21.15.116:0/4288411025 conn(0x7f22e4077710 msgr2=0x7f22e4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4288411025 shutdown_connections 2024-03-24T09:29:14.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4288411025 wait complete. 2024-03-24T09:29:14.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 Processor -- start 2024-03-24T09:29:14.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 -- start start 2024-03-24T09:29:14.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.943+0000 7f22e9f1e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e419d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 0x7f22e419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22e4197fd0 con 0x7f22e4106660 2024-03-24T09:29:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f22e4198140 con 0x7f22e41005a0 2024-03-24T09:29:14.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f22e4198300 con 0x7f22e4100f60 2024-03-24T09:29:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e419d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58710/0 (socket says 172.21.15.116:58710) 2024-03-24T09:29:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 -- 172.21.15.116:0/4210400337 learned_addr learned my addr 172.21.15.116:0/4210400337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e419d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:60872/0 (socket says 172.21.15.116:60872) 2024-03-24T09:29:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22dbfff640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 0x7f22e419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 -- 172.21.15.116:0/4210400337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 msgr2=0x7f22e419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 0x7f22e419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 -- 172.21.15.116:0/4210400337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 msgr2=0x7f22e419d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e419d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 -- 172.21.15.116:0/4210400337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc009900 con 0x7f22e4106660 2024-03-24T09:29:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e37fe640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e419d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e3fff640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4197980 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004af0 tx=0x7f22dc032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e17fa640 1 -- 172.21.15.116:0/4210400337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc0325d0 con 0x7f22e4106660 2024-03-24T09:29:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e4198580 con 0x7f22e4106660 2024-03-24T09:29:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e17fa640 1 -- 172.21.15.116:0/4210400337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22dc003970 con 0x7f22e4106660 2024-03-24T09:29:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e17fa640 1 -- 172.21.15.116:0/4210400337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22dc0046b0 con 0x7f22e4106660 2024-03-24T09:29:14.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.944+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e41a4870 con 0x7f22e4106660 2024-03-24T09:29:14.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.946+0000 7f22e17fa640 1 -- 172.21.15.116:0/4210400337 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f22dc03b040 con 0x7f22e4106660 2024-03-24T09:29:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 Processor -- start 2024-03-24T09:29:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 -- start start 2024-03-24T09:29:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e979f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97aace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5e970d840 con 0x55a5e979fd10 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a5e970ad70 con 0x55a5e979f3f0 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5e9706550 con 0x55a5e97a88f0 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e17fa640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22d00a2230 0x7f22d00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58720/0 (socket says 172.21.15.116:58720) 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 -- 172.21.15.116:0/3604289515 learned_addr learned my addr 172.21.15.116:0/3604289515 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e979f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e3fff640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22d00a2230 0x7f22d00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 -- 172.21.15.116:0/3604289515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 msgr2=0x55a5e97aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e17fa640 1 -- 172.21.15.116:0/4210400337 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f22dc0f0cc0 con 0x7f22e4106660 2024-03-24T09:29:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 -- 172.21.15.116:0/3604289515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 msgr2=0x55a5e979f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e979f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22e37fe640 1 -- 172.21.15.116:0/3604289515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5e969bde0 con 0x55a5e979fd10 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.947+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e979f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22e37fe640 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a83b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f22d40114f0 tx=0x7f22d40119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1313431c5f11d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22e3fff640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22d00a2230 0x7f22d00a46f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f22dc0bd140 tx=0x7f22dc0b28d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22d9ffb640 1 -- 172.21.15.116:0/3604289515 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4011250 con 0x55a5e979fd10 2024-03-24T09:29:14.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22d9ffb640 1 -- 172.21.15.116:0/3604289515 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22d40044e0 con 0x55a5e979fd10 2024-03-24T09:29:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22d9ffb640 1 -- 172.21.15.116:0/3604289515 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d400f5c0 con 0x55a5e979fd10 2024-03-24T09:29:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3604289515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 msgr2=0x55a5e97a83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.948+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a83b0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f22d40114f0 tx=0x7f22d40119b0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3604289515 shutdown_connections 2024-03-24T09:29:14.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a83b0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3604289515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e979f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3604289515 >> 172.21.15.116:0/3604289515 conn(0x55a5e9799b20 msgr2=0x55a5e9799f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3604289515 shutdown_connections 2024-03-24T09:29:14.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3604289515 wait complete. 2024-03-24T09:29:14.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- start start 2024-03-24T09:29:14.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 Processor -- start 2024-03-24T09:29:14.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e97a5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5e93d9430 con 0x55a5e979fd10 2024-03-24T09:29:14.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a5e9395e00 con 0x55a5e979f3f0 2024-03-24T09:29:14.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5e968b590 con 0x55a5e97a88f0 2024-03-24T09:29:14.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22ecbd8740 10 client.0 _ll_register_callbacks cb 0x55a5e96fafc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:14.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58722/0 (socket says 172.21.15.116:58722) 2024-03-24T09:29:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22e37fe640 1 -- 172.21.15.116:0/304718364 learned_addr learned my addr 172.21.15.116:0/304718364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22dbfff640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e97a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.949+0000 7f22dbfff640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22e37fe640 1 -- 172.21.15.116:0/304718364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 msgr2=0x55a5e97a63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22e37fe640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97a63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22e37fe640 1 -- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 msgr2=0x55a5e97a5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22e37fe640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e97a5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22e37fe640 1 -- 172.21.15.116:0/304718364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22d4009cc0 con 0x55a5e979fd10 2024-03-24T09:29:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22e37fe640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a5e90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f22d401b040 tx=0x7f22d4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f0ac424bb9e3f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22c2ffd640 1 -- 172.21.15.116:0/304718364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4005250 con 0x55a5e979fd10 2024-03-24T09:29:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22c2ffd640 1 -- 172.21.15.116:0/304718364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f22d40053b0 con 0x55a5e979fd10 2024-03-24T09:29:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a5e94057e0 con 0x55a5e979fd10 2024-03-24T09:29:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22c2ffd640 1 -- 172.21.15.116:0/304718364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f22d4005570 con 0x55a5e979fd10 2024-03-24T09:29:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22ecbd8740 10 client.17661 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:14.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a5e9704bd0 con 0x55a5e979fd10 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22c2ffd640 1 -- 172.21.15.116:0/304718364 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55a5e94057e0 con 0x55a5e979fd10 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.950+0000 7f22c2ffd640 1 -- 172.21.15.116:0/304718364 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f22d4004b40 con 0x55a5e979fd10 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22ecbd8740 10 client.17661 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22ecbd8740 10 client.17661 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22ecbd8740 20 client.17661 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e97b8980 0x55a5e97badc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22ecbd8740 4 client.17661 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5e9442720 con 0x55a5e97b8980 2024-03-24T09:29:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22dbfff640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e97b8980 0x55a5e97badc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22c2ffd640 10 client.17661 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.951+0000 7f22dbfff640 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e97b8980 0x55a5e97badc0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9b7d3666050dac0 server_cookie=9c6881d5b0a875d7 in_seq=0 out_seq=0 2024-03-24T09:29:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22c2ffd640 1 -- 172.21.15.116:0/304718364 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a5e9442720 con 0x55a5e97b8980 2024-03-24T09:29:14.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22c2ffd640 10 client.17661 handle_command_reply: tid=0 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 client.17661 shutdown 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 msgr2=0x55a5e97a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a5e90 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f22d401b040 tx=0x7f22d4003910 comp rx=0 tx=0).stop 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 shutdown_connections 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e97b8980 0x55a5e97badc0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e97a63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22c2ffd640 0 client.17661 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:14.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97a5e90 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/304718364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979f3f0 0x55a5e97a5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 >> 172.21.15.116:0/304718364 conn(0x55a5e9799b20 msgr2=0x55a5e979b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 shutdown_connections 2024-03-24T09:29:14.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 1 -- 172.21.15.116:0/304718364 wait complete. 2024-03-24T09:29:14.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.952+0000 7f22ecbd8740 20 client.17661 trim_cache size 0 max 16384 2024-03-24T09:29:14.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 Processor -- start 2024-03-24T09:29:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 -- start start 2024-03-24T09:29:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e979f3f0 0x55a5e97b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e97a88f0 0x55a5e97c06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5e979f060 con 0x55a5e979fd10 2024-03-24T09:29:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a5e97953e0 con 0x55a5e97a88f0 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5e979ab20 con 0x55a5e979f3f0 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97b4180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58738/0 (socket says 172.21.15.116:58738) 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22e37fe640 1 -- 172.21.15.116:0/161773134 learned_addr learned my addr 172.21.15.116:0/161773134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22dbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e97a88f0 0x55a5e97c06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.965+0000 7f22dbfff640 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e979f3f0 0x55a5e97b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22e37fe640 1 -- 172.21.15.116:0/161773134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e979f3f0 msgr2=0x55a5e97b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22e37fe640 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e979f3f0 0x55a5e97b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22e37fe640 1 -- 172.21.15.116:0/161773134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e97a88f0 msgr2=0x55a5e97c06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22e37fe640 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e97a88f0 0x55a5e97c06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22e37fe640 1 -- 172.21.15.116:0/161773134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5e96a0c80 con 0x55a5e979fd10 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22dbfff640 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e97a88f0 0x55a5e97c06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22e37fe640 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97b4180 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f22d4000c00 tx=0x7f22d40119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e7bc71166e59b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22d97fa640 1 -- 172.21.15.116:0/161773134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4009b50 con 0x55a5e979fd10 2024-03-24T09:29:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22d97fa640 1 -- 172.21.15.116:0/161773134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22d4005250 con 0x55a5e979fd10 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22d97fa640 1 -- 172.21.15.116:0/161773134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d40048b0 con 0x55a5e979fd10 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22ecbd8740 1 -- 172.21.15.116:0/161773134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 msgr2=0x55a5e97b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.966+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97b4180 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f22d4000c00 tx=0x7f22d40119b0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- 172.21.15.116:0/161773134 shutdown_connections 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e97a88f0 0x55a5e97c06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979fd10 0x55a5e97b4180 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/161773134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e979f3f0 0x55a5e97b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- 172.21.15.116:0/161773134 >> 172.21.15.116:0/161773134 conn(0x55a5e9799b20 msgr2=0x55a5e97a5910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- 172.21.15.116:0/161773134 shutdown_connections 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- 172.21.15.116:0/161773134 wait complete. 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- start start 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 Processor -- start 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 0x55a5e9816eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979fd10 0x55a5e98173f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e981bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a5e981c2c0 con 0x55a5e979f3f0 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a5e981c430 con 0x55a5e979fd10 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55a5e981c5f0 con 0x55a5e97a88f0 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22ecbd8740 10 client.0 _ll_register_callbacks cb 0x7f22cc009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 0x55a5e9816eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 0x55a5e9816eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58746/0 (socket says 172.21.15.116:58746) 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22dbfff640 1 -- 172.21.15.116:0/3708514724 learned_addr learned my addr 172.21.15.116:0/3708514724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:14.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.967+0000 7f22e37fe640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979fd10 0x55a5e98173f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e981bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 -- 172.21.15.116:0/3708514724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 msgr2=0x55a5e981bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e981bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 -- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979fd10 msgr2=0x55a5e98173f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979fd10 0x55a5e98173f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 -- 172.21.15.116:0/3708514724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc0128e0 con 0x55a5e979f3f0 2024-03-24T09:29:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22dbfff640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 0x55a5e9816eb0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009730 tx=0x7f22d40113a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab5957aabfb253bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22c2ffd640 1 -- 172.21.15.116:0/3708514724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22cc0038b0 con 0x55a5e979f3f0 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22c2ffd640 1 -- 172.21.15.116:0/3708514724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f22cc003a10 con 0x55a5e979f3f0 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22d40044e0 con 0x55a5e979f3f0 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22c2ffd640 1 -- 172.21.15.116:0/3708514724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f22cc003bd0 con 0x55a5e979f3f0 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 10 client.17667 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55a5e9704bd0 con 0x55a5e979f3f0 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22c2ffd640 1 -- 172.21.15.116:0/3708514724 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22d40044e0 con 0x55a5e979f3f0 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22c2ffd640 1 -- 172.21.15.116:0/3708514724 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f22cc030060 con 0x55a5e979f3f0 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 10 client.17667 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 10 client.17667 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 20 client.17667 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e9823f40 0x55a5e9826360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 4 client.17667 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.968+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a5e9491890 con 0x55a5e9823f40 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22e37fe640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e9823f40 0x55a5e9826360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22c2ffd640 10 client.17667 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22e37fe640 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e9823f40 0x55a5e9826360 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=187ba53e594d1a55 server_cookie=a47d855bc78914dd in_seq=0 out_seq=0 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22c2ffd640 1 -- 172.21.15.116:0/3708514724 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a5e9491890 con 0x55a5e9823f40 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22c2ffd640 10 client.17667 handle_command_reply: tid=0 2024-03-24T09:29:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 client.17667 shutdown 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 msgr2=0x55a5e9816eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 0x55a5e9816eb0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009730 tx=0x7f22d40113a0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 shutdown_connections 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55a5e9823f40 0x55a5e9826360 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55a5e97a88f0 0x55a5e981bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22c2ffd640 0 client.17667 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a5e979fd10 0x55a5e98173f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 --2- 172.21.15.116:0/3708514724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a5e979f3f0 0x55a5e9816eb0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 >> 172.21.15.116:0/3708514724 conn(0x55a5e9799b20 msgr2=0x55a5e97a6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 shutdown_connections 2024-03-24T09:29:14.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 1 -- 172.21.15.116:0/3708514724 wait complete. 2024-03-24T09:29:14.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.969+0000 7f22ecbd8740 20 client.17667 trim_cache size 0 max 16384 2024-03-24T09:29:14.994 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:14.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22d00a2230 msgr2=0x7f22d00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22d00a2230 0x7f22d00a46f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f22dc0bd140 tx=0x7f22dc0b28d0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 msgr2=0x7f22e4197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:14.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4197980 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004af0 tx=0x7f22dc032f30 comp rx=0 tx=0).stop 2024-03-24T09:29:14.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 shutdown_connections 2024-03-24T09:29:14.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e4106660 0x7f22e4197980 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22d00a2230 0x7f22d00a46f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22e4100f60 0x7f22e419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 --2- 172.21.15.116:0/4210400337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e41005a0 0x7f22e419d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:14.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 >> 172.21.15.116:0/4210400337 conn(0x7f22e4077710 msgr2=0x7f22e40ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:14.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 shutdown_connections 2024-03-24T09:29:14.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:14.970+0000 7f22e9f1e640 1 -- 172.21.15.116:0/4210400337 wait complete. 2024-03-24T09:29:14.998 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:29:15.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.100+0000 7f6a21325640 1 -- 172.21.15.116:0/241637963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c103270 msgr2=0x7f6a1c10b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.100+0000 7f6a21325640 1 --2- 172.21.15.116:0/241637963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c103270 0x7f6a1c10b750 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009900 tx=0x7f6a1002bcd0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- 172.21.15.116:0/241637963 shutdown_connections 2024-03-24T09:29:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 --2- 172.21.15.116:0/241637963 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c10bd20 0x7f6a1c10e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 --2- 172.21.15.116:0/241637963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c103270 0x7f6a1c10b750 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 --2- 172.21.15.116:0/241637963 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a1c102950 0x7f6a1c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- 172.21.15.116:0/241637963 >> 172.21.15.116:0/241637963 conn(0x7f6a1c076990 msgr2=0x7f6a1c076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- 172.21.15.116:0/241637963 shutdown_connections 2024-03-24T09:29:15.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- 172.21.15.116:0/241637963 wait complete. 2024-03-24T09:29:15.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 Processor -- start 2024-03-24T09:29:15.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- start start 2024-03-24T09:29:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 0x7f6a1c19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a1c103270 0x7f6a1c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c10bd20 0x7f6a1c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c198110 con 0x7f6a1c10bd20 2024-03-24T09:29:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c198280 con 0x7f6a1c102950 2024-03-24T09:29:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.101+0000 7f6a21325640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1c198440 con 0x7f6a1c103270 2024-03-24T09:29:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 0x7f6a1c19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c10bd20 0x7f6a1c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 0x7f6a1c19d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:60914/0 (socket says 172.21.15.116:60914) 2024-03-24T09:29:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1affd640 1 -- 172.21.15.116:0/2009785905 learned_addr learned my addr 172.21.15.116:0/2009785905 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a1c103270 0x7f6a1c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/2009785905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a1c103270 msgr2=0x7f6a1c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a1c103270 0x7f6a1c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/2009785905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 msgr2=0x7f6a1c19d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 0x7f6a1c19d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/2009785905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a10009560 con 0x7f6a1c10bd20 2024-03-24T09:29:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1affd640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 0x7f6a1c19d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c10bd20 0x7f6a1c197ac0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6a14005fd0 tx=0x7f6a1400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a03fff640 1 -- 172.21.15.116:0/2009785905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1400cdf0 con 0x7f6a1c10bd20 2024-03-24T09:29:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a03fff640 1 -- 172.21.15.116:0/2009785905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a14005020 con 0x7f6a1c10bd20 2024-03-24T09:29:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a03fff640 1 -- 172.21.15.116:0/2009785905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14010690 con 0x7f6a1c10bd20 2024-03-24T09:29:15.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a1c198720 con 0x7f6a1c10bd20 2024-03-24T09:29:15.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.102+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a1c1a4960 con 0x7f6a1c10bd20 2024-03-24T09:29:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.104+0000 7f6a03fff640 1 -- 172.21.15.116:0/2009785905 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6a14005190 con 0x7f6a1c10bd20 2024-03-24T09:29:15.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.104+0000 7f6a23fe8740 1 Processor -- start 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 -- start start 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7c17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a03fff640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6a040a2230 0x7f6a040a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56523d72f840 con 0x56523d7ca8f0 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56523d72cd70 con 0x56523d7c1d10 2024-03-24T09:29:15.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56523d728550 con 0x56523d7c13f0 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1a7fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7c17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:60920/0 (socket says 172.21.15.116:60920) 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1a7fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7c17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:38194/0 (socket says 172.21.15.116:38194) 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 -- 172.21.15.116:0/2343285944 learned_addr learned my addr 172.21.15.116:0/2343285944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6a040a2230 0x7f6a040a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a03fff640 1 -- 172.21.15.116:0/2009785905 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f6a14014070 con 0x7f6a1c10bd20 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 -- 172.21.15.116:0/2343285944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 msgr2=0x56523d7c17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7c17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 -- 172.21.15.116:0/2343285944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 msgr2=0x56523d7ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 -- 172.21.15.116:0/2343285944 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56523d6bdde0 con 0x56523d7c1d10 2024-03-24T09:29:15.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6a040a2230 0x7f6a040a46f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006010 tx=0x7f6a1002bcd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.105+0000 7f6a1affd640 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7ca3b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0115f0 tx=0x7f6a0c011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1894b85335505d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a017fa640 1 -- 172.21.15.116:0/2343285944 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c005020 con 0x56523d7c1d10 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a017fa640 1 -- 172.21.15.116:0/2343285944 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a0c005630 con 0x56523d7c1d10 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a017fa640 1 -- 172.21.15.116:0/2343285944 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c00f920 con 0x56523d7c1d10 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 -- 172.21.15.116:0/2343285944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 msgr2=0x56523d7ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7ca3b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0115f0 tx=0x7f6a0c011ab0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 -- 172.21.15.116:0/2343285944 shutdown_connections 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7ca3b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/2343285944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7c17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 -- 172.21.15.116:0/2343285944 >> 172.21.15.116:0/2343285944 conn(0x56523d7bbb20 msgr2=0x56523d7bbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 -- 172.21.15.116:0/2343285944 shutdown_connections 2024-03-24T09:29:15.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.106+0000 7f6a23fe8740 1 -- 172.21.15.116:0/2343285944 wait complete. 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 -- start start 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 Processor -- start 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7c7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c1d10 0x56523d7c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7ca8f0 0x56523d7c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56523d3fb430 con 0x56523d7c13f0 2024-03-24T09:29:15.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56523d3b7e00 con 0x56523d7ca8f0 2024-03-24T09:29:15.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56523d6ad590 con 0x56523d7c1d10 2024-03-24T09:29:15.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a23fe8740 10 client.0 _ll_register_callbacks cb 0x56523d71cfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:15.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7c7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58788/0 (socket says 172.21.15.116:58788) 2024-03-24T09:29:15.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 -- 172.21.15.116:0/544055017 learned_addr learned my addr 172.21.15.116:0/544055017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7ca8f0 0x56523d7c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c1d10 0x56523d7c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 -- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c1d10 msgr2=0x56523d7c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c1d10 0x56523d7c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 -- 172.21.15.116:0/544055017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7ca8f0 msgr2=0x56523d7c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7ca8f0 0x56523d7c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 -- 172.21.15.116:0/544055017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0c009dc0 con 0x56523d7c13f0 2024-03-24T09:29:15.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.107+0000 7f6a1affd640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7c7910 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0057c0 tx=0x7f6a0c005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e08162e79b06c5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f69feff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:15.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f69feff5640 1 -- 172.21.15.116:0/544055017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c005450 con 0x56523d7c13f0 2024-03-24T09:29:15.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f69feff5640 1 -- 172.21.15.116:0/544055017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f6a0c00f8d0 con 0x56523d7c13f0 2024-03-24T09:29:15.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56523d4277e0 con 0x56523d7c13f0 2024-03-24T09:29:15.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f69feff5640 1 -- 172.21.15.116:0/544055017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6a0c00fad0 con 0x56523d7c13f0 2024-03-24T09:29:15.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 10 client.17679 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:15.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56523d726bd0 con 0x56523d7c13f0 2024-03-24T09:29:15.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f69feff5640 1 -- 172.21.15.116:0/544055017 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56523d4277e0 con 0x56523d7c13f0 2024-03-24T09:29:15.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f69feff5640 1 -- 172.21.15.116:0/544055017 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f6a0c024a10 con 0x56523d7c13f0 2024-03-24T09:29:15.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 10 client.17679 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:15.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 10 client.17679 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 20 client.17679 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d7da940 0x56523d7dcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 4 client.17679 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56523d464720 con 0x56523d7da940 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.108+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d7da940 0x56523d7dcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.109+0000 7f69feff5640 10 client.17679 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:15.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.109+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d7da940 0x56523d7dcd60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9bb6108f2d1ca33f server_cookie=914aa8e1b4029642 in_seq=0 out_seq=0 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f69feff5640 1 -- 172.21.15.116:0/544055017 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56523d464720 con 0x56523d7da940 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f69feff5640 10 client.17679 handle_command_reply: tid=0 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 client.17679 shutdown 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 msgr2=0x56523d7c7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7c7910 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0057c0 tx=0x7f6a0c005230 comp rx=0 tx=0).stop 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 shutdown_connections 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d7da940 0x56523d7dcd60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7ca8f0 0x56523d7c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f69feff5640 0 client.17679 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c1d10 0x56523d7c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/544055017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7c7910 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 >> 172.21.15.116:0/544055017 conn(0x56523d7bbb20 msgr2=0x56523d7bd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 shutdown_connections 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 1 -- 172.21.15.116:0/544055017 wait complete. 2024-03-24T09:29:15.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.110+0000 7f6a23fe8740 20 client.17679 trim_cache size 0 max 16384 2024-03-24T09:29:15.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 Processor -- start 2024-03-24T09:29:15.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 -- start start 2024-03-24T09:29:15.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7ca040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7d6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7e2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56523d7c0e30 con 0x56523d7ca8f0 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56523d7bcb20 con 0x56523d7c1d10 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56523d7c3520 con 0x56523d7c13f0 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7e2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7e2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58802/0 (socket says 172.21.15.116:58802) 2024-03-24T09:29:15.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/4033999398 learned_addr learned my addr 172.21.15.116:0/4033999398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7d6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.123+0000 7f6a1affd640 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/4033999398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 msgr2=0x56523d7ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/4033999398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 msgr2=0x56523d7d6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7d6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/4033999398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56523d6c2c80 con 0x56523d7ca8f0 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7e2670 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6a140cc470 tx=0x7f6a1409aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61bfe47cf3656dff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a00ff9640 1 -- 172.21.15.116:0/4033999398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a1409b110 con 0x56523d7ca8f0 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a00ff9640 1 -- 172.21.15.116:0/4033999398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a1409f260 con 0x56523d7ca8f0 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a00ff9640 1 -- 172.21.15.116:0/4033999398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a14099520 con 0x56523d7ca8f0 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 -- 172.21.15.116:0/4033999398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 msgr2=0x56523d7e2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7e2670 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f6a140cc470 tx=0x7f6a1409aa70 comp rx=0 tx=0).stop 2024-03-24T09:29:15.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 -- 172.21.15.116:0/4033999398 shutdown_connections 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7ca8f0 0x56523d7e2670 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7d6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/4033999398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7c13f0 0x56523d7ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.124+0000 7f6a23fe8740 1 -- 172.21.15.116:0/4033999398 >> 172.21.15.116:0/4033999398 conn(0x56523d7bbb20 msgr2=0x56523d7bacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 -- 172.21.15.116:0/4033999398 shutdown_connections 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 -- 172.21.15.116:0/4033999398 wait complete. 2024-03-24T09:29:15.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 -- start start 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 Processor -- start 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7ca8f0 0x56523d7cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56523d7d2f80 con 0x56523d7c13f0 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56523d7d0200 con 0x56523d7c1d10 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56523d7d03c0 con 0x56523d7ca8f0 2024-03-24T09:29:15.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a23fe8740 10 client.0 _ll_register_callbacks cb 0x7f6a1400cf50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7d27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58818/0 (socket says 172.21.15.116:58818) 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 -- 172.21.15.116:0/1130178232 learned_addr learned my addr 172.21.15.116:0/1130178232 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1b7fe640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7ca8f0 0x56523d7cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 -- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7ca8f0 msgr2=0x56523d7cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7ca8f0 0x56523d7cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 -- 172.21.15.116:0/1130178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 msgr2=0x56523d7cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.125+0000 7f6a1affd640 1 -- 172.21.15.116:0/1130178232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a1c198720 con 0x56523d7c13f0 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a1affd640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7d27b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c004e70 tx=0x7f6a0c004ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207ede0550496113 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f69feff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f69feff5640 1 -- 172.21.15.116:0/1130178232 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0c011230 con 0x56523d7c13f0 2024-03-24T09:29:15.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f69feff5640 1 -- 172.21.15.116:0/1130178232 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f6a0c000e40 con 0x56523d7c13f0 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a0c005630 con 0x56523d7c13f0 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f69feff5640 1 -- 172.21.15.116:0/1130178232 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6a0c0054b0 con 0x56523d7c13f0 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 10 client.17685 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56523d6e8920 con 0x56523d7c13f0 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f69feff5640 1 -- 172.21.15.116:0/1130178232 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6a0c005630 con 0x56523d7c13f0 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f69feff5640 1 -- 172.21.15.116:0/1130178232 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f6a0c024440 con 0x56523d7c13f0 2024-03-24T09:29:15.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 10 client.17685 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:15.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 10 client.17685 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:29:15.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 20 client.17685 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:15.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d841d20 0x56523d844140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 4 client.17685 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:15.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56523d4b3890 con 0x56523d841d20 2024-03-24T09:29:15.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.126+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d841d20 0x56523d844140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f69feff5640 10 client.17685 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:15.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a1a7fc640 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d841d20 0x56523d844140 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d94a4e85d5acb41c server_cookie=1b3e81b8dc738813 in_seq=0 out_seq=0 2024-03-24T09:29:15.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f69feff5640 1 -- 172.21.15.116:0/1130178232 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56523d4b3890 con 0x56523d841d20 2024-03-24T09:29:15.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f69feff5640 10 client.17685 handle_command_reply: tid=0 2024-03-24T09:29:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 client.17685 shutdown 2024-03-24T09:29:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 msgr2=0x56523d7d27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7d27b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c004e70 tx=0x7f6a0c004ea0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 shutdown_connections 2024-03-24T09:29:15.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56523d7ca8f0 0x56523d7cfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x56523d841d20 0x56523d844140 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56523d7c1d10 0x56523d7cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f69feff5640 0 client.17685 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:15.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 --2- 172.21.15.116:0/1130178232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56523d7c13f0 0x56523d7d27b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 >> 172.21.15.116:0/1130178232 conn(0x56523d7bbb20 msgr2=0x56523d7ba430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 shutdown_connections 2024-03-24T09:29:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 1 -- 172.21.15.116:0/1130178232 wait complete. 2024-03-24T09:29:15.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.127+0000 7f6a23fe8740 20 client.17685 trim_cache size 0 max 16384 2024-03-24T09:29:15.151 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:15.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6a040a2230 msgr2=0x7f6a040a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6a040a2230 0x7f6a040a46f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a10006010 tx=0x7f6a1002bcd0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c10bd20 msgr2=0x7f6a1c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c10bd20 0x7f6a1c197ac0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f6a14005fd0 tx=0x7f6a1400ef10 comp rx=0 tx=0).stop 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 shutdown_connections 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a1c10bd20 0x7f6a1c197ac0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6a040a2230 0x7f6a040a46f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6a1c103270 0x7f6a1c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 --2- 172.21.15.116:0/2009785905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a1c102950 0x7f6a1c19d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 >> 172.21.15.116:0/2009785905 conn(0x7f6a1c076990 msgr2=0x7f6a1c0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 shutdown_connections 2024-03-24T09:29:15.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.128+0000 7f6a21325640 1 -- 172.21.15.116:0/2009785905 wait complete. 2024-03-24T09:29:15.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:29:15.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3294259794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf64100f60 msgr2=0x7fcf6410d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3294259794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf64100f60 0x7fcf6410d420 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c009ca0 tx=0x7fcf5c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3294259794 shutdown_connections 2024-03-24T09:29:15.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3294259794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf64100f60 0x7fcf6410d420 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3294259794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf641005a0 0x7fcf64100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3294259794 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf64106660 0x7fcf64106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3294259794 >> 172.21.15.116:0/3294259794 conn(0x7fcf64077710 msgr2=0x7fcf64077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3294259794 shutdown_connections 2024-03-24T09:29:15.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3294259794 wait complete. 2024-03-24T09:29:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 Processor -- start 2024-03-24T09:29:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.257+0000 7fcf68c8f640 1 -- start start 2024-03-24T09:29:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 0x7fcf6419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf64100f60 0x7fcf6419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf64106660 0x7fcf641979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64198030 con 0x7fcf641005a0 2024-03-24T09:29:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641981a0 con 0x7fcf64106660 2024-03-24T09:29:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64198360 con 0x7fcf64100f60 2024-03-24T09:29:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 0x7fcf6419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 0x7fcf6419d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58838/0 (socket says 172.21.15.116:58838) 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 -- 172.21.15.116:0/3282006013 learned_addr learned my addr 172.21.15.116:0/3282006013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf64106660 0x7fcf641979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf64100f60 0x7fcf6419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 -- 172.21.15.116:0/3282006013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf64100f60 msgr2=0x7fcf6419dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf64100f60 0x7fcf6419dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 -- 172.21.15.116:0/3282006013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf64106660 msgr2=0x7fcf641979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf64106660 0x7fcf641979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 -- 172.21.15.116:0/3282006013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf54009490 con 0x7fcf641005a0 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf6259c640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 0x7fcf6419d7e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ea60 tx=0x7fcf5400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf537fe640 1 -- 172.21.15.116:0/3282006013 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5400cde0 con 0x7fcf641005a0 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf537fe640 1 -- 172.21.15.116:0/3282006013 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf540044d0 con 0x7fcf641005a0 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c009900 con 0x7fcf641005a0 2024-03-24T09:29:15.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf537fe640 1 -- 172.21.15.116:0/3282006013 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54010690 con 0x7fcf641005a0 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.258+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641a46a0 con 0x7fcf641005a0 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.260+0000 7fcf537fe640 1 -- 172.21.15.116:0/3282006013 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fcf54003e40 con 0x7fcf641005a0 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 Processor -- start 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 -- start start 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf537fe640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcf4c0a2230 0x7fcf4c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b4256e2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b425650840 con 0x55b4256e2a40 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b42564dd70 con 0x55b4256e2120 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b425649550 con 0x55b4256efd00 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf61d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcf4c0a2230 0x7fcf4c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf61d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ef7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58842/0 (socket says 172.21.15.116:58842) 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf61d9b640 1 -- 172.21.15.116:0/887050839 learned_addr learned my addr 172.21.15.116:0/887050839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.261+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b4256e2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf537fe640 1 -- 172.21.15.116:0/3282006013 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fcf54014070 con 0x7fcf641005a0 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf61d9b640 1 -- 172.21.15.116:0/887050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 msgr2=0x55b4256f20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf61d9b640 1 -- 172.21.15.116:0/887050839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 msgr2=0x55b4256e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b4256e2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf61d9b640 1 -- 172.21.15.116:0/887050839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4255dede0 con 0x55b4256e2a40 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf6259c640 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b4256e2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ef7c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fcf58009c60 tx=0x7fcf58011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7eee99e4c9e1b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcf4c0a2230 0x7fcf4c0a46f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c004af0 tx=0x7fcf5c009820 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf50ff9640 1 -- 172.21.15.116:0/887050839 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58004230 con 0x55b4256e2a40 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf50ff9640 1 -- 172.21.15.116:0/887050839 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf58004390 con 0x55b4256e2a40 2024-03-24T09:29:15.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.262+0000 7fcf50ff9640 1 -- 172.21.15.116:0/887050839 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf580196b0 con 0x55b4256e2a40 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 -- 172.21.15.116:0/887050839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 msgr2=0x55b4256ef7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ef7c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fcf58009c60 tx=0x7fcf58011810 comp rx=0 tx=0).stop 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 -- 172.21.15.116:0/887050839 shutdown_connections 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ef7c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/887050839 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b4256e2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 -- 172.21.15.116:0/887050839 >> 172.21.15.116:0/887050839 conn(0x55b4256dcb20 msgr2=0x55b4256dcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 -- 172.21.15.116:0/887050839 shutdown_connections 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 -- 172.21.15.116:0/887050839 wait complete. 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 -- start start 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.263+0000 7fcf6b87a740 1 Processor -- start 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2120 0x55b4256ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ed270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256efd00 0x55b4256ed7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b42531c430 con 0x55b4256e2a40 2024-03-24T09:29:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4252d8e00 con 0x55b4256efd00 2024-03-24T09:29:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b4255ce590 con 0x55b4256e2120 2024-03-24T09:29:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 10 client.0 _ll_register_callbacks cb 0x55b42563dfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ed270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ed270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58854/0 (socket says 172.21.15.116:58854) 2024-03-24T09:29:15.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 -- 172.21.15.116:0/1539170888 learned_addr learned my addr 172.21.15.116:0/1539170888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256efd00 0x55b4256ed7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6259c640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2120 0x55b4256ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 -- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2120 msgr2=0x55b4256ecd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2120 0x55b4256ecd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 -- 172.21.15.116:0/1539170888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256efd00 msgr2=0x55b4256ed7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256efd00 0x55b4256ed7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 -- 172.21.15.116:0/1539170888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf58009890 con 0x55b4256e2a40 2024-03-24T09:29:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ed270 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcf58003eb0 tx=0x7fcf580052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12d843ae1d4daed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf49ffb640 1 -- 172.21.15.116:0/1539170888 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58004500 con 0x55b4256e2a40 2024-03-24T09:29:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf49ffb640 1 -- 172.21.15.116:0/1539170888 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fcf580035f0 con 0x55b4256e2a40 2024-03-24T09:29:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b4253487e0 con 0x55b4256e2a40 2024-03-24T09:29:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf49ffb640 1 -- 172.21.15.116:0/1539170888 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcf580037b0 con 0x55b4256e2a40 2024-03-24T09:29:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 10 client.17697 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.264+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b425647bd0 con 0x55b4256e2a40 2024-03-24T09:29:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf49ffb640 1 -- 172.21.15.116:0/1539170888 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b4253487e0 con 0x55b4256e2a40 2024-03-24T09:29:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf49ffb640 1 -- 172.21.15.116:0/1539170888 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcf58019450 con 0x55b4256e2a40 2024-03-24T09:29:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6b87a740 10 client.17697 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6b87a740 10 client.17697 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:29:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6b87a740 20 client.17697 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b4257021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6b87a740 4 client.17697 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b425385720 con 0x55b4256ffd80 2024-03-24T09:29:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6259c640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b4257021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf6259c640 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b4257021a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a4b78cdc8a264363 server_cookie=a4cb3eb223d7ebae in_seq=0 out_seq=0 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.265+0000 7fcf49ffb640 10 client.17697 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf49ffb640 1 -- 172.21.15.116:0/1539170888 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b425385720 con 0x55b4256ffd80 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf49ffb640 10 client.17697 handle_command_reply: tid=0 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 client.17697 shutdown 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 msgr2=0x55b4256ed270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ed270 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fcf58003eb0 tx=0x7fcf580052f0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 shutdown_connections 2024-03-24T09:29:15.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b4257021a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256efd00 0x55b4256ed7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2a40 0x55b4256ed270 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf49ffb640 0 client.17697 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/1539170888 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2120 0x55b4256ecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 >> 172.21.15.116:0/1539170888 conn(0x55b4256dcb20 msgr2=0x55b4256de870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 shutdown_connections 2024-03-24T09:29:15.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 1 -- 172.21.15.116:0/1539170888 wait complete. 2024-03-24T09:29:15.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.266+0000 7fcf6b87a740 20 client.17697 trim_cache size 0 max 16384 2024-03-24T09:29:15.287 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:14 smithi143 ceph-mon[35965]: pgmap v218: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:15.288 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:14 smithi143 ceph-mon[35965]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T09:29:15.288 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:14 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3754863973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:29:15.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 Processor -- start 2024-03-24T09:29:15.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 -- start start 2024-03-24T09:29:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 0x55b4256fb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2a40 0x55b425707ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b4256f20f0 con 0x55b4256e2120 2024-03-24T09:29:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4256e1760 con 0x55b4256e2a40 2024-03-24T09:29:15.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b4256ecb20 con 0x55b4256efd00 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 0x55b4256fb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 0x55b4256fb5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58860/0 (socket says 172.21.15.116:58860) 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf6259c640 1 -- 172.21.15.116:0/872684902 learned_addr learned my addr 172.21.15.116:0/872684902 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.279+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2a40 0x55b425707ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6259c640 1 -- 172.21.15.116:0/872684902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 msgr2=0x55b4256f7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6259c640 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6259c640 1 -- 172.21.15.116:0/872684902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2a40 msgr2=0x55b425707ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6259c640 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2a40 0x55b425707ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6259c640 1 -- 172.21.15.116:0/872684902 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4255e3c80 con 0x55b4256e2120 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6259c640 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 0x55b4256fb5a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cbd20 tx=0x7fcf540148c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb13558398676a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf4bfff640 1 -- 172.21.15.116:0/872684902 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5409f060 con 0x55b4256e2120 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf4bfff640 1 -- 172.21.15.116:0/872684902 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf540a36b0 con 0x55b4256e2120 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf4bfff640 1 -- 172.21.15.116:0/872684902 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54097580 con 0x55b4256e2120 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6b87a740 1 -- 172.21.15.116:0/872684902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 msgr2=0x55b4256fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 0x55b4256fb5a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cbd20 tx=0x7fcf540148c0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.280+0000 7fcf6b87a740 1 -- 172.21.15.116:0/872684902 shutdown_connections 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256efd00 0x55b4256f7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2a40 0x55b425707ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/872684902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256e2120 0x55b4256fb5a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- 172.21.15.116:0/872684902 >> 172.21.15.116:0/872684902 conn(0x55b4256dcb20 msgr2=0x55b4256eccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- 172.21.15.116:0/872684902 shutdown_connections 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- 172.21.15.116:0/872684902 wait complete. 2024-03-24T09:29:15.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- start start 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 Processor -- start 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b42575f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2a40 0x55b42575f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256efd00 0x55b4256f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b4256f41b0 con 0x55b4256efd00 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b4256f4320 con 0x55b4256e2120 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b4256f44e0 con 0x55b4256e2a40 2024-03-24T09:29:15.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6b87a740 10 client.0 _ll_register_callbacks cb 0x7fcf5400cf40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b42575f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf62d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256efd00 0x55b4256f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf61d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2a40 0x55b42575f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b42575f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:60980/0 (socket says 172.21.15.116:60980) 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6259c640 1 -- 172.21.15.116:0/4234787271 learned_addr learned my addr 172.21.15.116:0/4234787271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.281+0000 7fcf6259c640 1 -- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2a40 msgr2=0x55b42575f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6259c640 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2a40 0x55b42575f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6259c640 1 -- 172.21.15.116:0/4234787271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256efd00 msgr2=0x55b4256f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6259c640 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256efd00 0x55b4256f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6259c640 1 -- 172.21.15.116:0/4234787271 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c009900 con 0x55b4256e2120 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf62d9d640 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256efd00 0x55b4256f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6259c640 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b42575f3f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf540978e0 tx=0x7fcf54099280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a942e89daa6290d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf49ffb640 1 -- 172.21.15.116:0/4234787271 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5409f060 con 0x55b4256e2120 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf58004390 con 0x55b4256e2120 2024-03-24T09:29:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf49ffb640 1 -- 172.21.15.116:0/4234787271 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fcf540cbef0 con 0x55b4256e2120 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf49ffb640 1 -- 172.21.15.116:0/4234787271 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcf54097580 con 0x55b4256e2120 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 10 client.27524 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b425609920 con 0x55b4256e2120 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf49ffb640 1 -- 172.21.15.116:0/4234787271 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcf58004390 con 0x55b4256e2120 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf49ffb640 1 -- 172.21.15.116:0/4234787271 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcf5409b170 con 0x55b4256e2120 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 10 client.27524 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 10 client.27524 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:29:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 20 client.27524 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b425768f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 4 client.27524 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.282+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4253d4890 con 0x55b4256ffd80 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b425768f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf49ffb640 10 client.27524 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf61d9b640 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b425768f60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4b3a81b22f48e737 server_cookie=4f09586920f2a105 in_seq=0 out_seq=0 2024-03-24T09:29:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf49ffb640 1 -- 172.21.15.116:0/4234787271 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4253d4890 con 0x55b4256ffd80 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf49ffb640 10 client.27524 handle_command_reply: tid=0 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 client.27524 shutdown 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 msgr2=0x55b42575f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b42575f3f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcf540978e0 tx=0x7fcf54099280 comp rx=0 tx=0).stop 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 shutdown_connections 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55b4256ffd80 0x55b425768f60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b4256efd00 0x55b4256f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf49ffb640 0 client.27524 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b4256e2a40 0x55b42575f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 --2- 172.21.15.116:0/4234787271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4256e2120 0x55b42575f3f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 >> 172.21.15.116:0/4234787271 conn(0x55b4256dcb20 msgr2=0x55b4256ed540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 shutdown_connections 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 1 -- 172.21.15.116:0/4234787271 wait complete. 2024-03-24T09:29:15.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.283+0000 7fcf6b87a740 20 client.27524 trim_cache size 0 max 16384 2024-03-24T09:29:15.305 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:15.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcf4c0a2230 msgr2=0x7fcf4c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcf4c0a2230 0x7fcf4c0a46f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c004af0 tx=0x7fcf5c009820 comp rx=0 tx=0).stop 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 msgr2=0x7fcf6419d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 0x7fcf6419d7e0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400ea60 tx=0x7fcf5400ef20 comp rx=0 tx=0).stop 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 shutdown_connections 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf64106660 0x7fcf641979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fcf4c0a2230 0x7fcf4c0a46f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcf64100f60 0x7fcf6419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 --2- 172.21.15.116:0/3282006013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf641005a0 0x7fcf6419d7e0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 >> 172.21.15.116:0/3282006013 conn(0x7fcf64077710 msgr2=0x7fcf640ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 shutdown_connections 2024-03-24T09:29:15.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.284+0000 7fcf68c8f640 1 -- 172.21.15.116:0/3282006013 wait complete. 2024-03-24T09:29:15.307 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:14 smithi116 ceph-mon[32207]: pgmap v218: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:15.307 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:14 smithi116 ceph-mon[32207]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T09:29:15.308 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:14 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3754863973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:29:15.308 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:29:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:14 smithi117 ceph-mon[35788]: pgmap v218: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:14 smithi117 ceph-mon[35788]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T09:29:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:14 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3754863973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:29:15.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 -- 172.21.15.116:0/3547717656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 msgr2=0x7fd40410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 --2- 172.21.15.116:0/3547717656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40410d490 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc009cd0 tx=0x7fd3fc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 -- 172.21.15.116:0/3547717656 shutdown_connections 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 --2- 172.21.15.116:0/3547717656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40410d490 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 --2- 172.21.15.116:0/3547717656 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4041005c0 0x7fd404100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 --2- 172.21.15.116:0/3547717656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd404106660 0x7fd404106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.411+0000 7fd408c0e640 1 -- 172.21.15.116:0/3547717656 >> 172.21.15.116:0/3547717656 conn(0x7fd404076990 msgr2=0x7fd404076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 -- 172.21.15.116:0/3547717656 shutdown_connections 2024-03-24T09:29:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 -- 172.21.15.116:0/3547717656 wait complete. 2024-03-24T09:29:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 Processor -- start 2024-03-24T09:29:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 -- start start 2024-03-24T09:29:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4041005c0 0x7fd40419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd404106660 0x7fd4041968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd404196f30 con 0x7fd404100f80 2024-03-24T09:29:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd4041970a0 con 0x7fd404106660 2024-03-24T09:29:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.412+0000 7fd408c0e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd404197260 con 0x7fd4041005c0 2024-03-24T09:29:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40419cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58884/0 (socket says 172.21.15.116:58884) 2024-03-24T09:29:15.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 -- 172.21.15.116:0/2954288641 learned_addr learned my addr 172.21.15.116:0/2954288641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd40259c640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4041005c0 0x7fd40419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd402d9d640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd404106660 0x7fd4041968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 -- 172.21.15.116:0/2954288641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4041005c0 msgr2=0x7fd40419c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4041005c0 0x7fd40419c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 -- 172.21.15.116:0/2954288641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd404106660 msgr2=0x7fd4041968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd404106660 0x7fd4041968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 -- 172.21.15.116:0/2954288641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f4009490 con 0x7fd404100f80 2024-03-24T09:29:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40419cc70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec00e9f0 tx=0x7fd3ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fb7fe640 1 -- 172.21.15.116:0/2954288641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec009720 con 0x7fd404100f80 2024-03-24T09:29:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fb7fe640 1 -- 172.21.15.116:0/2954288641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3ec00cd80 con 0x7fd404100f80 2024-03-24T09:29:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd3fb7fe640 1 -- 172.21.15.116:0/2954288641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec010430 con 0x7fd404100f80 2024-03-24T09:29:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3fc009900 con 0x7fd404100f80 2024-03-24T09:29:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.413+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4041a35f0 con 0x7fd404100f80 2024-03-24T09:29:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.415+0000 7fd3fb7fe640 1 -- 172.21.15.116:0/2954288641 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd3ec005020 con 0x7fd404100f80 2024-03-24T09:29:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 Processor -- start 2024-03-24T09:29:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 -- start start 2024-03-24T09:29:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d64d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d6563b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d658ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b2d5bb840 con 0x561b2d64d3f0 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561b2d5b8d70 con 0x561b2d64dd10 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561b2d5b4550 con 0x561b2d6568f0 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd3fb7fe640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3f00a2230 0x7fd3f00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd402d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d64d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d6563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd402d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d64d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58890/0 (socket says 172.21.15.116:58890) 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd40259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d6563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:32772/0 (socket says 172.21.15.116:32772) 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd402d9d640 1 -- 172.21.15.116:0/3245403013 learned_addr learned my addr 172.21.15.116:0/3245403013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd3fb7fe640 1 -- 172.21.15.116:0/2954288641 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd3ec014070 con 0x7fd404100f80 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d658ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.416+0000 7fd402d9d640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3f00a2230 0x7fd3f00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd40259c640 1 -- 172.21.15.116:0/3245403013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 msgr2=0x561b2d658ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd40259c640 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d658ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd40259c640 1 -- 172.21.15.116:0/3245403013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 msgr2=0x561b2d64d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd40259c640 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d64d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd40259c640 1 -- 172.21.15.116:0/3245403013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2d549de0 con 0x561b2d64dd10 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd402d9d640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3f00a2230 0x7fd3f00a46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc006010 tx=0x7fd3fc005c50 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd40259c640 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d6563b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd404101700 tx=0x7fd3f4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=190b16a9e9cde67b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd3f8ff9640 1 -- 172.21.15.116:0/3245403013 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4004250 con 0x561b2d64dd10 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd3f8ff9640 1 -- 172.21.15.116:0/3245403013 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3f40043b0 con 0x561b2d64dd10 2024-03-24T09:29:15.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.417+0000 7fd3f8ff9640 1 -- 172.21.15.116:0/3245403013 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4004c40 con 0x561b2d64dd10 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3245403013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 msgr2=0x561b2d6563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d6563b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd404101700 tx=0x7fd3f4011810 comp rx=0 tx=0).stop 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3245403013 shutdown_connections 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d658ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d6563b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3245403013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d64d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3245403013 >> 172.21.15.116:0/3245403013 conn(0x561b2d647b20 msgr2=0x561b2d647f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3245403013 shutdown_connections 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3245403013 wait complete. 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 -- start start 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 Processor -- start 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.418+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d6538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d653e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d654360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b2d287430 con 0x561b2d64d3f0 2024-03-24T09:29:15.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561b2d243e00 con 0x561b2d6568f0 2024-03-24T09:29:15.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561b2d539590 con 0x561b2d64dd10 2024-03-24T09:29:15.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 10 client.0 _ll_register_callbacks cb 0x561b2d5a8fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:15.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d6538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d6538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58902/0 (socket says 172.21.15.116:58902) 2024-03-24T09:29:15.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 -- 172.21.15.116:0/3676754211 learned_addr learned my addr 172.21.15.116:0/3676754211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d653e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd402d9d640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d654360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 -- 172.21.15.116:0/3676754211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 msgr2=0x561b2d653e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d653e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 -- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 msgr2=0x561b2d654360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d654360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 -- 172.21.15.116:0/3676754211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3f40099e0 con 0x561b2d64d3f0 2024-03-24T09:29:15.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40259c640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d6538e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40048a0 tx=0x7fd3f40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d2e160da4d31c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd3e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676754211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4003ea0 con 0x561b2d64d3f0 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676754211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd3f4023430 con 0x561b2d64d3f0 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561b2d2b37e0 con 0x561b2d64d3f0 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676754211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3f40235f0 con 0x561b2d64d3f0 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 10 client.17715 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:15.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561b2d5b2bd0 con 0x561b2d64d3f0 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.419+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676754211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561b2d2b37e0 con 0x561b2d64d3f0 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676754211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd3f4021040 con 0x561b2d64d3f0 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd40b7eb740 10 client.17715 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd40b7eb740 10 client.17715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd40b7eb740 20 client.17715 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6668b0 0x561b2d668cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd40b7eb740 4 client.17715 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:15.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b2d2f0720 con 0x561b2d6668b0 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6668b0 0x561b2d668cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd3e9ffb640 10 client.17715 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.420+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6668b0 0x561b2d668cc0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e08720968cc02773 server_cookie=60c1613decaa3fc4 in_seq=0 out_seq=0 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676754211 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b2d2f0720 con 0x561b2d6668b0 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd3e9ffb640 10 client.17715 handle_command_reply: tid=0 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 client.17715 shutdown 2024-03-24T09:29:15.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 msgr2=0x561b2d6538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d6538e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40048a0 tx=0x7fd3f40038a0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 shutdown_connections 2024-03-24T09:29:15.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6668b0 0x561b2d668cc0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d654360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd3e9ffb640 0 client.17715 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:15.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d653e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676754211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d6538e0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 >> 172.21.15.116:0/3676754211 conn(0x561b2d647b20 msgr2=0x561b2d649870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.421+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 shutdown_connections 2024-03-24T09:29:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.422+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676754211 wait complete. 2024-03-24T09:29:15.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.422+0000 7fd40b7eb740 20 client.17715 trim_cache size 0 max 16384 2024-03-24T09:29:15.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 Processor -- start 2024-03-24T09:29:15.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 -- start start 2024-03-24T09:29:15.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d656010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d662180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d66e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b2d6592b0 con 0x561b2d64d3f0 2024-03-24T09:29:15.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561b2d64d060 con 0x561b2d64dd10 2024-03-24T09:29:15.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561b2d6486f0 con 0x561b2d6568f0 2024-03-24T09:29:15.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d656010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d656010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58908/0 (socket says 172.21.15.116:58908) 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 -- 172.21.15.116:0/4225784437 learned_addr learned my addr 172.21.15.116:0/4225784437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d662180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd402d9d640 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d66e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 -- 172.21.15.116:0/4225784437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 msgr2=0x561b2d66e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d66e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 -- 172.21.15.116:0/4225784437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 msgr2=0x561b2d662180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d662180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 -- 172.21.15.116:0/4225784437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b2d54ec80 con 0x561b2d64d3f0 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d662180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd40259c640 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d656010 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40099b0 tx=0x7fd3f4005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca85265bad21492c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd3ebfff640 1 -- 172.21.15.116:0/4225784437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4011d10 con 0x561b2d64d3f0 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd3ebfff640 1 -- 172.21.15.116:0/4225784437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3f4003ca0 con 0x561b2d64d3f0 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.435+0000 7fd3ebfff640 1 -- 172.21.15.116:0/4225784437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4019690 con 0x561b2d64d3f0 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 -- 172.21.15.116:0/4225784437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 msgr2=0x561b2d656010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d656010 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40099b0 tx=0x7fd3f4005b90 comp rx=0 tx=0).stop 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 -- 172.21.15.116:0/4225784437 shutdown_connections 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d6568f0 0x561b2d66e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d64dd10 0x561b2d662180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/4225784437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d656010 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 -- 172.21.15.116:0/4225784437 >> 172.21.15.116:0/4225784437 conn(0x561b2d647b20 msgr2=0x561b2d646cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 -- 172.21.15.116:0/4225784437 shutdown_connections 2024-03-24T09:29:15.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 -- 172.21.15.116:0/4225784437 wait complete. 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 -- start start 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.436+0000 7fd40b7eb740 1 Processor -- start 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d65a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d65a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d65ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561b2d6d0010 con 0x561b2d64d3f0 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561b2d6d0150 con 0x561b2d6568f0 2024-03-24T09:29:15.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561b2d6d0310 con 0x561b2d64dd10 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 10 client.0 _ll_register_callbacks cb 0x7fd3f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d65a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d65a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58920/0 (socket says 172.21.15.116:58920) 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 -- 172.21.15.116:0/3676625708 learned_addr learned my addr 172.21.15.116:0/3676625708 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d65a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd402d9d640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d65ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 -- 172.21.15.116:0/3676625708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 msgr2=0x561b2d65a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d65a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 -- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 msgr2=0x561b2d65ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d65ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 -- 172.21.15.116:0/3676625708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b2d5b2bd0 con 0x561b2d64d3f0 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40259c640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d65a210 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4019e40 tx=0x7fd3f4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4f2126a2da5e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd3e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676625708 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3f4019430 con 0x561b2d64d3f0 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676625708 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fd3f4000e40 con 0x561b2d64d3f0 2024-03-24T09:29:15.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3f40043b0 con 0x561b2d64d3f0 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676625708 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3f4019590 con 0x561b2d64d3f0 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 10 client.17721 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.437+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561b2d574920 con 0x561b2d64d3f0 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676625708 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3f40043b0 con 0x561b2d64d3f0 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676625708 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd3f40052c0 con 0x561b2d64d3f0 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 10 client.17721 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 10 client.17721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 20 client.17721 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:15.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6d16c0 0x561b2d6d3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 4 client.17721 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b2d33f890 con 0x561b2d6d16c0 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6d16c0 0x561b2d6d3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3e9ffb640 10 client.17721 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3fbfff640 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6d16c0 0x561b2d6d3ae0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27e71cf6cca58c server_cookie=af8472e303161fd8 in_seq=0 out_seq=0 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3e9ffb640 1 -- 172.21.15.116:0/3676625708 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b2d33f890 con 0x561b2d6d16c0 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3e9ffb640 10 client.17721 handle_command_reply: tid=0 2024-03-24T09:29:15.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 client.17721 shutdown 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 msgr2=0x561b2d65a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d65a210 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4019e40 tx=0x7fd3f4011810 comp rx=0 tx=0).stop 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 shutdown_connections 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x561b2d6d16c0 0x561b2d6d3ae0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561b2d6568f0 0x561b2d65ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561b2d64dd10 0x561b2d65a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd3e9ffb640 0 client.17721 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:15.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 --2- 172.21.15.116:0/3676625708 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561b2d64d3f0 0x561b2d65a210 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.438+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 >> 172.21.15.116:0/3676625708 conn(0x561b2d647b20 msgr2=0x561b2d653880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 shutdown_connections 2024-03-24T09:29:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd40b7eb740 1 -- 172.21.15.116:0/3676625708 wait complete. 2024-03-24T09:29:15.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd40b7eb740 20 client.17721 trim_cache size 0 max 16384 2024-03-24T09:29:15.456 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3f00a2230 msgr2=0x7fd3f00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3f00a2230 0x7fd3f00a46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc006010 tx=0x7fd3fc005c50 comp rx=0 tx=0).stop 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 msgr2=0x7fd40419cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40419cc70 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec00e9f0 tx=0x7fd3ec00edf0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 shutdown_connections 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd404106660 0x7fd4041968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd404100f80 0x7fd40419cc70 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3f00a2230 0x7fd3f00a46f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 --2- 172.21.15.116:0/2954288641 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4041005c0 0x7fd40419c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 >> 172.21.15.116:0/2954288641 conn(0x7fd404076990 msgr2=0x7fd4040feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 shutdown_connections 2024-03-24T09:29:15.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:15.439+0000 7fd408c0e640 1 -- 172.21.15.116:0/2954288641 wait complete. 2024-03-24T09:29:16.460 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:29:16.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:29:16.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:29:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 -- 172.21.15.116:0/2335850901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7010bd50 msgr2=0x7fce7010e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 --2- 172.21.15.116:0/2335850901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7010bd50 0x7fce7010e1c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fce68009ca0 tx=0x7fce6802c180 comp rx=0 tx=0).stop 2024-03-24T09:29:16.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 -- 172.21.15.116:0/2335850901 shutdown_connections 2024-03-24T09:29:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 --2- 172.21.15.116:0/2335850901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce7010bd50 0x7fce7010e1c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 --2- 172.21.15.116:0/2335850901 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce701032a0 0x7fce7010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 --2- 172.21.15.116:0/2335850901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 0x7fce70102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 -- 172.21.15.116:0/2335850901 >> 172.21.15.116:0/2335850901 conn(0x7fce70077770 msgr2=0x7fce70077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 -- 172.21.15.116:0/2335850901 shutdown_connections 2024-03-24T09:29:16.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 -- 172.21.15.116:0/2335850901 wait complete. 2024-03-24T09:29:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.578+0000 7fce76c0e640 1 Processor -- start 2024-03-24T09:29:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 -- start start 2024-03-24T09:29:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 0x7fce7019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 0x7fce7019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce7010bd50 0x7fce70197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce70198100 con 0x7fce701032a0 2024-03-24T09:29:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fce70198270 con 0x7fce70102980 2024-03-24T09:29:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce76c0e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fce70198430 con 0x7fce7010bd50 2024-03-24T09:29:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 0x7fce7019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 0x7fce7019de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38900/0 (socket says 172.21.15.116:38900) 2024-03-24T09:29:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 -- 172.21.15.116:0/1161762500 learned_addr learned my addr 172.21.15.116:0/1161762500 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce749aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 0x7fce7019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce751ab640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce7010bd50 0x7fce70197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 -- 172.21.15.116:0/1161762500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce7010bd50 msgr2=0x7fce70197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce7010bd50 0x7fce70197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 -- 172.21.15.116:0/1161762500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 msgr2=0x7fce7019d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 0x7fce7019d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 -- 172.21.15.116:0/1161762500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce68009900 con 0x7fce701032a0 2024-03-24T09:29:16.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce749aa640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 0x7fce7019d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:16.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6ffff640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 0x7fce7019de40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00e9f0 tx=0x7fce5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.579+0000 7fce6dffb640 1 -- 172.21.15.116:0/1161762500 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c009720 con 0x7fce701032a0 2024-03-24T09:29:16.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.580+0000 7fce6dffb640 1 -- 172.21.15.116:0/1161762500 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce5c00cd80 con 0x7fce701032a0 2024-03-24T09:29:16.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.580+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce70198710 con 0x7fce701032a0 2024-03-24T09:29:16.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.580+0000 7fce6dffb640 1 -- 172.21.15.116:0/1161762500 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c010430 con 0x7fce701032a0 2024-03-24T09:29:16.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.580+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce701a49a0 con 0x7fce701032a0 2024-03-24T09:29:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce6dffb640 1 -- 172.21.15.116:0/1161762500 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fce5c005020 con 0x7fce701032a0 2024-03-24T09:29:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce797ee740 1 Processor -- start 2024-03-24T09:29:16.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce6dffb640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fce600a2230 0x7fce600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce797ee740 1 -- start start 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce6dffb640 1 -- 172.21.15.116:0/1161762500 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fce5c014070 con 0x7fce701032a0 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce751ab640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fce600a2230 0x7fce600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.582+0000 7fce797ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b7425b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce797ee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce797ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b742ec60 0x5649b7431050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce797ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649b7393530 con 0x5649b742ec60 2024-03-24T09:29:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce797ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649b7390a60 con 0x5649b7425760 2024-03-24T09:29:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce797ee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649b738c240 con 0x5649b7426080 2024-03-24T09:29:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b7425b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b7425b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46050/0 (socket says 172.21.15.116:46050) 2024-03-24T09:29:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 -- 172.21.15.116:0/2439254549 learned_addr learned my addr 172.21.15.116:0/2439254549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce6ffff640 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce751ab640 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b742ec60 0x5649b7431050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce751ab640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fce600a2230 0x7fce600a46f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce68004af0 tx=0x7fce680324c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 -- 172.21.15.116:0/2439254549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 msgr2=0x5649b742e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 -- 172.21.15.116:0/2439254549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b742ec60 msgr2=0x5649b7431050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b742ec60 0x5649b7431050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 -- 172.21.15.116:0/2439254549 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b738a8c0 con 0x5649b7425760 2024-03-24T09:29:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce749aa640 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b7425b40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fce640114f0 tx=0x7fce640118f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a68aa35b95adc2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce5affd640 1 -- 172.21.15.116:0/2439254549 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64004030 con 0x5649b7425760 2024-03-24T09:29:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce5affd640 1 -- 172.21.15.116:0/2439254549 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce640035f0 con 0x5649b7425760 2024-03-24T09:29:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.583+0000 7fce5affd640 1 -- 172.21.15.116:0/2439254549 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6400f720 con 0x5649b7425760 2024-03-24T09:29:16.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 -- 172.21.15.116:0/2439254549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 msgr2=0x5649b7425b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b7425b40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fce640114f0 tx=0x7fce640118f0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 -- 172.21.15.116:0/2439254549 shutdown_connections 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b742ec60 0x5649b7431050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 --2- 172.21.15.116:0/2439254549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b7425b40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 -- 172.21.15.116:0/2439254549 >> 172.21.15.116:0/2439254549 conn(0x5649b741fe90 msgr2=0x5649b7420270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 -- 172.21.15.116:0/2439254549 shutdown_connections 2024-03-24T09:29:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 -- 172.21.15.116:0/2439254549 wait complete. 2024-03-24T09:29:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 -- start start 2024-03-24T09:29:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.584+0000 7fce797ee740 1 Processor -- start 2024-03-24T09:29:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7425760 0x5649b742bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b742c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b742c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649b7330f80 con 0x5649b7426080 2024-03-24T09:29:16.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649b705f4b0 con 0x5649b742ec60 2024-03-24T09:29:16.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649b7311ad0 con 0x5649b7425760 2024-03-24T09:29:16.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce797ee740 10 client.0 _ll_register_callbacks cb 0x5649b7344b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:16.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b742c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b742c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38924/0 (socket says 172.21.15.116:38924) 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 -- 172.21.15.116:0/3058635626 learned_addr learned my addr 172.21.15.116:0/3058635626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce751ab640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b742c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce749aa640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7425760 0x5649b742bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 -- 172.21.15.116:0/3058635626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7425760 msgr2=0x5649b742bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7425760 0x5649b742bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 -- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 msgr2=0x5649b742c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b742c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce6ffff640 1 -- 172.21.15.116:0/3058635626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce64009cc0 con 0x5649b7426080 2024-03-24T09:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.585+0000 7fce751ab640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b742c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce6ffff640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b742c170 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009380 tx=0x7fce5c0047f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b65fb199b0dc53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce4ffff640 1 -- 172.21.15.116:0/3058635626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c09b1d0 con 0x5649b7426080 2024-03-24T09:29:16.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce4ffff640 1 -- 172.21.15.116:0/3058635626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fce5c0b92a0 con 0x5649b7426080 2024-03-24T09:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5649b73b07a0 con 0x5649b7426080 2024-03-24T09:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce4ffff640 1 -- 172.21.15.116:0/3058635626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fce5c09eec0 con 0x5649b7426080 2024-03-24T09:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 10 client.17733 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5649b73426d0 con 0x5649b7426080 2024-03-24T09:29:16.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce4ffff640 1 -- 172.21.15.116:0/3058635626 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5649b73b07a0 con 0x5649b7426080 2024-03-24T09:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce4ffff640 1 -- 172.21.15.116:0/3058635626 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fce5c085040 con 0x5649b7426080 2024-03-24T09:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 10 client.17733 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 10 client.17733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 20 client.17733 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:16.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b743ed90 0x5649b74411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 4 client.17733 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649b73821f0 con 0x5649b743ed90 2024-03-24T09:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.586+0000 7fce749aa640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b743ed90 0x5649b74411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce4ffff640 10 client.17733 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce749aa640 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b743ed90 0x5649b74411b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d623347a6a0b470d server_cookie=4c38cc5a98feb671 in_seq=0 out_seq=0 2024-03-24T09:29:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce4ffff640 1 -- 172.21.15.116:0/3058635626 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649b73821f0 con 0x5649b743ed90 2024-03-24T09:29:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce4ffff640 10 client.17733 handle_command_reply: tid=0 2024-03-24T09:29:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 client.17733 shutdown 2024-03-24T09:29:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 msgr2=0x5649b742c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b742c170 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009380 tx=0x7fce5c0047f0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 shutdown_connections 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b743ed90 0x5649b74411b0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b742c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b742c170 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 --2- 172.21.15.116:0/3058635626 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7425760 0x5649b742bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce4ffff640 0 client.17733 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.587+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 >> 172.21.15.116:0/3058635626 conn(0x5649b741fe90 msgr2=0x5649b7421be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.588+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 shutdown_connections 2024-03-24T09:29:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.588+0000 7fce797ee740 1 -- 172.21.15.116:0/3058635626 wait complete. 2024-03-24T09:29:16.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.588+0000 7fce797ee740 20 client.17733 trim_cache size 0 max 16384 2024-03-24T09:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 Processor -- start 2024-03-24T09:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 -- start start 2024-03-24T09:29:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 0x5649b742e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b7446a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649b743d570 con 0x5649b7425760 2024-03-24T09:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649b7425130 con 0x5649b742ec60 2024-03-24T09:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce797ee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649b742a960 con 0x5649b7426080 2024-03-24T09:29:16.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 0x5649b742e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 0x5649b742e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38926/0 (socket says 172.21.15.116:38926) 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 -- 172.21.15.116:0/1274903676 learned_addr learned my addr 172.21.15.116:0/1274903676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce751ab640 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b7446a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce6ffff640 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 -- 172.21.15.116:0/1274903676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 msgr2=0x5649b742c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 -- 172.21.15.116:0/1274903676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 msgr2=0x5649b7446a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b7446a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 -- 172.21.15.116:0/1274903676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b7326c60 con 0x5649b7425760 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.605+0000 7fce749aa640 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 0x5649b742e360 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fce64000c00 tx=0x7fce640038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2215c1bffc6fe7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce5a7fc640 1 -- 172.21.15.116:0/1274903676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6402d070 con 0x5649b7425760 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce5a7fc640 1 -- 172.21.15.116:0/1274903676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce64009b50 con 0x5649b7425760 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce5a7fc640 1 -- 172.21.15.116:0/1274903676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce64005200 con 0x5649b7425760 2024-03-24T09:29:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 -- 172.21.15.116:0/1274903676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 msgr2=0x5649b742e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 0x5649b742e360 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fce64000c00 tx=0x7fce640038c0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 -- 172.21.15.116:0/1274903676 shutdown_connections 2024-03-24T09:29:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b742ec60 0x5649b7446a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b7426080 0x5649b742c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 --2- 172.21.15.116:0/1274903676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7425760 0x5649b742e360 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 -- 172.21.15.116:0/1274903676 >> 172.21.15.116:0/1274903676 conn(0x5649b741fe90 msgr2=0x5649b74210c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 -- 172.21.15.116:0/1274903676 shutdown_connections 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 -- 172.21.15.116:0/1274903676 wait complete. 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.606+0000 7fce797ee740 1 -- start start 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 Processor -- start 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b74a2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b74a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b742ec60 0x5649b7432c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649b7433200 con 0x5649b7426080 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5649b7433370 con 0x5649b7425760 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5649b7433530 con 0x5649b742ec60 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce797ee740 10 client.0 _ll_register_callbacks cb 0x7fce5c09a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b74a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce749aa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b74a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b74a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38930/0 (socket says 172.21.15.116:38930) 2024-03-24T09:29:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 -- 172.21.15.116:0/677657858 learned_addr learned my addr 172.21.15.116:0/677657858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce751ab640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b742ec60 0x5649b7432c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 -- 172.21.15.116:0/677657858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b742ec60 msgr2=0x5649b7432c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b742ec60 0x5649b7432c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 -- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 msgr2=0x5649b74a2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b74a2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 -- 172.21.15.116:0/677657858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b738a8c0 con 0x5649b7426080 2024-03-24T09:29:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce749aa640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b74a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce6ffff640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b74a2d80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0971d0 tx=0x7fce5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e8e0a4be466e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.607+0000 7fce4ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce4ffff640 1 -- 172.21.15.116:0/677657858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0ce610 con 0x5649b7426080 2024-03-24T09:29:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce4ffff640 1 -- 172.21.15.116:0/677657858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fce5c0b92a0 con 0x5649b7426080 2024-03-24T09:29:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce640035f0 con 0x5649b7426080 2024-03-24T09:29:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce4ffff640 1 -- 172.21.15.116:0/677657858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fce5c09eec0 con 0x5649b7426080 2024-03-24T09:29:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 10 client.17739 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5649b734c970 con 0x5649b7426080 2024-03-24T09:29:16.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce4ffff640 1 -- 172.21.15.116:0/677657858 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fce640035f0 con 0x5649b7426080 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce4ffff640 1 -- 172.21.15.116:0/677657858 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fce5c085040 con 0x5649b7426080 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 10 client.17739 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 10 client.17739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 20 client.17739 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b74aa2e0 0x5649b74ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 4 client.17739 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:29:16.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5649b708b860 con 0x5649b74aa2e0 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.608+0000 7fce749aa640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b74aa2e0 0x5649b74ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.609+0000 7fce749aa640 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b74aa2e0 0x5649b74ac6d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15fa4e70f07ed7dc server_cookie=52ad9cf6bd86b6f in_seq=0 out_seq=0 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.609+0000 7fce4ffff640 10 client.17739 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce4ffff640 1 -- 172.21.15.116:0/677657858 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5649b708b860 con 0x5649b74aa2e0 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce4ffff640 10 client.17739 handle_command_reply: tid=0 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 client.17739 shutdown 2024-03-24T09:29:16.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 msgr2=0x5649b74a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b74a2d80 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fce5c0971d0 tx=0x7fce5c0046c0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 shutdown_connections 2024-03-24T09:29:16.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5649b742ec60 0x5649b7432c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649b7426080 0x5649b74a2d80 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5649b74aa2e0 0x5649b74ac6d0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 --2- 172.21.15.116:0/677657858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5649b7425760 0x5649b74a2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 >> 172.21.15.116:0/677657858 conn(0x5649b741fe90 msgr2=0x5649b74218e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce4ffff640 0 client.17739 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 shutdown_connections 2024-03-24T09:29:16.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 1 -- 172.21.15.116:0/677657858 wait complete. 2024-03-24T09:29:16.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.610+0000 7fce797ee740 20 client.17739 trim_cache size 0 max 16384 2024-03-24T09:29:16.631 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:29:16.631 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:29:16.631 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "569b3015-9e1a-43a1-ac6e-2f918709dc59", 2024-03-24T09:29:16.631 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:29:16.632 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:29:16.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fce600a2230 msgr2=0x7fce600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fce600a2230 0x7fce600a46f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce68004af0 tx=0x7fce680324c0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 msgr2=0x7fce7019de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 0x7fce7019de40 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00e9f0 tx=0x7fce5c00edf0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 shutdown_connections 2024-03-24T09:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fce600a2230 0x7fce600a46f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fce7010bd50 0x7fce70197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce701032a0 0x7fce7019de40 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 --2- 172.21.15.116:0/1161762500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce70102980 0x7fce7019d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 >> 172.21.15.116:0/1161762500 conn(0x7fce70077770 msgr2=0x7fce700fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 shutdown_connections 2024-03-24T09:29:16.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.611+0000 7fce76c0e640 1 -- 172.21.15.116:0/1161762500 wait complete. 2024-03-24T09:29:16.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:29:16.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:29:16.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 -- 172.21.15.116:0/319433241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 msgr2=0x7feb74100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 --2- 172.21.15.116:0/319433241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb74100030 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7feb68009930 tx=0x7feb6802bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:16.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 -- 172.21.15.116:0/319433241 shutdown_connections 2024-03-24T09:29:16.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 --2- 172.21.15.116:0/319433241 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb74100570 0x7feb7410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 --2- 172.21.15.116:0/319433241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb74100030 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 --2- 172.21.15.116:0/319433241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb741076b0 0x7feb74107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 -- 172.21.15.116:0/319433241 >> 172.21.15.116:0/319433241 conn(0x7feb74076b30 msgr2=0x7feb740fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 -- 172.21.15.116:0/319433241 shutdown_connections 2024-03-24T09:29:16.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 -- 172.21.15.116:0/319433241 wait complete. 2024-03-24T09:29:16.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.764+0000 7feb7a38a640 1 Processor -- start 2024-03-24T09:29:16.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 -- start start 2024-03-24T09:29:16.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb7419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb74100570 0x7feb7419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 0x7feb7419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb7419eee0 con 0x7feb740ffbb0 2024-03-24T09:29:16.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feb7419f050 con 0x7feb74100570 2024-03-24T09:29:16.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb7a38a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7feb7419f210 con 0x7feb741076b0 2024-03-24T09:29:16.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb7419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb78927640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 0x7feb7419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb78927640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 0x7feb7419e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:36554/0 (socket says 172.21.15.116:36554) 2024-03-24T09:29:16.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb7419a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38950/0 (socket says 172.21.15.116:38950) 2024-03-24T09:29:16.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb78927640 1 -- 172.21.15.116:0/922463373 learned_addr learned my addr 172.21.15.116:0/922463373 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.765+0000 7feb737fe640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb74100570 0x7feb7419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb78927640 1 -- 172.21.15.116:0/922463373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb74100570 msgr2=0x7feb7419a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb78927640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb74100570 0x7feb7419a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb78927640 1 -- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 msgr2=0x7feb7419a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb78927640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb7419a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb78927640 1 -- 172.21.15.116:0/922463373 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb68009560 con 0x7feb741076b0 2024-03-24T09:29:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb73fff640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb7419a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.766+0000 7feb78927640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 0x7feb7419e8e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b970 tx=0x7feb6c00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.767+0000 7feb717fa640 1 -- 172.21.15.116:0/922463373 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c00caa0 con 0x7feb741076b0 2024-03-24T09:29:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.767+0000 7feb717fa640 1 -- 172.21.15.116:0/922463373 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb6c00cc00 con 0x7feb741076b0 2024-03-24T09:29:16.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.767+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7419f4f0 con 0x7feb741076b0 2024-03-24T09:29:16.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.767+0000 7feb717fa640 1 -- 172.21.15.116:0/922463373 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0075d0 con 0x7feb741076b0 2024-03-24T09:29:16.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.767+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb741a6d70 con 0x7feb741076b0 2024-03-24T09:29:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 Processor -- start 2024-03-24T09:29:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb717fa640 1 -- 172.21.15.116:0/922463373 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7feb6c014440 con 0x7feb741076b0 2024-03-24T09:29:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 -- start start 2024-03-24T09:29:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a26dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a27300 0x559a92a2f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a322d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559a92942f30 con 0x559a92a269e0 2024-03-24T09:29:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559a9298c810 con 0x559a92a27300 2024-03-24T09:29:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.770+0000 7feb7cf68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559a92989d40 con 0x559a92a2fee0 2024-03-24T09:29:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb717fa640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7feb5c0a22b0 0x7feb5c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb73fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a27300 0x559a92a2f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb73fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a27300 0x559a92a2f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:46092/0 (socket says 172.21.15.116:46092) 2024-03-24T09:29:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb73fff640 1 -- 172.21.15.116:0/562283452 learned_addr learned my addr 172.21.15.116:0/562283452 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb737fe640 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb737fe640 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a26dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb78927640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7feb5c0a22b0 0x7feb5c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.771+0000 7feb717fa640 1 -- 172.21.15.116:0/922463373 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7feb6c0cdb20 con 0x7feb741076b0 2024-03-24T09:29:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb78927640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7feb5c0a22b0 0x7feb5c0a4770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0cb020 tx=0x7feb6c021000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb737fe640 1 -- 172.21.15.116:0/562283452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 msgr2=0x559a92a322d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb737fe640 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a322d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb737fe640 1 -- 172.21.15.116:0/562283452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a27300 msgr2=0x559a92a2f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb737fe640 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a27300 0x559a92a2f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb737fe640 1 -- 172.21.15.116:0/562283452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a92946ea0 con 0x559a92a269e0 2024-03-24T09:29:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb737fe640 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a26dc0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7feb68036040 tx=0x7feb6802c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963cb3d4fd946038 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb5a7fc640 1 -- 172.21.15.116:0/562283452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68004af0 con 0x559a92a269e0 2024-03-24T09:29:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb5a7fc640 1 -- 172.21.15.116:0/562283452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb680044e0 con 0x559a92a269e0 2024-03-24T09:29:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.772+0000 7feb5a7fc640 1 -- 172.21.15.116:0/562283452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68003df0 con 0x559a92a269e0 2024-03-24T09:29:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.773+0000 7feb7cf68740 1 -- 172.21.15.116:0/562283452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 msgr2=0x559a92a26dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.773+0000 7feb7cf68740 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a26dc0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7feb68036040 tx=0x7feb6802c6b0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.773+0000 7feb7cf68740 1 -- 172.21.15.116:0/562283452 shutdown_connections 2024-03-24T09:29:16.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.773+0000 7feb7cf68740 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.773+0000 7feb7cf68740 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a27300 0x559a92a2f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.774+0000 7feb7cf68740 1 --2- 172.21.15.116:0/562283452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a26dc0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.774+0000 7feb7cf68740 1 -- 172.21.15.116:0/562283452 >> 172.21.15.116:0/562283452 conn(0x559a92a21110 msgr2=0x559a92a214f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.774+0000 7feb7cf68740 1 -- 172.21.15.116:0/562283452 shutdown_connections 2024-03-24T09:29:16.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.774+0000 7feb7cf68740 1 -- 172.21.15.116:0/562283452 wait complete. 2024-03-24T09:29:16.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.774+0000 7feb7cf68740 1 -- start start 2024-03-24T09:29:16.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.774+0000 7feb7cf68740 1 Processor -- start 2024-03-24T09:29:16.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a2cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a2d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a2fee0 0x559a92a2da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559a92985520 con 0x559a92a269e0 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559a92658430 con 0x559a92a2fee0 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559a92614e00 con 0x559a92a27300 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb7cf68740 10 client.0 _ll_register_callbacks cb 0x559a92923d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a2cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a2cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38970/0 (socket says 172.21.15.116:38970) 2024-03-24T09:29:16.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 -- 172.21.15.116:0/3011169058 learned_addr learned my addr 172.21.15.116:0/3011169058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb73fff640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a2d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a2fee0 0x559a92a2da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 -- 172.21.15.116:0/3011169058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 msgr2=0x559a92a2d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a2d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 -- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a2fee0 msgr2=0x559a92a2da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a2fee0 0x559a92a2da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.775+0000 7feb737fe640 1 -- 172.21.15.116:0/3011169058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7410edc0 con 0x559a92a269e0 2024-03-24T09:29:16.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb737fe640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a2cfb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7feb6802cc90 tx=0x7feb68000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68b31173e5ea3da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb537fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:16.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb537fe640 1 -- 172.21.15.116:0/3011169058 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68003f80 con 0x559a92a269e0 2024-03-24T09:29:16.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb537fe640 1 -- 172.21.15.116:0/3011169058 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7feb6802de40 con 0x559a92a269e0 2024-03-24T09:29:16.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559a926847e0 con 0x559a92a269e0 2024-03-24T09:29:16.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb537fe640 1 -- 172.21.15.116:0/3011169058 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feb68003920 con 0x559a92a269e0 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.776+0000 7feb7cf68740 10 client.17751 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559a92983ba0 con 0x559a92a269e0 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb537fe640 1 -- 172.21.15.116:0/3011169058 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559a926847e0 con 0x559a92a269e0 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb537fe640 1 -- 172.21.15.116:0/3011169058 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7feb6805a070 con 0x559a92a269e0 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 10 client.17751 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 10 client.17751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:16.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 20 client.17751 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92a3ff10 0x559a92a42330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 4 client.17751 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.777+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a926c1720 con 0x559a92a3ff10 2024-03-24T09:29:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.778+0000 7feb73fff640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92a3ff10 0x559a92a42330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.778+0000 7feb537fe640 10 client.17751 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.778+0000 7feb73fff640 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92a3ff10 0x559a92a42330 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37d8f0dc27f87a2 server_cookie=e341eee2ee50b037 in_seq=0 out_seq=0 2024-03-24T09:29:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.779+0000 7feb537fe640 1 -- 172.21.15.116:0/3011169058 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559a926c1720 con 0x559a92a3ff10 2024-03-24T09:29:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.779+0000 7feb537fe640 10 client.17751 handle_command_reply: tid=0 2024-03-24T09:29:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.779+0000 7feb7cf68740 1 client.17751 shutdown 2024-03-24T09:29:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 msgr2=0x559a92a2cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a2cfb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7feb6802cc90 tx=0x7feb68000bc0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 shutdown_connections 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92a3ff10 0x559a92a42330 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a2fee0 0x559a92a2da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a2d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 --2- 172.21.15.116:0/3011169058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a269e0 0x559a92a2cfb0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 >> 172.21.15.116:0/3011169058 conn(0x559a92a21110 msgr2=0x559a92a225d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb537fe640 0 client.17751 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 shutdown_connections 2024-03-24T09:29:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 1 -- 172.21.15.116:0/3011169058 wait complete. 2024-03-24T09:29:16.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.780+0000 7feb7cf68740 20 client.17751 trim_cache size 0 max 16384 2024-03-24T09:29:16.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.805+0000 7feb7cf68740 1 Processor -- start 2024-03-24T09:29:16.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.805+0000 7feb7cf68740 1 -- start start 2024-03-24T09:29:16.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.805+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a2f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.805+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a3b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.805+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 0x559a92a2bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.805+0000 7feb7cf68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559a9290a430 con 0x559a92a2fee0 2024-03-24T09:29:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb7cf68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559a92a21a50 con 0x559a92a269e0 2024-03-24T09:29:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb7cf68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559a92a328a0 con 0x559a92a27300 2024-03-24T09:29:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 0x559a92a2bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 0x559a92a2bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38978/0 (socket says 172.21.15.116:38978) 2024-03-24T09:29:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 -- 172.21.15.116:0/1498409883 learned_addr learned my addr 172.21.15.116:0/1498409883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb73fff640 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a3b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a2f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 -- 172.21.15.116:0/1498409883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 msgr2=0x559a92a3b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a3b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 -- 172.21.15.116:0/1498409883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 msgr2=0x559a92a2f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a2f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 -- 172.21.15.116:0/1498409883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a9291fb20 con 0x559a92a2fee0 2024-03-24T09:29:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.806+0000 7feb737fe640 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 0x559a92a2bfb0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7feb6802c170 tx=0x7feb68004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7083d63bfcf05e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.807+0000 7feb59ffb640 1 -- 172.21.15.116:0/1498409883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68003a00 con 0x559a92a2fee0 2024-03-24T09:29:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.807+0000 7feb59ffb640 1 -- 172.21.15.116:0/1498409883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb68043480 con 0x559a92a2fee0 2024-03-24T09:29:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.807+0000 7feb59ffb640 1 -- 172.21.15.116:0/1498409883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb68004070 con 0x559a92a2fee0 2024-03-24T09:29:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 -- 172.21.15.116:0/1498409883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 msgr2=0x559a92a2bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 0x559a92a2bfb0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7feb6802c170 tx=0x7feb68004930 comp rx=0 tx=0).stop 2024-03-24T09:29:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 -- 172.21.15.116:0/1498409883 shutdown_connections 2024-03-24T09:29:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a2fee0 0x559a92a2bfb0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a27300 0x559a92a3b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1498409883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a2f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 -- 172.21.15.116:0/1498409883 >> 172.21.15.116:0/1498409883 conn(0x559a92a21110 msgr2=0x559a92a2b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 -- 172.21.15.116:0/1498409883 shutdown_connections 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 -- 172.21.15.116:0/1498409883 wait complete. 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 -- start start 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.808+0000 7feb7cf68740 1 Processor -- start 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a29b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 0x559a92a2a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a2a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559a92aa51f0 con 0x559a92a27300 2024-03-24T09:29:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559a92aa5330 con 0x559a92a269e0 2024-03-24T09:29:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559a92aa5470 con 0x559a92a2fee0 2024-03-24T09:29:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb7cf68740 10 client.0 _ll_register_callbacks cb 0x7feb64009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 0x559a92a2a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 0x559a92a2a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38992/0 (socket says 172.21.15.116:38992) 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb73fff640 1 -- 172.21.15.116:0/1388155838 learned_addr learned my addr 172.21.15.116:0/1388155838 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.809+0000 7feb737fe640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a2a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb73fff640 1 -- 172.21.15.116:0/1388155838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 msgr2=0x559a92a2a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb73fff640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a2a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb737fe640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a29b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb73fff640 1 -- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 msgr2=0x559a92a29b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb73fff640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a29b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb73fff640 1 -- 172.21.15.116:0/1388155838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a92983ba0 con 0x559a92a27300 2024-03-24T09:29:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb737fe640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a29b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:29:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb73fff640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 0x559a92a2a0c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x559a92a2c810 tx=0x7feb64006410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c9006b7b52e4b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb537fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:16.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb537fe640 1 -- 172.21.15.116:0/1388155838 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64005e80 con 0x559a92a27300 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb680044e0 con 0x559a92a27300 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.810+0000 7feb537fe640 1 -- 172.21.15.116:0/1388155838 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7feb6401a040 con 0x559a92a27300 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb537fe640 1 -- 172.21.15.116:0/1388155838 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7feb6401fcc0 con 0x559a92a27300 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 10 client.17757 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559a92945700 con 0x559a92a27300 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb537fe640 1 -- 172.21.15.116:0/1388155838 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7feb680044e0 con 0x559a92a27300 2024-03-24T09:29:16.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb537fe640 1 -- 172.21.15.116:0/1388155838 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7feb64025020 con 0x559a92a27300 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 10 client.17757 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 10 client.17757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 20 client.17757 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92aa6c50 0x559a92aa9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 4 client.17757 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.811+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559a92710890 con 0x559a92aa6c50 2024-03-24T09:29:16.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.812+0000 7feb737fe640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92aa6c50 0x559a92aa9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.812+0000 7feb737fe640 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92aa6c50 0x559a92aa9070 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22eaccc4bea8656a server_cookie=1cbe75c7c553c2f3 in_seq=0 out_seq=0 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.812+0000 7feb537fe640 10 client.17757 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.812+0000 7feb537fe640 1 -- 172.21.15.116:0/1388155838 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559a92710890 con 0x559a92aa6c50 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.812+0000 7feb537fe640 10 client.17757 handle_command_reply: tid=0 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.812+0000 7feb7cf68740 1 client.17757 shutdown 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 msgr2=0x559a92a2a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 0x559a92a2a0c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x559a92a2c810 tx=0x7feb64006410 comp rx=0 tx=0).stop 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 shutdown_connections 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559a92a2fee0 0x559a92a2a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559a92a27300 0x559a92a2a0c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559a92aa6c50 0x559a92aa9070 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 --2- 172.21.15.116:0/1388155838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559a92a269e0 0x559a92a29b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 >> 172.21.15.116:0/1388155838 conn(0x559a92a21110 msgr2=0x559a92a2cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb537fe640 0 client.17757 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 shutdown_connections 2024-03-24T09:29:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 1 -- 172.21.15.116:0/1388155838 wait complete. 2024-03-24T09:29:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.813+0000 7feb7cf68740 20 client.17757 trim_cache size 0 max 16384 2024-03-24T09:29:16.825 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:29:16.825 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:29:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:29:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "569b3015-9e1a-43a1-ac6e-2f918709dc59": { 2024-03-24T09:29:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:29:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "569b3015-9e1a-43a1-ac6e-2f918709dc59", 2024-03-24T09:29:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:29:16.826 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:29:16.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:29:16.827 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:29:16.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7feb5c0a22b0 msgr2=0x7feb5c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7feb5c0a22b0 0x7feb5c0a4770 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0cb020 tx=0x7feb6c021000 comp rx=0 tx=0).stop 2024-03-24T09:29:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 msgr2=0x7feb7419e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 0x7feb7419e8e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00b970 tx=0x7feb6c00bd70 comp rx=0 tx=0).stop 2024-03-24T09:29:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 shutdown_connections 2024-03-24T09:29:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7feb5c0a22b0 0x7feb5c0a4770 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb741076b0 0x7feb7419e8e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb74100570 0x7feb7419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 --2- 172.21.15.116:0/922463373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb740ffbb0 0x7feb7419a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:16.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 >> 172.21.15.116:0/922463373 conn(0x7feb74076b30 msgr2=0x7feb7410d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:16.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 shutdown_connections 2024-03-24T09:29:16.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:16.814+0000 7feb7a38a640 1 -- 172.21.15.116:0/922463373 wait complete. 2024-03-24T09:29:16.830 INFO:tasks.cephfs.filesystem:scrub status for tag:569b3015-9e1a-43a1-ac6e-2f918709dc59 - {'path': '/', 'tag': '569b3015-9e1a-43a1-ac6e-2f918709dc59', 'options': 'recursive,force'} 2024-03-24T09:29:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:17 smithi143 ceph-mon[35965]: pgmap v220: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:29:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:17 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:17.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:17 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:29:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:17 smithi117 ceph-mon[35788]: pgmap v220: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:29:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:17 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:17.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:17 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:29:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:17 smithi116 ceph-mon[32207]: pgmap v220: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:29:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:17 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:17 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:29:18.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:29:18.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:18.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:18.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:18.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:18.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:18.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:29:18.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:29:18.278 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:18 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:29:18.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:29:18.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:18.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:18.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:18.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:18.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:18.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:29:18.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:29:18.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:18 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:29:18.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:29:18.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:18.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:18.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:18.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:18.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:18.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:18.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:29:18.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:29:18.322 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:18 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:29:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:19 smithi143 ceph-mon[35965]: pgmap v221: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2024-03-24T09:29:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:19 smithi117 ceph-mon[35788]: pgmap v221: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2024-03-24T09:29:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:19 smithi116 ceph-mon[32207]: pgmap v221: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 221 B/s wr, 11 op/s 2024-03-24T09:29:21.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:21 smithi143 ceph-mon[35965]: pgmap v222: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:21 smithi117 ceph-mon[35788]: pgmap v222: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:21.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:21 smithi116 ceph-mon[32207]: pgmap v222: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:29:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:23 smithi143 ceph-mon[35965]: pgmap v223: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:29:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:23 smithi117 ceph-mon[35788]: pgmap v223: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:29:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:23 smithi116 ceph-mon[32207]: pgmap v223: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:29:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:25 smithi143 ceph-mon[35965]: pgmap v224: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:29:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:25 smithi117 ceph-mon[35788]: pgmap v224: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:29:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:25 smithi116 ceph-mon[32207]: pgmap v224: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:29:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:27 smithi143 ceph-mon[35965]: pgmap v225: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 89 B/s wr, 9 op/s 2024-03-24T09:29:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:27 smithi117 ceph-mon[35788]: pgmap v225: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 89 B/s wr, 9 op/s 2024-03-24T09:29:27.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:27 smithi116 ceph-mon[32207]: pgmap v225: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 89 B/s wr, 9 op/s 2024-03-24T09:29:28.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:28 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:29:28.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:28 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:29:28.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:28 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:29:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:29 smithi143 ceph-mon[35965]: pgmap v226: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:29:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:29 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:29:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:29 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:29:29.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:29 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:29:29.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:29 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:29:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:29 smithi117 ceph-mon[35788]: pgmap v226: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:29:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:29 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:29:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:29 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:29:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:29 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:29:29.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:29 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:29:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:29 smithi116 ceph-mon[32207]: pgmap v226: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:29:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:29 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:29:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:29 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:29:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:29 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:29:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:29 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:29:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:31 smithi117 ceph-mon[35788]: pgmap v227: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:31 smithi116 ceph-mon[32207]: pgmap v227: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:31.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:31 smithi143 ceph-mon[35965]: pgmap v227: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:33 smithi117 ceph-mon[35788]: pgmap v228: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:33 smithi116 ceph-mon[32207]: pgmap v228: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:33 smithi143 ceph-mon[35965]: pgmap v228: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:35 smithi117 ceph-mon[35788]: pgmap v229: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:35.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:35 smithi116 ceph-mon[32207]: pgmap v229: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:35 smithi143 ceph-mon[35965]: pgmap v229: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:37 smithi117 ceph-mon[35788]: pgmap v230: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:37 smithi116 ceph-mon[32207]: pgmap v230: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:37 smithi143 ceph-mon[35965]: pgmap v230: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:39 smithi117 ceph-mon[35788]: pgmap v231: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:39 smithi116 ceph-mon[32207]: pgmap v231: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:39 smithi143 ceph-mon[35965]: pgmap v231: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:41 smithi117 ceph-mon[35788]: pgmap v232: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:41.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:41 smithi116 ceph-mon[32207]: pgmap v232: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:41 smithi143 ceph-mon[35965]: pgmap v232: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:43 smithi117 ceph-mon[35788]: pgmap v233: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:43.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:43 smithi116 ceph-mon[32207]: pgmap v233: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:43 smithi143 ceph-mon[35965]: pgmap v233: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:45 smithi117 ceph-mon[35788]: pgmap v234: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:45 smithi116 ceph-mon[32207]: pgmap v234: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:45.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:45 smithi143 ceph-mon[35965]: pgmap v234: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:46.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:29:46.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:29:46.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 -- 172.21.15.116:0/4258384144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f01032a0 msgr2=0x7f98f010b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:46.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 --2- 172.21.15.116:0/4258384144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f01032a0 0x7f98f010b780 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f98e4009900 tx=0x7f98e402bcd0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 -- 172.21.15.116:0/4258384144 shutdown_connections 2024-03-24T09:29:46.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 --2- 172.21.15.116:0/4258384144 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98f010bd50 0x7f98f010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 --2- 172.21.15.116:0/4258384144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f01032a0 0x7f98f010b780 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 --2- 172.21.15.116:0/4258384144 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f0102980 0x7f98f0102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 -- 172.21.15.116:0/4258384144 >> 172.21.15.116:0/4258384144 conn(0x7f98f0077770 msgr2=0x7f98f0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:46.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 -- 172.21.15.116:0/4258384144 shutdown_connections 2024-03-24T09:29:46.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 -- 172.21.15.116:0/4258384144 wait complete. 2024-03-24T09:29:46.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 Processor -- start 2024-03-24T09:29:46.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.947+0000 7f98f6640640 1 -- start start 2024-03-24T09:29:46.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 0x7f98f019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f01032a0 0x7f98f019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98f010bd50 0x7f98f0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98f0198180 con 0x7f98f0102980 2024-03-24T09:29:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f98f01982f0 con 0x7f98f01032a0 2024-03-24T09:29:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f98f01984b0 con 0x7f98f010bd50 2024-03-24T09:29:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 0x7f98f019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 0x7f98f019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60994/0 (socket says 172.21.15.116:60994) 2024-03-24T09:29:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 -- 172.21.15.116:0/2292772981 learned_addr learned my addr 172.21.15.116:0/2292772981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:46.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f01032a0 0x7f98f019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98f010bd50 0x7f98f0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 -- 172.21.15.116:0/2292772981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98f010bd50 msgr2=0x7f98f0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98f010bd50 0x7f98f0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 -- 172.21.15.116:0/2292772981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f01032a0 msgr2=0x7f98f019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f01032a0 0x7f98f019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 -- 172.21.15.116:0/2292772981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e0009490 con 0x7f98f0102980 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f01032a0 0x7f98f019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98effff640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 0x7f98f019d930 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f98e000ea60 tx=0x7f98e000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98ed7fa640 1 -- 172.21.15.116:0/2292772981 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e000cde0 con 0x7f98f0102980 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98ed7fa640 1 -- 172.21.15.116:0/2292772981 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98e00044d0 con 0x7f98f0102980 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e4009560 con 0x7f98f0102980 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.948+0000 7f98ed7fa640 1 -- 172.21.15.116:0/2292772981 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e0010690 con 0x7f98f0102980 2024-03-24T09:29:46.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.949+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f01a47f0 con 0x7f98f0102980 2024-03-24T09:29:46.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98ed7fa640 1 -- 172.21.15.116:0/2292772981 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f98e0003e40 con 0x7f98f0102980 2024-03-24T09:29:46.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 Processor -- start 2024-03-24T09:29:46.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 -- start start 2024-03-24T09:29:46.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 0x564f601205b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f60120af0 0x564f60129190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601296d0 0x564f6012bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f60044f30 con 0x564f60120af0 2024-03-24T09:29:46.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f6008e810 con 0x564f601296d0 2024-03-24T09:29:46.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f6008bd40 con 0x564f601201d0 2024-03-24T09:29:46.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98ed7fa640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f98d80a2230 0x7f98d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f60120af0 0x564f60129190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f4bdd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 0x564f601205b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f4bdd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 0x564f601205b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:34148/0 (socket says 172.21.15.116:34148) 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f60120af0 0x564f60129190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32774/0 (socket says 172.21.15.116:32774) 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f4bdd640 1 -- 172.21.15.116:0/2824696685 learned_addr learned my addr 172.21.15.116:0/2824696685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601296d0 0x564f6012bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.951+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f98d80a2230 0x7f98d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98ed7fa640 1 -- 172.21.15.116:0/2292772981 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98e0014070 con 0x7f98f0102980 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 -- 172.21.15.116:0/2824696685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601296d0 msgr2=0x564f6012bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601296d0 0x564f6012bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 -- 172.21.15.116:0/2824696685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f60120af0 msgr2=0x564f60129190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f60120af0 0x564f60129190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 -- 172.21.15.116:0/2824696685 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f60048ea0 con 0x564f601201d0 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 0x564f601205b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98e8011da0 tx=0x7f98e800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7969a64a28e204c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:46.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f98d80a2230 0x7f98d80a46f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98e800fc90 tx=0x7f98e801e000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98d67fc640 1 -- 172.21.15.116:0/2824696685 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e801dbf0 con 0x564f601201d0 2024-03-24T09:29:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98d67fc640 1 -- 172.21.15.116:0/2824696685 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98e8009de0 con 0x564f601201d0 2024-03-24T09:29:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.952+0000 7f98d67fc640 1 -- 172.21.15.116:0/2824696685 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e80042b0 con 0x564f601201d0 2024-03-24T09:29:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 -- 172.21.15.116:0/2824696685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 msgr2=0x564f601205b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 0x564f601205b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98e8011da0 tx=0x7f98e800f790 comp rx=0 tx=0).stop 2024-03-24T09:29:46.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 -- 172.21.15.116:0/2824696685 shutdown_connections 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601296d0 0x564f6012bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f60120af0 0x564f60129190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 --2- 172.21.15.116:0/2824696685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601201d0 0x564f601205b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 -- 172.21.15.116:0/2824696685 >> 172.21.15.116:0/2824696685 conn(0x564f6011a900 msgr2=0x564f6011ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 -- 172.21.15.116:0/2824696685 shutdown_connections 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 -- 172.21.15.116:0/2824696685 wait complete. 2024-03-24T09:29:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 -- start start 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.953+0000 7f98f921e740 1 Processor -- start 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60126800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60126d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f60127280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f60087520 con 0x564f601296d0 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f5fd5a430 con 0x564f601201d0 2024-03-24T09:29:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f5fd16e00 con 0x564f60120af0 2024-03-24T09:29:46.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 10 client.0 _ll_register_callbacks cb 0x564f60025d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:46.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f4bdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f60127280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60126800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f4bdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f60127280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32782/0 (socket says 172.21.15.116:32782) 2024-03-24T09:29:46.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f4bdd640 1 -- 172.21.15.116:0/465175161 learned_addr learned my addr 172.21.15.116:0/465175161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:46.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60126d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 -- 172.21.15.116:0/465175161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 msgr2=0x564f60126d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60126d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 -- 172.21.15.116:0/465175161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 msgr2=0x564f60127280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f60127280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 -- 172.21.15.116:0/465175161 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e8019040 con 0x564f601201d0 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f60127280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98effff640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60126800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f98e00cbc70 tx=0x7f98e009a870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17983739b642fe9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:29:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98cf7fe640 1 -- 172.21.15.116:0/465175161 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e009f060 con 0x564f601201d0 2024-03-24T09:29:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f5fd867e0 con 0x564f601201d0 2024-03-24T09:29:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.954+0000 7f98cf7fe640 1 -- 172.21.15.116:0/465175161 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98e00a36b0 con 0x564f601201d0 2024-03-24T09:29:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98cf7fe640 1 -- 172.21.15.116:0/465175161 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f98e00cbef0 con 0x564f601201d0 2024-03-24T09:29:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 10 client.27584 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564f60085ba0 con 0x564f601201d0 2024-03-24T09:29:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98cf7fe640 1 -- 172.21.15.116:0/465175161 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98e00b7020 con 0x564f601201d0 2024-03-24T09:29:46.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98cf7fe640 1 -- 172.21.15.116:0/465175161 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f98e00cebe0 con 0x564f601201d0 2024-03-24T09:29:46.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 10 client.27584 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:46.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 10 client.27584 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:46.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 20 client.27584 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f60139b80 0x564f6013bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 4 client.27584 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f5fdc3720 con 0x564f60139b80 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.955+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f60139b80 0x564f6013bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98cf7fe640 10 client.27584 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f60139b80 0x564f6013bf70 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b74764d0833f712 server_cookie=9b9afbfd49c1faef in_seq=0 out_seq=0 2024-03-24T09:29:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98cf7fe640 1 -- 172.21.15.116:0/465175161 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564f5fdc3720 con 0x564f60139b80 2024-03-24T09:29:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98cf7fe640 10 client.27584 handle_command_reply: tid=0 2024-03-24T09:29:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 client.27584 shutdown 2024-03-24T09:29:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 msgr2=0x564f60126800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60126800 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f98e00cbc70 tx=0x7f98e009a870 comp rx=0 tx=0).stop 2024-03-24T09:29:46.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 shutdown_connections 2024-03-24T09:29:46.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f60127280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60126d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f60139b80 0x564f6013bf70 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 --2- 172.21.15.116:0/465175161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60126800 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98cf7fe640 0 client.27584 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:46.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 >> 172.21.15.116:0/465175161 conn(0x564f6011a900 msgr2=0x564f6011c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 shutdown_connections 2024-03-24T09:29:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.956+0000 7f98f921e740 1 -- 172.21.15.116:0/465175161 wait complete. 2024-03-24T09:29:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.957+0000 7f98f921e740 20 client.27584 trim_cache size 0 max 16384 2024-03-24T09:29:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 Processor -- start 2024-03-24T09:29:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 -- start start 2024-03-24T09:29:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60134f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60141810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f601270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f6000c430 con 0x564f601296d0 2024-03-24T09:29:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f60135440 con 0x564f601201d0 2024-03-24T09:29:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f6011bb30 con 0x564f60120af0 2024-03-24T09:29:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f601270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98effff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60134f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98ef7fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f601270a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32786/0 (socket says 172.21.15.116:32786) 2024-03-24T09:29:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 -- 172.21.15.116:0/3396593867 learned_addr learned my addr 172.21.15.116:0/3396593867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 -- 172.21.15.116:0/3396593867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 msgr2=0x564f60141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 -- 172.21.15.116:0/3396593867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 msgr2=0x564f60134f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:46.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60134f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 -- 172.21.15.116:0/3396593867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f60021b20 con 0x564f601296d0 2024-03-24T09:29:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98effff640 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60134f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.974+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f601270a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009d40 tx=0x7f98e8009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0603082b4a495ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98d5ffb640 1 -- 172.21.15.116:0/3396593867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e8011c50 con 0x564f601296d0 2024-03-24T09:29:46.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98d5ffb640 1 -- 172.21.15.116:0/3396593867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98e8003e30 con 0x564f601296d0 2024-03-24T09:29:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98d5ffb640 1 -- 172.21.15.116:0/3396593867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e8011390 con 0x564f601296d0 2024-03-24T09:29:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 -- 172.21.15.116:0/3396593867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 msgr2=0x564f601270a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f601270a0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009d40 tx=0x7f98e8009480 comp rx=0 tx=0).stop 2024-03-24T09:29:46.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 -- 172.21.15.116:0/3396593867 shutdown_connections 2024-03-24T09:29:46.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601296d0 0x564f601270a0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f60120af0 0x564f60141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 --2- 172.21.15.116:0/3396593867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f601201d0 0x564f60134f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:46.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 -- 172.21.15.116:0/3396593867 >> 172.21.15.116:0/3396593867 conn(0x564f6011a900 msgr2=0x564f60119b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:46.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 -- 172.21.15.116:0/3396593867 shutdown_connections 2024-03-24T09:29:46.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 -- 172.21.15.116:0/3396593867 wait complete. 2024-03-24T09:29:46.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.975+0000 7f98f921e740 1 -- start start 2024-03-24T09:29:46.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 Processor -- start 2024-03-24T09:29:46.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 0x564f60131120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f60120af0 0x564f60131660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601296d0 0x564f6012d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:46.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564f6012dc50 con 0x564f601201d0 2024-03-24T09:29:46.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564f6012ddc0 con 0x564f60120af0 2024-03-24T09:29:46.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564f6012df80 con 0x564f601296d0 2024-03-24T09:29:46.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f921e740 10 client.0 _ll_register_callbacks cb 0x7f98e000cf60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 0x564f60131120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 0x564f60131120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32800/0 (socket says 172.21.15.116:32800) 2024-03-24T09:29:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 -- 172.21.15.116:0/4093380873 learned_addr learned my addr 172.21.15.116:0/4093380873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f60120af0 0x564f60131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98f4bdd640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601296d0 0x564f6012d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 -- 172.21.15.116:0/4093380873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601296d0 msgr2=0x564f6012d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601296d0 0x564f6012d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 -- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f60120af0 msgr2=0x564f60131660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f60120af0 0x564f60131660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 -- 172.21.15.116:0/4093380873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f60085ba0 con 0x564f601201d0 2024-03-24T09:29:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98effff640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 0x564f60131120 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f98e009ac10 tx=0x7f98e00cedc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbef625d12b0656 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.976+0000 7f98cf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98cf7fe640 1 -- 172.21.15.116:0/4093380873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e00a36b0 con 0x564f601201d0 2024-03-24T09:29:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98cf7fe640 1 -- 172.21.15.116:0/4093380873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f98e00b8030 con 0x564f601201d0 2024-03-24T09:29:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98e8009de0 con 0x564f601201d0 2024-03-24T09:29:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98cf7fe640 1 -- 172.21.15.116:0/4093380873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f98e009f060 con 0x564f601201d0 2024-03-24T09:29:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 10 client.17775 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7f98e802a040 con 0x564f601201d0 2024-03-24T09:29:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98cf7fe640 1 -- 172.21.15.116:0/4093380873 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f98e8009de0 con 0x564f601201d0 2024-03-24T09:29:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98cf7fe640 1 -- 172.21.15.116:0/4093380873 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f98e00a3220 con 0x564f601201d0 2024-03-24T09:29:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 10 client.17775 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 20 client.17775 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f601a0d40 0x564f601a3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 4 client.17775 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:29:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564f5fe12890 con 0x564f601a0d40 2024-03-24T09:29:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.977+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f601a0d40 0x564f601a3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98cf7fe640 10 client.17775 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98ef7fe640 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f601a0d40 0x564f601a3130 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b0a9907dd610a5b server_cookie=c1facf88eab62e6b in_seq=0 out_seq=0 2024-03-24T09:29:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98cf7fe640 1 -- 172.21.15.116:0/4093380873 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564f5fe12890 con 0x564f601a0d40 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98cf7fe640 10 client.17775 handle_command_reply: tid=0 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 client.17775 shutdown 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 msgr2=0x564f60131120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 0x564f60131120 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f98e009ac10 tx=0x7f98e00cedc0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 shutdown_connections 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564f601296d0 0x564f6012d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564f601a0d40 0x564f601a3130 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564f60120af0 0x564f60131660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 --2- 172.21.15.116:0/4093380873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564f601201d0 0x564f60131120 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98cf7fe640 0 client.17775 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 >> 172.21.15.116:0/4093380873 conn(0x564f6011a900 msgr2=0x564f6011bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 shutdown_connections 2024-03-24T09:29:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 1 -- 172.21.15.116:0/4093380873 wait complete. 2024-03-24T09:29:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.978+0000 7f98f921e740 20 client.17775 trim_cache size 0 max 16384 2024-03-24T09:29:47.008 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:29:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:29:47.008 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:29:47.008 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:29:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f98d80a2230 msgr2=0x7f98d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f98d80a2230 0x7f98d80a46f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98e800fc90 tx=0x7f98e801e000 comp rx=0 tx=0).stop 2024-03-24T09:29:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 msgr2=0x7f98f019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 0x7f98f019d930 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f98e000ea60 tx=0x7f98e000ef20 comp rx=0 tx=0).stop 2024-03-24T09:29:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 shutdown_connections 2024-03-24T09:29:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f98d80a2230 0x7f98d80a46f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f98f010bd50 0x7f98f0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98f01032a0 0x7f98f019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 --2- 172.21.15.116:0/2292772981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98f0102980 0x7f98f019d930 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 >> 172.21.15.116:0/2292772981 conn(0x7f98f0077770 msgr2=0x7f98f00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 shutdown_connections 2024-03-24T09:29:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:46.979+0000 7f98f6640640 1 -- 172.21.15.116:0/2292772981 wait complete. 2024-03-24T09:29:47.011 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:29:47.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:29:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 -- 172.21.15.116:0/384558860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881016a0 msgr2=0x7f2088101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 --2- 172.21.15.116:0/384558860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881016a0 0x7f2088101b20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f2078009930 tx=0x7f207802bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 -- 172.21.15.116:0/384558860 shutdown_connections 2024-03-24T09:29:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 --2- 172.21.15.116:0/384558860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2088102060 0x7f208810e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 --2- 172.21.15.116:0/384558860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20881016a0 0x7f2088101b20 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 --2- 172.21.15.116:0/384558860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20881076b0 0x7f2088107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 -- 172.21.15.116:0/384558860 >> 172.21.15.116:0/384558860 conn(0x7f20880fd1b0 msgr2=0x7f20880ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 -- 172.21.15.116:0/384558860 shutdown_connections 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 -- 172.21.15.116:0/384558860 wait complete. 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 Processor -- start 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.114+0000 7f208f2d4640 1 -- start start 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20881016a0 0x7f208819d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 0x7f208819de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20881076b0 0x7f2088197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2088198140 con 0x7f2088102060 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f20881982b0 con 0x7f20881016a0 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2088198470 con 0x7f20881076b0 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 0x7f208819de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 0x7f208819de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32822/0 (socket says 172.21.15.116:32822) 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 -- 172.21.15.116:0/3345026521 learned_addr learned my addr 172.21.15.116:0/3345026521 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208d070640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20881016a0 0x7f208819d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208d871640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20881076b0 0x7f2088197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 -- 172.21.15.116:0/3345026521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20881076b0 msgr2=0x7f2088197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20881076b0 0x7f2088197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 -- 172.21.15.116:0/3345026521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20881016a0 msgr2=0x7f208819d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20881016a0 0x7f208819d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 -- 172.21.15.116:0/3345026521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2078009560 con 0x7f2088102060 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208c86f640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 0x7f208819de80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2078009230 tx=0x7f2078005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078004350 con 0x7f2088102060 2024-03-24T09:29:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20881986f0 con 0x7f2088102060 2024-03-24T09:29:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.115+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f207802de90 con 0x7f2088102060 2024-03-24T09:29:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.116+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078005220 con 0x7f2088102060 2024-03-24T09:29:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.116+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20881a49e0 con 0x7f2088102060 2024-03-24T09:29:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.116+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2088068950 con 0x7f2088102060 2024-03-24T09:29:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.117+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2078003c30 con 0x7f2088102060 2024-03-24T09:29:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.118+0000 7f207e7fc640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f20740a2230 0x7f20740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.118+0000 7f208d070640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f20740a2230 0x7f20740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.118+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f20780f0a90 con 0x7f2088102060 2024-03-24T09:29:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.119+0000 7f208d070640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f20740a2230 0x7f20740a46f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2084009b50 tx=0x7f2084005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.122+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f20780b93c0 con 0x7f2088102060 2024-03-24T09:29:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:47 smithi117 ceph-mon[35788]: pgmap v235: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:47.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:47 smithi116 ceph-mon[32207]: pgmap v235: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:47.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:47 smithi143 ceph-mon[35965]: pgmap v235: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:47.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.334+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2088198fe0 con 0x7f2088102060 2024-03-24T09:29:47.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.335+0000 7f207e7fc640 1 -- 172.21.15.116:0/3345026521 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f20780d4ed0 con 0x7f2088102060 2024-03-24T09:29:47.336 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:29:47.336 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:29:47.336 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:29:47.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.336+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f20740a2230 msgr2=0x7f20740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.336+0000 7f208f2d4640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f20740a2230 0x7f20740a46f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2084009b50 tx=0x7f2084005f90 comp rx=0 tx=0).stop 2024-03-24T09:29:47.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.336+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 msgr2=0x7f208819de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.336+0000 7f208f2d4640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 0x7f208819de80 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f2078009230 tx=0x7f2078005dd0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 shutdown_connections 2024-03-24T09:29:47.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f20740a2230 0x7f20740a46f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f20881076b0 0x7f2088197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2088102060 0x7f208819de80 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 --2- 172.21.15.116:0/3345026521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20881016a0 0x7f208819d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 >> 172.21.15.116:0/3345026521 conn(0x7f20880fd1b0 msgr2=0x7f20880fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 shutdown_connections 2024-03-24T09:29:47.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.337+0000 7f208f2d4640 1 -- 172.21.15.116:0/3345026521 wait complete. 2024-03-24T09:29:47.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:29:47.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 -- 172.21.15.116:0/2610581183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c100bc0 msgr2=0x7fe82c100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 --2- 172.21.15.116:0/2610581183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c100bc0 0x7fe82c100fa0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe828009900 tx=0x7fe82802bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 -- 172.21.15.116:0/2610581183 shutdown_connections 2024-03-24T09:29:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 --2- 172.21.15.116:0/2610581183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe82c105b50 0x7fe82c10d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 --2- 172.21.15.116:0/2610581183 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c105190 0x7fe82c105610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 --2- 172.21.15.116:0/2610581183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c100bc0 0x7fe82c100fa0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 -- 172.21.15.116:0/2610581183 >> 172.21.15.116:0/2610581183 conn(0x7fe82c0773d0 msgr2=0x7fe82c0777e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 -- 172.21.15.116:0/2610581183 shutdown_connections 2024-03-24T09:29:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.452+0000 7fe834607640 1 -- 172.21.15.116:0/2610581183 wait complete. 2024-03-24T09:29:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 Processor -- start 2024-03-24T09:29:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 -- start start 2024-03-24T09:29:47.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe82c100bc0 0x7fe82c19d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c105190 0x7fe82c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 0x7fe82c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c198060 con 0x7fe82c105b50 2024-03-24T09:29:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c1981d0 con 0x7fe82c100bc0 2024-03-24T09:29:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe834607640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c198390 con 0x7fe82c105190 2024-03-24T09:29:47.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 0x7fe82c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 0x7fe82c197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32832/0 (socket says 172.21.15.116:32832) 2024-03-24T09:29:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 -- 172.21.15.116:0/3929341320 learned_addr learned my addr 172.21.15.116:0/3929341320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe8323a3640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe82c100bc0 0x7fe82c19d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c105190 0x7fe82c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 -- 172.21.15.116:0/3929341320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c105190 msgr2=0x7fe82c19dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c105190 0x7fe82c19dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 -- 172.21.15.116:0/3929341320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe82c100bc0 msgr2=0x7fe82c19d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe82c100bc0 0x7fe82c19d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 -- 172.21.15.116:0/3929341320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe828009560 con 0x7fe82c105b50 2024-03-24T09:29:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.453+0000 7fe832ba4640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 0x7fe82c197a10 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe82400eb10 tx=0x7fe82400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.454+0000 7fe81f7fe640 1 -- 172.21.15.116:0/3929341320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824004b20 con 0x7fe82c105b50 2024-03-24T09:29:47.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.454+0000 7fe81f7fe640 1 -- 172.21.15.116:0/3929341320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe824004c80 con 0x7fe82c105b50 2024-03-24T09:29:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.454+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82c198670 con 0x7fe82c105b50 2024-03-24T09:29:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.454+0000 7fe81f7fe640 1 -- 172.21.15.116:0/3929341320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824010690 con 0x7fe82c105b50 2024-03-24T09:29:47.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.454+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe82c1a4900 con 0x7fe82c105b50 2024-03-24T09:29:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe81f7fe640 1 -- 172.21.15.116:0/3929341320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe824005020 con 0x7fe82c105b50 2024-03-24T09:29:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 Processor -- start 2024-03-24T09:29:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 -- start start 2024-03-24T09:29:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb6347d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb634d10 0x55b0bb63d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb5a2840 con 0x55b0bb6343f0 2024-03-24T09:29:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb59fd70 con 0x55b0bb63d8f0 2024-03-24T09:29:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8371e8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb59b550 con 0x55b0bb634d10 2024-03-24T09:29:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe831ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb6347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe831ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb6347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32838/0 (socket says 172.21.15.116:32838) 2024-03-24T09:29:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe831ba2640 1 -- 172.21.15.116:0/3563561540 learned_addr learned my addr 172.21.15.116:0/3563561540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe8323a3640 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb634d10 0x55b0bb63d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.457+0000 7fe81f7fe640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8180a2230 0x7fe8180a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe832ba4640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8180a2230 0x7fe8180a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe831ba2640 1 -- 172.21.15.116:0/3563561540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb634d10 msgr2=0x55b0bb63d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb634d10 0x55b0bb63d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe831ba2640 1 -- 172.21.15.116:0/3563561540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 msgr2=0x55b0bb63fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe831ba2640 1 -- 172.21.15.116:0/3563561540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0bb530de0 con 0x55b0bb6343f0 2024-03-24T09:29:47.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe8323a3640 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb634d10 0x55b0bb63d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe81f7fe640 1 -- 172.21.15.116:0/3929341320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe824014070 con 0x7fe82c105b50 2024-03-24T09:29:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb6347d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe8200137c0 tx=0x7fe820009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b64661a2b6905b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe81cbfe640 1 -- 172.21.15.116:0/3563561540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820013d40 con 0x55b0bb6343f0 2024-03-24T09:29:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe832ba4640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8180a2230 0x7fe8180a46f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe82409ad40 tx=0x7fe8240ca040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe81cbfe640 1 -- 172.21.15.116:0/3563561540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8200044e0 con 0x55b0bb6343f0 2024-03-24T09:29:47.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.458+0000 7fe81cbfe640 1 -- 172.21.15.116:0/3563561540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820003850 con 0x55b0bb6343f0 2024-03-24T09:29:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- 172.21.15.116:0/3563561540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 msgr2=0x55b0bb6347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb6347d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe8200137c0 tx=0x7fe820009780 comp rx=0 tx=0).stop 2024-03-24T09:29:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- 172.21.15.116:0/3563561540 shutdown_connections 2024-03-24T09:29:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb634d10 0x55b0bb63d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3563561540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb6347d0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- 172.21.15.116:0/3563561540 >> 172.21.15.116:0/3563561540 conn(0x55b0bb62eb20 msgr2=0x55b0bb62ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- 172.21.15.116:0/3563561540 shutdown_connections 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- 172.21.15.116:0/3563561540 wait complete. 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- start start 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 Processor -- start 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb63a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb634d10 0x55b0bb63aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb26e430 con 0x55b0bb6343f0 2024-03-24T09:29:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb22ae00 con 0x55b0bb634d10 2024-03-24T09:29:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb520590 con 0x55b0bb63d8f0 2024-03-24T09:29:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.459+0000 7fe8371e8740 10 client.0 _ll_register_callbacks cb 0x55b0bb58ffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb63a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe8323a3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb634d10 0x55b0bb63aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb63a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32848/0 (socket says 172.21.15.116:32848) 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 -- 172.21.15.116:0/1178294666 learned_addr learned my addr 172.21.15.116:0/1178294666 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 -- 172.21.15.116:0/1178294666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 msgr2=0x55b0bb63b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 -- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb634d10 msgr2=0x55b0bb63aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb634d10 0x55b0bb63aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 -- 172.21.15.116:0/1178294666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe82001b040 con 0x55b0bb6343f0 2024-03-24T09:29:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe831ba2640 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb63a9b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x55b0bb633da0 tx=0x7fe820011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0dfd6f22bb4ba92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe815ffb640 1 -- 172.21.15.116:0/1178294666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820005390 con 0x55b0bb6343f0 2024-03-24T09:29:47.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe815ffb640 1 -- 172.21.15.116:0/1178294666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe8200054f0 con 0x55b0bb6343f0 2024-03-24T09:29:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b0bb29a7e0 con 0x55b0bb6343f0 2024-03-24T09:29:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe815ffb640 1 -- 172.21.15.116:0/1178294666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe8200046c0 con 0x55b0bb6343f0 2024-03-24T09:29:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe8371e8740 10 client.17793 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b0bb599bd0 con 0x55b0bb6343f0 2024-03-24T09:29:47.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.460+0000 7fe815ffb640 1 -- 172.21.15.116:0/1178294666 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b0bb29a7e0 con 0x55b0bb6343f0 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe815ffb640 1 -- 172.21.15.116:0/1178294666 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe8200205c0 con 0x55b0bb6343f0 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8371e8740 10 client.17793 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8371e8740 10 client.17793 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8371e8740 20 client.17793 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb64d8e0 0x55b0bb64fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8371e8740 4 client.17793 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:47.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0bb2d7720 con 0x55b0bb64d8e0 2024-03-24T09:29:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8323a3640 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb64d8e0 0x55b0bb64fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe8323a3640 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb64d8e0 0x55b0bb64fcd0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e7f2c9a6c9623daf server_cookie=d9aa766125de517 in_seq=0 out_seq=0 2024-03-24T09:29:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.461+0000 7fe815ffb640 10 client.17793 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe815ffb640 1 -- 172.21.15.116:0/1178294666 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0bb2d7720 con 0x55b0bb64d8e0 2024-03-24T09:29:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe815ffb640 10 client.17793 handle_command_reply: tid=0 2024-03-24T09:29:47.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 client.17793 shutdown 2024-03-24T09:29:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 msgr2=0x55b0bb63a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb63a9b0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x55b0bb633da0 tx=0x7fe820011d70 comp rx=0 tx=0).stop 2024-03-24T09:29:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 shutdown_connections 2024-03-24T09:29:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb63b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb634d10 0x55b0bb63aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb64d8e0 0x55b0bb64fcd0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1178294666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb6343f0 0x55b0bb63a9b0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe815ffb640 0 client.17793 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 >> 172.21.15.116:0/1178294666 conn(0x55b0bb62eb20 msgr2=0x55b0bb630870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 shutdown_connections 2024-03-24T09:29:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 1 -- 172.21.15.116:0/1178294666 wait complete. 2024-03-24T09:29:47.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.462+0000 7fe8371e8740 20 client.17793 trim_cache size 0 max 16384 2024-03-24T09:29:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 Processor -- start 2024-03-24T09:29:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 -- start start 2024-03-24T09:29:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb6343f0 0x55b0bb63d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb649180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb655590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb63b980 con 0x55b0bb634d10 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb6334e0 con 0x55b0bb6343f0 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.475+0000 7fe8371e8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb64a500 con 0x55b0bb63d8f0 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb649180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe831ba2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb6343f0 0x55b0bb63d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb649180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32850/0 (socket says 172.21.15.116:32850) 2024-03-24T09:29:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 -- 172.21.15.116:0/1023926864 learned_addr learned my addr 172.21.15.116:0/1023926864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe831ba2640 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb655590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 -- 172.21.15.116:0/1023926864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 msgr2=0x55b0bb655590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb655590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 -- 172.21.15.116:0/1023926864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb6343f0 msgr2=0x55b0bb63d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb6343f0 0x55b0bb63d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 -- 172.21.15.116:0/1023926864 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0bb535c80 con 0x55b0bb634d10 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe831ba2640 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb655590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe8323a3640 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb649180 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe828009230 tx=0x7fe828009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23b8e4dff0bd08f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe817fff640 1 -- 172.21.15.116:0/1023926864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828004a00 con 0x55b0bb634d10 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe817fff640 1 -- 172.21.15.116:0/1023926864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe82802c7a0 con 0x55b0bb634d10 2024-03-24T09:29:47.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.476+0000 7fe817fff640 1 -- 172.21.15.116:0/1023926864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82804a330 con 0x55b0bb634d10 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- 172.21.15.116:0/1023926864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 msgr2=0x55b0bb649180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb649180 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe828009230 tx=0x7fe828009260 comp rx=0 tx=0).stop 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- 172.21.15.116:0/1023926864 shutdown_connections 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb63d8f0 0x55b0bb655590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb649180 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- 172.21.15.116:0/1023926864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb6343f0 0x55b0bb63d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- 172.21.15.116:0/1023926864 >> 172.21.15.116:0/1023926864 conn(0x55b0bb62eb20 msgr2=0x55b0bb62d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- 172.21.15.116:0/1023926864 shutdown_connections 2024-03-24T09:29:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- 172.21.15.116:0/1023926864 wait complete. 2024-03-24T09:29:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- start start 2024-03-24T09:29:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 Processor -- start 2024-03-24T09:29:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb6343f0 0x55b0bb641730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb641c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb6b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb642370 con 0x55b0bb634d10 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb6b3100 con 0x55b0bb63d8f0 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b0bb6b3270 con 0x55b0bb6343f0 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8371e8740 10 client.0 _ll_register_callbacks cb 0x7fe820009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8323a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb641c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8323a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb641c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32858/0 (socket says 172.21.15.116:32858) 2024-03-24T09:29:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe8323a3640 1 -- 172.21.15.116:0/3576014310 learned_addr learned my addr 172.21.15.116:0/3576014310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb6343f0 0x55b0bb641730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.477+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb6b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8323a3640 1 -- 172.21.15.116:0/3576014310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb6343f0 msgr2=0x55b0bb641730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8323a3640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb6343f0 0x55b0bb641730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8323a3640 1 -- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 msgr2=0x55b0bb6b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8323a3640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb6b2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8323a3640 1 -- 172.21.15.116:0/3576014310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe828030070 con 0x55b0bb634d10 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb6b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8323a3640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb641c70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe82802bc70 tx=0x7fe82802c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c371076c557bc22e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe815ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe815ffb640 1 -- 172.21.15.116:0/3576014310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82805b070 con 0x55b0bb634d10 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe815ffb640 1 -- 172.21.15.116:0/3576014310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe828046040 con 0x55b0bb634d10 2024-03-24T09:29:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8200044e0 con 0x55b0bb634d10 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe815ffb640 1 -- 172.21.15.116:0/3576014310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe82802cb20 con 0x55b0bb634d10 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 10 client.17799 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b0bb55b920 con 0x55b0bb634d10 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe815ffb640 1 -- 172.21.15.116:0/3576014310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe8200044e0 con 0x55b0bb634d10 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe815ffb640 1 -- 172.21.15.116:0/3576014310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe82805e020 con 0x55b0bb634d10 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 10 client.17799 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 10 client.17799 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 20 client.17799 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb6b4750 0x55b0bb6b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 4 client.17799 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.478+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b0bb326890 con 0x55b0bb6b4750 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb6b4750 0x55b0bb6b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe831ba2640 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb6b4750 0x55b0bb6b6b70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab4db10a46110e7 server_cookie=d51dc6e2d76511b6 in_seq=0 out_seq=0 2024-03-24T09:29:47.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe815ffb640 10 client.17799 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe815ffb640 1 -- 172.21.15.116:0/3576014310 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b0bb326890 con 0x55b0bb6b4750 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe815ffb640 10 client.17799 handle_command_reply: tid=0 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 client.17799 shutdown 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 msgr2=0x55b0bb641c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb641c70 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe82802bc70 tx=0x7fe82802c700 comp rx=0 tx=0).stop 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 shutdown_connections 2024-03-24T09:29:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0bb63d8f0 0x55b0bb6b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55b0bb6b4750 0x55b0bb6b6b70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0bb634d10 0x55b0bb641c70 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 --2- 172.21.15.116:0/3576014310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0bb6343f0 0x55b0bb641730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe815ffb640 0 client.17799 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 >> 172.21.15.116:0/3576014310 conn(0x55b0bb62eb20 msgr2=0x55b0bb63b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 shutdown_connections 2024-03-24T09:29:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 1 -- 172.21.15.116:0/3576014310 wait complete. 2024-03-24T09:29:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.479+0000 7fe8371e8740 20 client.17799 trim_cache size 0 max 16384 2024-03-24T09:29:47.504 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:47.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8180a2230 msgr2=0x7fe8180a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8180a2230 0x7fe8180a46f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe82409ad40 tx=0x7fe8240ca040 comp rx=0 tx=0).stop 2024-03-24T09:29:47.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 msgr2=0x7fe82c197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 0x7fe82c197a10 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe82400eb10 tx=0x7fe82400c2d0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 shutdown_connections 2024-03-24T09:29:47.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c105b50 0x7fe82c197a10 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe8180a2230 0x7fe8180a46f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe82c105190 0x7fe82c19dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 --2- 172.21.15.116:0/3929341320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe82c100bc0 0x7fe82c19d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 >> 172.21.15.116:0/3929341320 conn(0x7fe82c0773d0 msgr2=0x7fe82c0fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 shutdown_connections 2024-03-24T09:29:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.480+0000 7fe834607640 1 -- 172.21.15.116:0/3929341320 wait complete. 2024-03-24T09:29:47.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:29:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 -- 172.21.15.116:0/1036760782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f0100f80 msgr2=0x7f55f010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 --2- 172.21.15.116:0/1036760782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f0100f80 0x7f55f010d440 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009ca0 tx=0x7f55e802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 -- 172.21.15.116:0/1036760782 shutdown_connections 2024-03-24T09:29:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 --2- 172.21.15.116:0/1036760782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f0100f80 0x7f55f010d440 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 --2- 172.21.15.116:0/1036760782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f01005c0 0x7f55f0100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 --2- 172.21.15.116:0/1036760782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f0106680 0x7f55f0106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 -- 172.21.15.116:0/1036760782 >> 172.21.15.116:0/1036760782 conn(0x7f55f00776e0 msgr2=0x7f55f0077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 -- 172.21.15.116:0/1036760782 shutdown_connections 2024-03-24T09:29:47.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.611+0000 7f55f4aed640 1 -- 172.21.15.116:0/1036760782 wait complete. 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 Processor -- start 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 -- start start 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 0x7f55f019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f0100f80 0x7f55f019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f0106680 0x7f55f01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55f0198030 con 0x7f55f01005c0 2024-03-24T09:29:47.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55f01981a0 con 0x7f55f0106680 2024-03-24T09:29:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55f4aed640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f55f0198360 con 0x7f55f0100f80 2024-03-24T09:29:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 0x7f55f019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 0x7f55f019d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32874/0 (socket says 172.21.15.116:32874) 2024-03-24T09:29:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 -- 172.21.15.116:0/881812764 learned_addr learned my addr 172.21.15.116:0/881812764 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55eed9d640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f0106680 0x7f55f01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55edd9b640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f0100f80 0x7f55f019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 -- 172.21.15.116:0/881812764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f0100f80 msgr2=0x7f55f019dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f0100f80 0x7f55f019dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 -- 172.21.15.116:0/881812764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f0106680 msgr2=0x7f55f01979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f0106680 0x7f55f01979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.612+0000 7f55ee59c640 1 -- 172.21.15.116:0/881812764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e8009900 con 0x7f55f01005c0 2024-03-24T09:29:47.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.613+0000 7f55ee59c640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 0x7f55f019d7e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f55d800e9f0 tx=0x7f55d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.613+0000 7f55e77fe640 1 -- 172.21.15.116:0/881812764 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d8009700 con 0x7f55f01005c0 2024-03-24T09:29:47.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.613+0000 7f55e77fe640 1 -- 172.21.15.116:0/881812764 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d800cd80 con 0x7f55f01005c0 2024-03-24T09:29:47.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.613+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f0198640 con 0x7f55f01005c0 2024-03-24T09:29:47.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.613+0000 7f55e77fe640 1 -- 172.21.15.116:0/881812764 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d8018840 con 0x7f55f01005c0 2024-03-24T09:29:47.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.613+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f01a4880 con 0x7f55f01005c0 2024-03-24T09:29:47.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55e77fe640 1 -- 172.21.15.116:0/881812764 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f55d80189a0 con 0x7f55f01005c0 2024-03-24T09:29:47.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 Processor -- start 2024-03-24T09:29:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 -- start start 2024-03-24T09:29:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f72be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f7b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f7e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562566edc840 con 0x562566f73120 2024-03-24T09:29:47.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562566ed9d70 con 0x562566f7bd00 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.615+0000 7f55f77af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562566ed5550 con 0x562566f72800 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f7b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f7b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32876/0 (socket says 172.21.15.116:32876) 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 -- 172.21.15.116:0/1752604199 learned_addr learned my addr 172.21.15.116:0/1752604199 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55eed9d640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f72be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55edd9b640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f7e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55e77fe640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f55dc0a2230 0x7f55dc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 -- 172.21.15.116:0/1752604199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 msgr2=0x562566f72be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f72be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 -- 172.21.15.116:0/1752604199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 msgr2=0x562566f7e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f7e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55e77fe640 1 -- 172.21.15.116:0/881812764 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f55d8014070 con 0x7f55f01005c0 2024-03-24T09:29:47.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 -- 172.21.15.116:0/1752604199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562566e6ade0 con 0x562566f73120 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55edd9b640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f7e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55eed9d640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f72be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55eed9d640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f55dc0a2230 0x7f55dc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55ee59c640 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f7b7c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f55d80cbaa0 tx=0x7f55d800cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69413bddce7f5abc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55e4ff9640 1 -- 172.21.15.116:0/1752604199 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d809ae80 con 0x562566f73120 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55e4ff9640 1 -- 172.21.15.116:0/1752604199 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d809ef20 con 0x562566f73120 2024-03-24T09:29:47.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.616+0000 7f55e4ff9640 1 -- 172.21.15.116:0/1752604199 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80cbf10 con 0x562566f73120 2024-03-24T09:29:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 -- 172.21.15.116:0/1752604199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 msgr2=0x562566f7b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55eed9d640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f55dc0a2230 0x7f55dc0a46f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55e8004af0 tx=0x7f55e80095c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f7b7c0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f55d80cbaa0 tx=0x7f55d800cef0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 -- 172.21.15.116:0/1752604199 shutdown_connections 2024-03-24T09:29:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f7e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f7b7c0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 --2- 172.21.15.116:0/1752604199 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f72be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 -- 172.21.15.116:0/1752604199 >> 172.21.15.116:0/1752604199 conn(0x562566f6cf30 msgr2=0x562566f6d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 -- 172.21.15.116:0/1752604199 shutdown_connections 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.617+0000 7f55f77af740 1 -- 172.21.15.116:0/1752604199 wait complete. 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 -- start start 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 Processor -- start 2024-03-24T09:29:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 0x562566f78d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f73120 0x562566f79280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f797c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562566ba8430 con 0x562566f72800 2024-03-24T09:29:47.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562566b64e00 con 0x562566f7bd00 2024-03-24T09:29:47.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562566e5a590 con 0x562566f73120 2024-03-24T09:29:47.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 10 client.0 _ll_register_callbacks cb 0x562566ec9fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 0x562566f78d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 0x562566f78d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32884/0 (socket says 172.21.15.116:32884) 2024-03-24T09:29:47.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 -- 172.21.15.116:0/2478420742 learned_addr learned my addr 172.21.15.116:0/2478420742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55eed9d640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55edd9b640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f73120 0x562566f79280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 -- 172.21.15.116:0/2478420742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f73120 msgr2=0x562566f79280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f73120 0x562566f79280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 -- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 msgr2=0x562566f797c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f797c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 -- 172.21.15.116:0/2478420742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f01a4880 con 0x562566f72800 2024-03-24T09:29:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55eed9d640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55ee59c640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 0x562566f78d40 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f55d80cbbe0 tx=0x7f55d809ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a8736103bf638f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55d5ffb640 1 -- 172.21.15.116:0/2478420742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d80a3570 con 0x562566f72800 2024-03-24T09:29:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55d5ffb640 1 -- 172.21.15.116:0/2478420742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f55d80d4130 con 0x562566f72800 2024-03-24T09:29:47.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.618+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562566bd47e0 con 0x562566f72800 2024-03-24T09:29:47.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55d5ffb640 1 -- 172.21.15.116:0/2478420742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f55d809a2d0 con 0x562566f72800 2024-03-24T09:29:47.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 10 client.17811 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562566ed3bd0 con 0x562566f72800 2024-03-24T09:29:47.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55d5ffb640 1 -- 172.21.15.116:0/2478420742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562566bd47e0 con 0x562566f72800 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55d5ffb640 1 -- 172.21.15.116:0/2478420742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f55d80d6070 con 0x562566f72800 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 10 client.17811 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 10 client.17811 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 20 client.17811 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566f8bb10 0x562566f8df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 4 client.17811 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:47.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562566c11720 con 0x562566f8bb10 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55edd9b640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566f8bb10 0x562566f8df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55d5ffb640 10 client.17811 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.619+0000 7f55edd9b640 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566f8bb10 0x562566f8df30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dae571522f536a5f server_cookie=6790e58d155b343c in_seq=0 out_seq=0 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55d5ffb640 1 -- 172.21.15.116:0/2478420742 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562566c11720 con 0x562566f8bb10 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55d5ffb640 10 client.17811 handle_command_reply: tid=0 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 client.17811 shutdown 2024-03-24T09:29:47.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 msgr2=0x562566f78d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 0x562566f78d40 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f55d80cbbe0 tx=0x7f55d809ae80 comp rx=0 tx=0).stop 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 shutdown_connections 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566f8bb10 0x562566f8df30 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f7bd00 0x562566f797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f73120 0x562566f79280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 --2- 172.21.15.116:0/2478420742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f72800 0x562566f78d40 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55d5ffb640 0 client.17811 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:47.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 >> 172.21.15.116:0/2478420742 conn(0x562566f6cf30 msgr2=0x562566f67cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 shutdown_connections 2024-03-24T09:29:47.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 1 -- 172.21.15.116:0/2478420742 wait complete. 2024-03-24T09:29:47.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.620+0000 7f55f77af740 20 client.17811 trim_cache size 0 max 16384 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.633+0000 7f55f77af740 1 Processor -- start 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.633+0000 7f55f77af740 1 -- start start 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.633+0000 7f55f77af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f7b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55f77af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f73120 0x562566f87530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55f77af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 0x562566f93800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55f77af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562566f64240 con 0x562566f7bd00 2024-03-24T09:29:47.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55f77af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562566f8e5e0 con 0x562566f73120 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55f77af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562566f71fa0 con 0x562566f72800 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 0x562566f93800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 0x562566f93800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32898/0 (socket says 172.21.15.116:32898) 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 -- 172.21.15.116:0/2060904600 learned_addr learned my addr 172.21.15.116:0/2060904600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55ee59c640 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f7b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55edd9b640 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f73120 0x562566f87530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 -- 172.21.15.116:0/2060904600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 msgr2=0x562566f7b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f7b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 -- 172.21.15.116:0/2060904600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f73120 msgr2=0x562566f87530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f73120 0x562566f87530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 -- 172.21.15.116:0/2060904600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562566e6fc80 con 0x562566f7bd00 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55eed9d640 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 0x562566f93800 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e8043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525bc7abc5b616cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55d7fff640 1 -- 172.21.15.116:0/2060904600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e8034c60 con 0x562566f7bd00 2024-03-24T09:29:47.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55d7fff640 1 -- 172.21.15.116:0/2060904600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55e8032ad0 con 0x562566f7bd00 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.634+0000 7f55d7fff640 1 -- 172.21.15.116:0/2060904600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e80042a0 con 0x562566f7bd00 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- 172.21.15.116:0/2060904600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 msgr2=0x562566f93800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 0x562566f93800 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f55e8036000 tx=0x7f55e8043000 comp rx=0 tx=0).stop 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- 172.21.15.116:0/2060904600 shutdown_connections 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f7bd00 0x562566f93800 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f73120 0x562566f87530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- 172.21.15.116:0/2060904600 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f72800 0x562566f7b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- 172.21.15.116:0/2060904600 >> 172.21.15.116:0/2060904600 conn(0x562566f6cf30 msgr2=0x562566f67cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- 172.21.15.116:0/2060904600 shutdown_connections 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- 172.21.15.116:0/2060904600 wait complete. 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- start start 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 Processor -- start 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f72800 0x562566f80470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f809b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f7bd00 0x562566f80ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562566ff1e40 con 0x562566f73120 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562566ff1f80 con 0x562566f72800 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562566ff20c0 con 0x562566f7bd00 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55f77af740 10 client.0 _ll_register_callbacks cb 0x7f55e0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.635+0000 7f55edd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f809b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f809b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32912/0 (socket says 172.21.15.116:32912) 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 -- 172.21.15.116:0/1637658451 learned_addr learned my addr 172.21.15.116:0/1637658451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55ee59c640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f72800 0x562566f80470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55eed9d640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f7bd00 0x562566f80ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 -- 172.21.15.116:0/1637658451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f7bd00 msgr2=0x562566f80ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f7bd00 0x562566f80ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 -- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f72800 msgr2=0x562566f80470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f72800 0x562566f80470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 -- 172.21.15.116:0/1637658451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e8030070 con 0x562566f73120 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55edd9b640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f809b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x562566f6e0c0 tx=0x7f55e0010950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99c42f09d6f783f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55d5ffb640 1 -- 172.21.15.116:0/1637658451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e00205c0 con 0x562566f73120 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55d5ffb640 1 -- 172.21.15.116:0/1637658451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f55e0020720 con 0x562566f73120 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55d809ef20 con 0x562566f73120 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55d5ffb640 1 -- 172.21.15.116:0/1637658451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f55e0004030 con 0x562566f73120 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55f77af740 10 client.17817 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562566e95920 con 0x562566f73120 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55d5ffb640 1 -- 172.21.15.116:0/1637658451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f55d809ef20 con 0x562566f73120 2024-03-24T09:29:47.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.636+0000 7f55d5ffb640 1 -- 172.21.15.116:0/1637658451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f55e001e3f0 con 0x562566f73120 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 10 client.17817 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 10 client.17817 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 20 client.17817 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566ff3910 0x562566ff5d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 4 client.17817 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562566c60890 con 0x562566ff3910 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55ee59c640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566ff3910 0x562566ff5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55ee59c640 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566ff3910 0x562566ff5d30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=89ad7ff514a3b496 server_cookie=3152db44a8282531 in_seq=0 out_seq=0 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55d5ffb640 10 client.17817 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55d5ffb640 1 -- 172.21.15.116:0/1637658451 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562566c60890 con 0x562566ff3910 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55d5ffb640 10 client.17817 handle_command_reply: tid=0 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 client.17817 shutdown 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 msgr2=0x562566f809b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f809b0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x562566f6e0c0 tx=0x7f55e0010950 comp rx=0 tx=0).stop 2024-03-24T09:29:47.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 shutdown_connections 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x562566ff3910 0x562566ff5d30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562566f7bd00 0x562566f80ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55d5ffb640 0 client.17817 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562566f73120 0x562566f809b0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 --2- 172.21.15.116:0/1637658451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562566f72800 0x562566f80470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 >> 172.21.15.116:0/1637658451 conn(0x562566f6cf30 msgr2=0x562566f78ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 shutdown_connections 2024-03-24T09:29:47.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.637+0000 7f55f77af740 1 -- 172.21.15.116:0/1637658451 wait complete. 2024-03-24T09:29:47.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f77af740 20 client.17817 trim_cache size 0 max 16384 2024-03-24T09:29:47.660 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:47.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f55dc0a2230 msgr2=0x7f55dc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f55dc0a2230 0x7f55dc0a46f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f55e8004af0 tx=0x7f55e80095c0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 msgr2=0x7f55f019d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 0x7f55f019d7e0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f55d800e9f0 tx=0x7f55d800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 shutdown_connections 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55f0106680 0x7f55f01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f55dc0a2230 0x7f55dc0a46f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55f0100f80 0x7f55f019dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 --2- 172.21.15.116:0/881812764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55f01005c0 0x7f55f019d7e0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 >> 172.21.15.116:0/881812764 conn(0x7f55f00776e0 msgr2=0x7f55f00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 shutdown_connections 2024-03-24T09:29:47.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.638+0000 7f55f4aed640 1 -- 172.21.15.116:0/881812764 wait complete. 2024-03-24T09:29:47.663 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:29:47.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2274153077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe81005a0 msgr2=0x7fdfe8100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2274153077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe81005a0 0x7fdfe8100a20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009900 tx=0x7fdfd002bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:47.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2274153077 shutdown_connections 2024-03-24T09:29:47.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2274153077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe8100f60 0x7fdfe810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2274153077 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe81005a0 0x7fdfe8100a20 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2274153077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfe8106660 0x7fdfe8106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2274153077 >> 172.21.15.116:0/2274153077 conn(0x7fdfe8077710 msgr2=0x7fdfe8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2274153077 shutdown_connections 2024-03-24T09:29:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.766+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2274153077 wait complete. 2024-03-24T09:29:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 Processor -- start 2024-03-24T09:29:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 -- start start 2024-03-24T09:29:47.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe81005a0 0x7fdfe810ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 0x7fdfe810e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfe8106660 0x7fdfe810e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe81a6030 con 0x7fdfe8100f60 2024-03-24T09:29:47.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe81a61a0 con 0x7fdfe81005a0 2024-03-24T09:29:47.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfedfc2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe81a6360 con 0x7fdfe8106660 2024-03-24T09:29:47.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 0x7fdfe810e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 0x7fdfe810e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32934/0 (socket says 172.21.15.116:32934) 2024-03-24T09:29:47.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 -- 172.21.15.116:0/2481959960 learned_addr learned my addr 172.21.15.116:0/2481959960 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe81005a0 0x7fdfe810ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfe8106660 0x7fdfe810e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 -- 172.21.15.116:0/2481959960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfe8106660 msgr2=0x7fdfe810e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfe8106660 0x7fdfe810e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 -- 172.21.15.116:0/2481959960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe81005a0 msgr2=0x7fdfe810ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe81005a0 0x7fdfe810ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 -- 172.21.15.116:0/2481959960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe0009840 con 0x7fdfe8100f60 2024-03-24T09:29:47.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.767+0000 7fdfdffff640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 0x7fdfe810e330 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0005940 tx=0x7fdfd0005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.768+0000 7fdfe57fa640 1 -- 172.21.15.116:0/2481959960 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0004350 con 0x7fdfe8100f60 2024-03-24T09:29:47.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.768+0000 7fdfe57fa640 1 -- 172.21.15.116:0/2481959960 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfd00044b0 con 0x7fdfe8100f60 2024-03-24T09:29:47.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.768+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd0009560 con 0x7fdfe8100f60 2024-03-24T09:29:47.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.768+0000 7fdfe57fa640 1 -- 172.21.15.116:0/2481959960 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00052b0 con 0x7fdfe8100f60 2024-03-24T09:29:47.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.768+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe81a6960 con 0x7fdfe8100f60 2024-03-24T09:29:47.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdfe57fa640 1 -- 172.21.15.116:0/2481959960 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fdfd0003cc0 con 0x7fdfe8100f60 2024-03-24T09:29:47.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 Processor -- start 2024-03-24T09:29:47.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 -- start start 2024-03-24T09:29:47.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b287d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b33ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d4a96840 con 0x5633d4b283f0 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d4a93d70 con 0x5633d4b318f0 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633d4a8f550 con 0x5633d4b28d10 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b287d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdfe57fa640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdfd40a2230 0x7fdfd40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b287d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32942/0 (socket says 172.21.15.116:32942) 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.770+0000 7fdfe7fff640 1 -- 172.21.15.116:0/1721782981 learned_addr learned my addr 172.21.15.116:0/1721782981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfdffff640 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b33ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdfd40a2230 0x7fdfd40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe77fe640 1 -- 172.21.15.116:0/1721782981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 msgr2=0x5633d4b33ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b33ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe77fe640 1 -- 172.21.15.116:0/1721782981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 msgr2=0x5633d4b287d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b287d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe77fe640 1 -- 172.21.15.116:0/1721782981 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633d4a24de0 con 0x5633d4b28d10 2024-03-24T09:29:47.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b287d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe57fa640 1 -- 172.21.15.116:0/2481959960 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdfd00f0490 con 0x7fdfe8100f60 2024-03-24T09:29:47.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b313b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009c20 tx=0x7fdfd8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14498ad1ff4bed2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdfd40a2230 0x7fdfd40a46f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009b80 tx=0x7fdfe0004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfddffb640 1 -- 172.21.15.116:0/1721782981 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8004050 con 0x5633d4b28d10 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfddffb640 1 -- 172.21.15.116:0/1721782981 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfd8005020 con 0x5633d4b28d10 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.771+0000 7fdfddffb640 1 -- 172.21.15.116:0/1721782981 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8004bc0 con 0x5633d4b28d10 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1721782981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 msgr2=0x5633d4b313b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b313b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009c20 tx=0x7fdfd8011810 comp rx=0 tx=0).stop 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1721782981 shutdown_connections 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b33ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b313b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1721782981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b287d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1721782981 >> 172.21.15.116:0/1721782981 conn(0x5633d4b22b20 msgr2=0x5633d4b22f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1721782981 shutdown_connections 2024-03-24T09:29:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1721782981 wait complete. 2024-03-24T09:29:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 -- start start 2024-03-24T09:29:47.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.772+0000 7fdff0b9b740 1 Processor -- start 2024-03-24T09:29:47.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 0x5633d4b2e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b2ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 0x5633d4b2f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d4762430 con 0x5633d4b318f0 2024-03-24T09:29:47.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d471ee00 con 0x5633d4b28d10 2024-03-24T09:29:47.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633d4a14590 con 0x5633d4b283f0 2024-03-24T09:29:47.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 10 client.0 _ll_register_callbacks cb 0x5633d4a83fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 0x5633d4b2f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 0x5633d4b2e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 0x5633d4b2f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32944/0 (socket says 172.21.15.116:32944) 2024-03-24T09:29:47.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 -- 172.21.15.116:0/1614731883 learned_addr learned my addr 172.21.15.116:0/1614731883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 0x5633d4b2e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:34310/0 (socket says 172.21.15.116:34310) 2024-03-24T09:29:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfdffff640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b2ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 -- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 msgr2=0x5633d4b2e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 0x5633d4b2e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 -- 172.21.15.116:0/1614731883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 msgr2=0x5633d4b2ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b2ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 -- 172.21.15.116:0/1614731883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd8009850 con 0x5633d4b318f0 2024-03-24T09:29:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 0x5633d4b2e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfdffff640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b2ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 0x5633d4b2f3a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00135e0 tx=0x7fdfe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a594524442fb4ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/1614731883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe00223f0 con 0x5633d4b318f0 2024-03-24T09:29:47.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/1614731883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdfe0022550 con 0x5633d4b318f0 2024-03-24T09:29:47.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633d478e7e0 con 0x5633d4b318f0 2024-03-24T09:29:47.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/1614731883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdfe0022710 con 0x5633d4b318f0 2024-03-24T09:29:47.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.773+0000 7fdff0b9b740 10 client.17829 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633d4a8dbd0 con 0x5633d4b318f0 2024-03-24T09:29:47.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/1614731883 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5633d478e7e0 con 0x5633d4b318f0 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/1614731883 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fdfe000f740 con 0x5633d4b318f0 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 10 client.17829 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 10 client.17829 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 20 client.17829 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4b41a60 0x5633d4b43e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 4 client.17829 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:47.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633d47cb720 con 0x5633d4b41a60 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4b41a60 0x5633d4b43e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdfc6ffd640 10 client.17829 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.774+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4b41a60 0x5633d4b43e80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69eb59c50cf976d6 server_cookie=b75511ab06a465ac in_seq=0 out_seq=0 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/1614731883 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633d47cb720 con 0x5633d4b41a60 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdfc6ffd640 10 client.17829 handle_command_reply: tid=0 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 client.17829 shutdown 2024-03-24T09:29:47.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 msgr2=0x5633d4b2f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 0x5633d4b2f3a0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fdfe00135e0 tx=0x7fdfe0004a40 comp rx=0 tx=0).stop 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 shutdown_connections 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b318f0 0x5633d4b2f3a0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4b41a60 0x5633d4b43e80 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b2ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdfc6ffd640 0 client.17829 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/1614731883 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b283f0 0x5633d4b2e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 >> 172.21.15.116:0/1614731883 conn(0x5633d4b22b20 msgr2=0x5633d4b24870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 shutdown_connections 2024-03-24T09:29:47.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 1 -- 172.21.15.116:0/1614731883 wait complete. 2024-03-24T09:29:47.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.775+0000 7fdff0b9b740 20 client.17829 trim_cache size 0 max 16384 2024-03-24T09:29:47.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.788+0000 7fdff0b9b740 1 Processor -- start 2024-03-24T09:29:47.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.788+0000 7fdff0b9b740 1 -- start start 2024-03-24T09:29:47.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b3d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b30750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b318f0 0x5633d4b49770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d4b27d60 con 0x5633d4b283f0 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d47cb750 con 0x5633d4b28d10 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633d4b40230 con 0x5633d4b318f0 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b3d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b3d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32960/0 (socket says 172.21.15.116:32960) 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 -- 172.21.15.116:0/3008944905 learned_addr learned my addr 172.21.15.116:0/3008944905 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfdffff640 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b30750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b318f0 0x5633d4b49770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 -- 172.21.15.116:0/3008944905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b318f0 msgr2=0x5633d4b49770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b318f0 0x5633d4b49770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 -- 172.21.15.116:0/3008944905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 msgr2=0x5633d4b30750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b30750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 -- 172.21.15.116:0/3008944905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633d4a29c80 con 0x5633d4b283f0 2024-03-24T09:29:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b3d180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009c20 tx=0x7fdfd80096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee302e8b01c25ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfdd7fa640 1 -- 172.21.15.116:0/3008944905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd802a040 con 0x5633d4b283f0 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfdd7fa640 1 -- 172.21.15.116:0/3008944905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfd8011d10 con 0x5633d4b283f0 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.789+0000 7fdfdd7fa640 1 -- 172.21.15.116:0/3008944905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd802e6d0 con 0x5633d4b283f0 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 -- 172.21.15.116:0/3008944905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 msgr2=0x5633d4b3d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b3d180 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009c20 tx=0x7fdfd80096a0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 -- 172.21.15.116:0/3008944905 shutdown_connections 2024-03-24T09:29:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b318f0 0x5633d4b49770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b28d10 0x5633d4b30750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/3008944905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b3d180 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 -- 172.21.15.116:0/3008944905 >> 172.21.15.116:0/3008944905 conn(0x5633d4b22b20 msgr2=0x5633d4b30040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 -- 172.21.15.116:0/3008944905 shutdown_connections 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 -- 172.21.15.116:0/3008944905 wait complete. 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 -- start start 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.790+0000 7fdff0b9b740 1 Processor -- start 2024-03-24T09:29:47.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b39820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b34b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b350c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633d4b35600 con 0x5633d4b283f0 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633d4b35770 con 0x5633d4b318f0 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633d4b35930 con 0x5633d4b28d10 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 10 client.0 _ll_register_callbacks cb 0x7fdfe0003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b39820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b39820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32974/0 (socket says 172.21.15.116:32974) 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 -- 172.21.15.116:0/222330365 learned_addr learned my addr 172.21.15.116:0/222330365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe7fff640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfdffff640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b34b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 -- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 msgr2=0x5633d4b34b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b34b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 -- 172.21.15.116:0/222330365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 msgr2=0x5633d4b350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 -- 172.21.15.116:0/222330365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633d4a24de0 con 0x5633d4b283f0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfe77fe640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b39820 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009350 tx=0x7fdfd802eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19665bf3b8a4cc3d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfc6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/222330365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd802a040 con 0x5633d4b283f0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/222330365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fdfd8004c90 con 0x5633d4b283f0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.791+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd8005020 con 0x5633d4b283f0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/222330365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fdfd8005680 con 0x5633d4b283f0 2024-03-24T09:29:47.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 10 client.17835 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5633d4a4f920 con 0x5633d4b283f0 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/222330365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fdfd8005020 con 0x5633d4b283f0 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/222330365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fdfd8021040 con 0x5633d4b283f0 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 10 client.17835 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 10 client.17835 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 20 client.17835 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4ba7f60 0x5633d4baa350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 4 client.17835 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633d481a890 con 0x5633d4ba7f60 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdfdffff640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4ba7f60 0x5633d4baa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdfc6ffd640 10 client.17835 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.792+0000 7fdfdffff640 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4ba7f60 0x5633d4baa350 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af0bfae5badcb795 server_cookie=dd9d71c4a0be25c2 in_seq=0 out_seq=0 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfc6ffd640 1 -- 172.21.15.116:0/222330365 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633d481a890 con 0x5633d4ba7f60 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfc6ffd640 10 client.17835 handle_command_reply: tid=0 2024-03-24T09:29:47.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 client.17835 shutdown 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 msgr2=0x5633d4b39820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b39820 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009350 tx=0x7fdfd802eab0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 shutdown_connections 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5633d4ba7f60 0x5633d4baa350 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633d4b318f0 0x5633d4b350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633d4b28d10 0x5633d4b34b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfc6ffd640 0 client.17835 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:29:47.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 --2- 172.21.15.116:0/222330365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633d4b283f0 0x5633d4b39820 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 >> 172.21.15.116:0/222330365 conn(0x5633d4b22b20 msgr2=0x5633d4b21cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 shutdown_connections 2024-03-24T09:29:47.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 1 -- 172.21.15.116:0/222330365 wait complete. 2024-03-24T09:29:47.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdff0b9b740 20 client.17835 trim_cache size 0 max 16384 2024-03-24T09:29:47.814 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdfd40a2230 msgr2=0x7fdfd40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdfd40a2230 0x7fdfd40a46f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009b80 tx=0x7fdfe0004ab0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 msgr2=0x7fdfe810e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 0x7fdfe810e330 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0005940 tx=0x7fdfd0005df0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 shutdown_connections 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fdfd40a2230 0x7fdfd40a46f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdfe8106660 0x7fdfe810e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfe8100f60 0x7fdfe810e330 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 --2- 172.21.15.116:0/2481959960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfe81005a0 0x7fdfe810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 >> 172.21.15.116:0/2481959960 conn(0x7fdfe8077710 msgr2=0x7fdfe80ff3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.793+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 shutdown_connections 2024-03-24T09:29:47.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.794+0000 7fdfedfc2640 1 -- 172.21.15.116:0/2481959960 wait complete. 2024-03-24T09:29:47.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:29:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 -- 172.21.15.116:0/2270390832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c10bd20 msgr2=0x7f088c10e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 --2- 172.21.15.116:0/2270390832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c10bd20 0x7f088c10e150 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0884009900 tx=0x7f088402bcd0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 -- 172.21.15.116:0/2270390832 shutdown_connections 2024-03-24T09:29:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 --2- 172.21.15.116:0/2270390832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c10bd20 0x7f088c10e150 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 --2- 172.21.15.116:0/2270390832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c103270 0x7f088c10b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 --2- 172.21.15.116:0/2270390832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 0x7f088c102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 -- 172.21.15.116:0/2270390832 >> 172.21.15.116:0/2270390832 conn(0x7f088c076990 msgr2=0x7f088c076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 -- 172.21.15.116:0/2270390832 shutdown_connections 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.920+0000 7f0890fa0640 1 -- 172.21.15.116:0/2270390832 wait complete. 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 Processor -- start 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 -- start start 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 0x7f088c1005a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 0x7f088c100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c10bd20 0x7f088c101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f088c1a5ff0 con 0x7f088c103270 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f088c1a6160 con 0x7f088c102950 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0890fa0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f088c1a6320 con 0x7f088c10bd20 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0889d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 0x7f088c100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0889d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 0x7f088c100ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32986/0 (socket says 172.21.15.116:32986) 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f0889d9b640 1 -- 172.21.15.116:0/1740215550 learned_addr learned my addr 172.21.15.116:0/1740215550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f088ad9d640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c10bd20 0x7f088c101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.921+0000 7f088a59c640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 0x7f088c1005a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0889d9b640 1 -- 172.21.15.116:0/1740215550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c10bd20 msgr2=0x7f088c101020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0889d9b640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c10bd20 0x7f088c101020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0889d9b640 1 -- 172.21.15.116:0/1740215550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 msgr2=0x7f088c1005a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0889d9b640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 0x7f088c1005a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0889d9b640 1 -- 172.21.15.116:0/1740215550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c009840 con 0x7f088c103270 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f088a59c640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 0x7f088c1005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0889d9b640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 0x7f088c100ae0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f087400e9f0 tx=0x7f087400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f08837fe640 1 -- 172.21.15.116:0/1740215550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874009720 con 0x7f088c103270 2024-03-24T09:29:47.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f08837fe640 1 -- 172.21.15.116:0/1740215550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f087400cd80 con 0x7f088c103270 2024-03-24T09:29:47.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0884009560 con 0x7f088c103270 2024-03-24T09:29:47.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f08837fe640 1 -- 172.21.15.116:0/1740215550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874010430 con 0x7f088c103270 2024-03-24T09:29:47.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.922+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f088c1a6980 con 0x7f088c103270 2024-03-24T09:29:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.924+0000 7f08837fe640 1 -- 172.21.15.116:0/1740215550 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f0874005020 con 0x7f088c103270 2024-03-24T09:29:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.924+0000 7f0893b77740 1 Processor -- start 2024-03-24T09:29:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 -- start start 2024-03-24T09:29:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d33f0 0x5633570d37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 0x5633570dc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570dece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563357041840 con 0x5633570dc8f0 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56335703ed70 con 0x5633570d3d10 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0893b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56335703a550 con 0x5633570d33f0 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f08837fe640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08780a22e0 0x7f08780a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f0889d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570dece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 0x5633570dc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088ad9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d33f0 0x5633570d37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088ad9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d33f0 0x5633570d37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:34350/0 (socket says 172.21.15.116:34350) 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088ad9d640 1 -- 172.21.15.116:0/3385535496 learned_addr learned my addr 172.21.15.116:0/3385535496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 0x5633570dc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:33952/0 (socket says 172.21.15.116:33952) 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088ad9d640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08780a22e0 0x7f08780a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f08837fe640 1 -- 172.21.15.116:0/1740215550 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0874014070 con 0x7f088c103270 2024-03-24T09:29:47.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 -- 172.21.15.116:0/3385535496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d33f0 msgr2=0x5633570d37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d33f0 0x5633570d37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 -- 172.21.15.116:0/3385535496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 msgr2=0x5633570dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570dece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.925+0000 7f088a59c640 1 -- 172.21.15.116:0/3385535496 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563356fcfde0 con 0x5633570d3d10 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f088a59c640 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 0x5633570dc3b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=972b125d3cdb8e22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f088ad9d640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08780a22e0 0x7f08780a47a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0884005ab0 tx=0x7f08840058d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0880ff9640 1 -- 172.21.15.116:0/3385535496 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c004d20 con 0x5633570d3d10 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0880ff9640 1 -- 172.21.15.116:0/3385535496 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f087c004e80 con 0x5633570d3d10 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0880ff9640 1 -- 172.21.15.116:0/3385535496 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c005910 con 0x5633570d3d10 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 -- 172.21.15.116:0/3385535496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 msgr2=0x5633570dc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 0x5633570dc3b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c00ed70 comp rx=0 tx=0).stop 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 -- 172.21.15.116:0/3385535496 shutdown_connections 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570dece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d3d10 0x5633570dc3b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 --2- 172.21.15.116:0/3385535496 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d33f0 0x5633570d37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.926+0000 7f0893b77740 1 -- 172.21.15.116:0/3385535496 >> 172.21.15.116:0/3385535496 conn(0x5633570cdb20 msgr2=0x5633570cdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 -- 172.21.15.116:0/3385535496 shutdown_connections 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 -- 172.21.15.116:0/3385535496 wait complete. 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 -- start start 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 Processor -- start 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570d9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570d9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570da3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563356d0d430 con 0x5633570dc8f0 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563356cc9e00 con 0x5633570d33f0 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563356fbf590 con 0x5633570d3d10 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0893b77740 10 client.0 _ll_register_callbacks cb 0x56335702efc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f088a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570d9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f088ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570da3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f088a59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570d9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:33956/0 (socket says 172.21.15.116:33956) 2024-03-24T09:29:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f088a59c640 1 -- 172.21.15.116:0/4131161106 learned_addr learned my addr 172.21.15.116:0/4131161106 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.927+0000 7f0889d9b640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570d9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088ad9d640 1 -- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 msgr2=0x5633570d9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088ad9d640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570d9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088ad9d640 1 -- 172.21.15.116:0/4131161106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 msgr2=0x5633570d9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088ad9d640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570d9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088ad9d640 1 -- 172.21.15.116:0/4131161106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f087c00e540 con 0x5633570dc8f0 2024-03-24T09:29:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088a59c640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570d9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f088ad9d640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570da3d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f088402c7b0 tx=0x7f0884009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83da7021e487e0f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0871ffb640 1 -- 172.21.15.116:0/4131161106 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884003800 con 0x5633570dc8f0 2024-03-24T09:29:47.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0871ffb640 1 -- 172.21.15.116:0/4131161106 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0884003960 con 0x5633570dc8f0 2024-03-24T09:29:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563356d76720 con 0x5633570dc8f0 2024-03-24T09:29:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0871ffb640 1 -- 172.21.15.116:0/4131161106 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0884003b20 con 0x5633570dc8f0 2024-03-24T09:29:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 10 client.17847 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563357038bd0 con 0x5633570dc8f0 2024-03-24T09:29:47.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0871ffb640 1 -- 172.21.15.116:0/4131161106 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563356d76720 con 0x5633570dc8f0 2024-03-24T09:29:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0871ffb640 1 -- 172.21.15.116:0/4131161106 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f088404f480 con 0x5633570dc8f0 2024-03-24T09:29:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 10 client.17847 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 10 client.17847 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:29:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 20 client.17847 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633570eca30 0x5633570eee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 4 client.17847 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.928+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563356dc5890 con 0x5633570eca30 2024-03-24T09:29:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.929+0000 7f088a59c640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633570eca30 0x5633570eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.929+0000 7f088a59c640 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633570eca30 0x5633570eee60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2ac7a2c6863e16c server_cookie=d6894313c1040c55 in_seq=0 out_seq=0 2024-03-24T09:29:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.929+0000 7f0871ffb640 10 client.17847 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:47.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0871ffb640 1 -- 172.21.15.116:0/4131161106 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563356dc5890 con 0x5633570eca30 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0871ffb640 10 client.17847 handle_command_reply: tid=0 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 client.17847 shutdown 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 msgr2=0x5633570da3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570da3d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f088402c7b0 tx=0x7f0884009260 comp rx=0 tx=0).stop 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 shutdown_connections 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5633570eca30 0x5633570eee60 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x5633570da3d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0871ffb640 0 client.17847 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570d9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 --2- 172.21.15.116:0/4131161106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570d9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 >> 172.21.15.116:0/4131161106 conn(0x5633570cdb20 msgr2=0x5633570cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 shutdown_connections 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 1 -- 172.21.15.116:0/4131161106 wait complete. 2024-03-24T09:29:47.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.930+0000 7f0893b77740 20 client.17847 trim_cache size 0 max 16384 2024-03-24T09:29:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 Processor -- start 2024-03-24T09:29:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 -- start start 2024-03-24T09:29:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 0x5633570e8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570f4720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570dc8f0 0x5633570d9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633570e86c0 con 0x5633570d33f0 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5633570ceb20 con 0x5633570dc8f0 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0893b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633570d5520 con 0x5633570d3d10 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 0x5633570e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 0x5633570e8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33006/0 (socket says 172.21.15.116:33006) 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 -- 172.21.15.116:0/2586944664 learned_addr learned my addr 172.21.15.116:0/2586944664 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088ad9d640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570dc8f0 0x5633570d9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0889d9b640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 -- 172.21.15.116:0/2586944664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 msgr2=0x5633570f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570f4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 -- 172.21.15.116:0/2586944664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570dc8f0 msgr2=0x5633570d9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570dc8f0 0x5633570d9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088a59c640 1 -- 172.21.15.116:0/2586944664 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563356fd4c80 con 0x5633570d33f0 2024-03-24T09:29:47.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f0889d9b640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.943+0000 7f088ad9d640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570dc8f0 0x5633570d9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f088a59c640 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 0x5633570e8180 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e51d3a5c2fbb8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0873fff640 1 -- 172.21.15.116:0/2586944664 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c02a070 con 0x5633570d33f0 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0873fff640 1 -- 172.21.15.116:0/2586944664 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f087c005450 con 0x5633570d33f0 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0873fff640 1 -- 172.21.15.116:0/2586944664 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c011a20 con 0x5633570d33f0 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 -- 172.21.15.116:0/2586944664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 msgr2=0x5633570e8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 0x5633570e8180 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c009fd0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 -- 172.21.15.116:0/2586944664 shutdown_connections 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570dc8f0 0x5633570d9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 --2- 172.21.15.116:0/2586944664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570d33f0 0x5633570e8180 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 -- 172.21.15.116:0/2586944664 >> 172.21.15.116:0/2586944664 conn(0x5633570cdb20 msgr2=0x5633570db900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 -- 172.21.15.116:0/2586944664 shutdown_connections 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.944+0000 7f0893b77740 1 -- 172.21.15.116:0/2586944664 wait complete. 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 -- start start 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 Processor -- start 2024-03-24T09:29:47.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570e0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x563357151d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633570e12e0 con 0x5633570dc8f0 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563357152290 con 0x5633570d33f0 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5633571523d0 con 0x5633570d3d10 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0893b77740 10 client.0 _ll_register_callbacks cb 0x7f0884009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:47.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x563357151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x563357151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33012/0 (socket says 172.21.15.116:33012) 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 -- 172.21.15.116:0/162405628 learned_addr learned my addr 172.21.15.116:0/162405628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088a59c640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f0889d9b640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570e0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 -- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 msgr2=0x5633570e0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570e0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 -- 172.21.15.116:0/162405628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 msgr2=0x5633570e04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:47.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570e04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088ad9d640 1 -- 172.21.15.116:0/162405628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563356fcfde0 con 0x5633570dc8f0 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.945+0000 7f088a59c640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570e04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f088ad9d640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x563357151d50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0884005c40 tx=0x7f088404f6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586657af6c758881 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0871ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0871ffb640 1 -- 172.21.15.116:0/162405628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08840046c0 con 0x5633570dc8f0 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0871ffb640 1 -- 172.21.15.116:0/162405628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f088402c810 con 0x5633570dc8f0 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563356d397e0 con 0x5633570dc8f0 2024-03-24T09:29:47.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0871ffb640 1 -- 172.21.15.116:0/162405628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f088402c9d0 con 0x5633570dc8f0 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 10 client.17853 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563356ffa920 con 0x5633570dc8f0 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0871ffb640 1 -- 172.21.15.116:0/162405628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563356d397e0 con 0x5633570dc8f0 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0871ffb640 1 -- 172.21.15.116:0/162405628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f08840050d0 con 0x5633570dc8f0 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 10 client.17853 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 10 client.17853 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 20 client.17853 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:47.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563357153850 0x563357155c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 4 client.17853 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563356ce5700 con 0x563357153850 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.946+0000 7f088a59c640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563357153850 0x563357155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f088a59c640 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563357153850 0x563357155c70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8cabfad92b15e990 server_cookie=4c6878748bd7c426 in_seq=0 out_seq=0 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0871ffb640 10 client.17853 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0871ffb640 1 -- 172.21.15.116:0/162405628 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563356ce5700 con 0x563357153850 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0871ffb640 10 client.17853 handle_command_reply: tid=0 2024-03-24T09:29:47.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 client.17853 shutdown 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 msgr2=0x563357151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x563357151d50 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f0884005c40 tx=0x7f088404f6b0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 shutdown_connections 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x563357153850 0x563357155c70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633570dc8f0 0x563357151d50 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0871ffb640 0 client.17853 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:29:47.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5633570d3d10 0x5633570e0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 --2- 172.21.15.116:0/162405628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5633570d33f0 0x5633570e04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 >> 172.21.15.116:0/162405628 conn(0x5633570cdb20 msgr2=0x5633570cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 shutdown_connections 2024-03-24T09:29:47.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 1 -- 172.21.15.116:0/162405628 wait complete. 2024-03-24T09:29:47.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.947+0000 7f0893b77740 20 client.17853 trim_cache size 0 max 16384 2024-03-24T09:29:47.967 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08780a22e0 msgr2=0x7f08780a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08780a22e0 0x7f08780a47a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0884005ab0 tx=0x7f08840058d0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 msgr2=0x7f088c100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 0x7f088c100ae0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f087400e9f0 tx=0x7f087400edf0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 shutdown_connections 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f08780a22e0 0x7f08780a47a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f088c10bd20 0x7f088c101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f088c103270 0x7f088c100ae0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 --2- 172.21.15.116:0/1740215550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f088c102950 0x7f088c1005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 >> 172.21.15.116:0/1740215550 conn(0x7f088c076990 msgr2=0x7f088c0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 shutdown_connections 2024-03-24T09:29:47.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:47.948+0000 7f0890fa0640 1 -- 172.21.15.116:0/1740215550 wait complete. 2024-03-24T09:29:47.970 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:29:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 -- 172.21.15.116:0/3133239002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501016b0 msgr2=0x7fe950101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 --2- 172.21.15.116:0/3133239002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501016b0 0x7fe950101b30 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fe944009930 tx=0x7fe94402bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 -- 172.21.15.116:0/3133239002 shutdown_connections 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 --2- 172.21.15.116:0/3133239002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 0x7fe95010e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 --2- 172.21.15.116:0/3133239002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501016b0 0x7fe950101b30 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 --2- 172.21.15.116:0/3133239002 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9501076c0 0x7fe950107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 -- 172.21.15.116:0/3133239002 >> 172.21.15.116:0/3133239002 conn(0x7fe9500fd140 msgr2=0x7fe9500ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 -- 172.21.15.116:0/3133239002 shutdown_connections 2024-03-24T09:29:48.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.075+0000 7fe95689d640 1 -- 172.21.15.116:0/3133239002 wait complete. 2024-03-24T09:29:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 Processor -- start 2024-03-24T09:29:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 -- start start 2024-03-24T09:29:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9501016b0 0x7fe95019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 0x7fe95019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 0x7fe950197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe950198130 con 0x7fe9501076c0 2024-03-24T09:29:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe9501982a0 con 0x7fe950102070 2024-03-24T09:29:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe95689d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe950198460 con 0x7fe9501016b0 2024-03-24T09:29:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe954e3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 0x7fe950197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe954e3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 0x7fe950197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33018/0 (socket says 172.21.15.116:33018) 2024-03-24T09:29:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe954e3a640 1 -- 172.21.15.116:0/2762038767 learned_addr learned my addr 172.21.15.116:0/2762038767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 0x7fe95019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.076+0000 7fe94ffff640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9501016b0 0x7fe95019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94f7fe640 1 -- 172.21.15.116:0/2762038767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9501016b0 msgr2=0x7fe95019d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9501016b0 0x7fe95019d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94f7fe640 1 -- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 msgr2=0x7fe950197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 0x7fe950197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94f7fe640 1 -- 172.21.15.116:0/2762038767 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe948009840 con 0x7fe950102070 2024-03-24T09:29:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe954e3a640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 0x7fe950197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 0x7fe95019de70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe944009230 tx=0x7fe944005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94d7fa640 1 -- 172.21.15.116:0/2762038767 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944004350 con 0x7fe950102070 2024-03-24T09:29:48.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe944009560 con 0x7fe950102070 2024-03-24T09:29:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94d7fa640 1 -- 172.21.15.116:0/2762038767 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe94402de90 con 0x7fe950102070 2024-03-24T09:29:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe94d7fa640 1 -- 172.21.15.116:0/2762038767 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944005220 con 0x7fe950102070 2024-03-24T09:29:48.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.077+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9501a47f0 con 0x7fe950102070 2024-03-24T09:29:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.079+0000 7fe94d7fa640 1 -- 172.21.15.116:0/2762038767 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe9440035f0 con 0x7fe950102070 2024-03-24T09:29:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.079+0000 7fe94d7fa640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9380a2230 0x7fe9380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94ffff640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9380a2230 0x7fe9380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94d7fa640 1 -- 172.21.15.116:0/2762038767 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe9440f0200 con 0x7fe950102070 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 Processor -- start 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 -- start start 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c62830500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c6283d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c628400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c6279e840 con 0x560c62830a40 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560c6279bd70 con 0x560c62830120 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe959480740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560c62797550 con 0x560c6283dd00 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94ffff640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9380a2230 0x7fe9380a46f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe940009b50 tx=0x7fe940005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c6283d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c6283d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33030/0 (socket says 172.21.15.116:33030) 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3006891948 learned_addr learned my addr 172.21.15.116:0/3006891948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94ffff640 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c62830500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe954e3a640 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c628400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3006891948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 msgr2=0x560c628400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c628400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3006891948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 msgr2=0x560c62830500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c62830500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3006891948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c6272cde0 con 0x560c62830a40 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe954e3a640 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c628400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.080+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c6283d7c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe9440b92c0 tx=0x7fe9440044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a39773d7ccfb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:48.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe9367fc640 1 -- 172.21.15.116:0/3006891948 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9440f0030 con 0x560c62830a40 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe9367fc640 1 -- 172.21.15.116:0/3006891948 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9440c0a40 con 0x560c62830a40 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe9367fc640 1 -- 172.21.15.116:0/3006891948 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9440038a0 con 0x560c62830a40 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 -- 172.21.15.116:0/3006891948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 msgr2=0x560c6283d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c6283d7c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe9440b92c0 tx=0x7fe9440044b0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 -- 172.21.15.116:0/3006891948 shutdown_connections 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c628400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c6283d7c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 --2- 172.21.15.116:0/3006891948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c62830500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 -- 172.21.15.116:0/3006891948 >> 172.21.15.116:0/3006891948 conn(0x560c6282ab20 msgr2=0x560c6282af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 -- 172.21.15.116:0/3006891948 shutdown_connections 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.081+0000 7fe959480740 1 -- 172.21.15.116:0/3006891948 wait complete. 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 -- start start 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 Processor -- start 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c6283ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830a40 0x560c6283b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 0x560c6283b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c6246a430 con 0x560c6283dd00 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560c62426e00 con 0x560c62830120 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560c6271c590 con 0x560c62830a40 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe959480740 10 client.0 _ll_register_callbacks cb 0x560c6278bfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 0x560c6283b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 0x560c6283b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33032/0 (socket says 172.21.15.116:33032) 2024-03-24T09:29:48.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c6283ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 -- 172.21.15.116:0/1948573268 learned_addr learned my addr 172.21.15.116:0/1948573268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830a40 0x560c6283b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 -- 172.21.15.116:0/1948573268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830a40 msgr2=0x560c6283b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830a40 0x560c6283b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 -- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 msgr2=0x560c6283ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c6283ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.082+0000 7fe954e3a640 1 -- 172.21.15.116:0/1948573268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9501a47f0 con 0x560c6283dd00 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe954e3a640 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 0x560c6283b820 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe94800efc0 tx=0x7fe9480118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835a37c922a24d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe92f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe92f7fe640 1 -- 172.21.15.116:0/1948573268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe948016070 con 0x560c6283dd00 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560c624967e0 con 0x560c6283dd00 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe92f7fe640 1 -- 172.21.15.116:0/1948573268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe948003970 con 0x560c6283dd00 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe92f7fe640 1 -- 172.21.15.116:0/1948573268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe94800f690 con 0x560c6283dd00 2024-03-24T09:29:48.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 10 client.17865 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560c62795bd0 con 0x560c6283dd00 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe92f7fe640 1 -- 172.21.15.116:0/1948573268 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560c624967e0 con 0x560c6283dd00 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe92f7fe640 1 -- 172.21.15.116:0/1948573268 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe94801a8f0 con 0x560c6283dd00 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 10 client.17865 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 10 client.17865 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 20 client.17865 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:48.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c6284dc50 0x560c62850070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 4 client.17865 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c624d3720 con 0x560c6284dc50 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.083+0000 7fe94ffff640 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c6284dc50 0x560c62850070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.084+0000 7fe94ffff640 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c6284dc50 0x560c62850070 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5527915f5cd5420d server_cookie=b1c5f71e30b08c36 in_seq=0 out_seq=0 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.084+0000 7fe92f7fe640 10 client.17865 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe92f7fe640 1 -- 172.21.15.116:0/1948573268 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c624d3720 con 0x560c6284dc50 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe92f7fe640 10 client.17865 handle_command_reply: tid=0 2024-03-24T09:29:48.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 client.17865 shutdown 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 msgr2=0x560c6283b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 0x560c6283b820 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe94800efc0 tx=0x7fe9480118d0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 shutdown_connections 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c6283dd00 0x560c6283b820 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830a40 0x560c6283b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c6284dc50 0x560c62850070 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 --2- 172.21.15.116:0/1948573268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830120 0x560c6283ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe92f7fe640 0 client.17865 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:48.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 >> 172.21.15.116:0/1948573268 conn(0x560c6282ab20 msgr2=0x560c6282c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:48.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 shutdown_connections 2024-03-24T09:29:48.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 1 -- 172.21.15.116:0/1948573268 wait complete. 2024-03-24T09:29:48.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.085+0000 7fe959480740 20 client.17865 trim_cache size 0 max 16384 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 Processor -- start 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 -- start start 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 0x560c6283d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830a40 0x560c62829430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c6283b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c624d3720 con 0x560c62830120 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560c628369d0 con 0x560c62830a40 2024-03-24T09:29:48.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe959480740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560c62829970 con 0x560c6283dd00 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 0x560c6283d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 0x560c6283d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33034/0 (socket says 172.21.15.116:33034) 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe94ffff640 1 -- 172.21.15.116:0/1924261057 learned_addr learned my addr 172.21.15.116:0/1924261057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830a40 0x560c62829430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.098+0000 7fe954e3a640 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c6283b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe94ffff640 1 -- 172.21.15.116:0/1924261057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 msgr2=0x560c6283b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe94ffff640 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c6283b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe94ffff640 1 -- 172.21.15.116:0/1924261057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830a40 msgr2=0x560c62829430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe94ffff640 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830a40 0x560c62829430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe94ffff640 1 -- 172.21.15.116:0/1924261057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c62731c80 con 0x560c62830120 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe954e3a640 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c6283b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe94ffff640 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 0x560c6283d4d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe940009a10 tx=0x7fe940014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda4d259e94c4fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe935ffb640 1 -- 172.21.15.116:0/1924261057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94002e070 con 0x560c62830120 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe935ffb640 1 -- 172.21.15.116:0/1924261057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe94002ab90 con 0x560c62830120 2024-03-24T09:29:48.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe935ffb640 1 -- 172.21.15.116:0/1924261057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940024040 con 0x560c62830120 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 -- 172.21.15.116:0/1924261057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 msgr2=0x560c6283d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 0x560c6283d4d0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe940009a10 tx=0x7fe940014000 comp rx=0 tx=0).stop 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 -- 172.21.15.116:0/1924261057 shutdown_connections 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c6283dd00 0x560c6283b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c62830a40 0x560c62829430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 --2- 172.21.15.116:0/1924261057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830120 0x560c6283d4d0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.099+0000 7fe959480740 1 -- 172.21.15.116:0/1924261057 >> 172.21.15.116:0/1924261057 conn(0x560c6282ab20 msgr2=0x560c6282c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 -- 172.21.15.116:0/1924261057 shutdown_connections 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 -- 172.21.15.116:0/1924261057 wait complete. 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 -- start start 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 Processor -- start 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830120 0x560c628ac220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c628ac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c6283dd00 0x560c62841b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c62842100 con 0x560c62830a40 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560c62842270 con 0x560c6283dd00 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560c62842430 con 0x560c62830120 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe959480740 10 client.0 _ll_register_callbacks cb 0x7fe948009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c628ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe94f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c628ac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33048/0 (socket says 172.21.15.116:33048) 2024-03-24T09:29:48.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe94ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830120 0x560c628ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3640036821 learned_addr learned my addr 172.21.15.116:0/3640036821 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.100+0000 7fe954e3a640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c6283dd00 0x560c62841b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3640036821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830120 msgr2=0x560c628ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830120 0x560c628ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c6283dd00 msgr2=0x560c62841b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c6283dd00 0x560c62841b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94f7fe640 1 -- 172.21.15.116:0/3640036821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe940017070 con 0x560c62830a40 2024-03-24T09:29:48.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94ffff640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830120 0x560c628ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe954e3a640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c6283dd00 0x560c62841b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe94f7fe640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c628ac760 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe9440bc880 tx=0x7fe9440bc740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11b2487f58d9eb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe92f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe92f7fe640 1 -- 172.21.15.116:0/3640036821 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe944003aa0 con 0x560c62830a40 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe92f7fe640 1 -- 172.21.15.116:0/3640036821 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7fe9440f0040 con 0x560c62830a40 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9440c0a40 con 0x560c62830a40 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe92f7fe640 1 -- 172.21.15.116:0/3640036821 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe9440bc8f0 con 0x560c62830a40 2024-03-24T09:29:48.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 10 client.17871 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560c62757920 con 0x560c62830a40 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe92f7fe640 1 -- 172.21.15.116:0/3640036821 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe9440c0a40 con 0x560c62830a40 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe92f7fe640 1 -- 172.21.15.116:0/3640036821 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe94403c070 con 0x560c62830a40 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 10 client.17871 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 10 client.17871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 20 client.17871 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c628b4ef0 0x560c628b7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:48.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 4 client.17871 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.101+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c62522890 con 0x560c628b4ef0 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe94ffff640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c628b4ef0 0x560c628b7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe92f7fe640 10 client.17871 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe94ffff640 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c628b4ef0 0x560c628b7310 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46ab2ccb85686469 server_cookie=227ae6425b5afbbe in_seq=0 out_seq=0 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe92f7fe640 1 -- 172.21.15.116:0/3640036821 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c62522890 con 0x560c628b4ef0 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe92f7fe640 10 client.17871 handle_command_reply: tid=0 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 client.17871 shutdown 2024-03-24T09:29:48.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 msgr2=0x560c628ac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c628ac760 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe9440bc880 tx=0x7fe9440bc740 comp rx=0 tx=0).stop 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 shutdown_connections 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x560c628b4ef0 0x560c628b7310 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560c6283dd00 0x560c62841b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c62830a40 0x560c628ac760 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 --2- 172.21.15.116:0/3640036821 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x560c62830120 0x560c628ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe92f7fe640 0 client.17871 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:48.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 >> 172.21.15.116:0/3640036821 conn(0x560c6282ab20 msgr2=0x560c6283c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:48.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 shutdown_connections 2024-03-24T09:29:48.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 1 -- 172.21.15.116:0/3640036821 wait complete. 2024-03-24T09:29:48.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.102+0000 7fe959480740 20 client.17871 trim_cache size 0 max 16384 2024-03-24T09:29:48.120 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:29:48.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9380a2230 msgr2=0x7fe9380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9380a2230 0x7fe9380a46f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe940009b50 tx=0x7fe940005f90 comp rx=0 tx=0).stop 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 msgr2=0x7fe95019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 0x7fe95019de70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe944009230 tx=0x7fe944005dc0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 shutdown_connections 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9501076c0 0x7fe950197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe950102070 0x7fe95019de70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe9380a2230 0x7fe9380a46f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 --2- 172.21.15.116:0/2762038767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9501016b0 0x7fe95019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:48.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 >> 172.21.15.116:0/2762038767 conn(0x7fe9500fd140 msgr2=0x7fe9500fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:48.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 shutdown_connections 2024-03-24T09:29:48.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:48.103+0000 7fe95689d640 1 -- 172.21.15.116:0/2762038767 wait complete. 2024-03-24T09:29:48.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:48 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3345026521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:29:48.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:48 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3345026521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:29:48.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:48 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3345026521' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:29:49.124 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:29:49.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:29:49.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:29:49.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 -- 172.21.15.116:0/3136722309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470105b10 msgr2=0x7f0470105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 --2- 172.21.15.116:0/3136722309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470105b10 0x7f0470105fc0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0468009cd0 tx=0x7f046802c180 comp rx=0 tx=0).stop 2024-03-24T09:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 -- 172.21.15.116:0/3136722309 shutdown_connections 2024-03-24T09:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 --2- 172.21.15.116:0/3136722309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470105b10 0x7f0470105fc0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 --2- 172.21.15.116:0/3136722309 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470101030 0x7f04701054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 --2- 172.21.15.116:0/3136722309 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0470100710 0x7f0470100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 -- 172.21.15.116:0/3136722309 >> 172.21.15.116:0/3136722309 conn(0x7f0470076990 msgr2=0x7f0470076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 -- 172.21.15.116:0/3136722309 shutdown_connections 2024-03-24T09:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.242+0000 7f0477563640 1 -- 172.21.15.116:0/3136722309 wait complete. 2024-03-24T09:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 Processor -- start 2024-03-24T09:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 -- start start 2024-03-24T09:29:49.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0470100710 0x7f047019e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 0x7f047019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470105b10 0x7f04701a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04701a30f0 con 0x7f0470101030 2024-03-24T09:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f04701a3260 con 0x7f0470105b10 2024-03-24T09:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0477563640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f04701a3420 con 0x7f0470100710 2024-03-24T09:29:49.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0474afe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 0x7f047019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0475b00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470105b10 0x7f04701a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0474afe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 0x7f047019e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33064/0 (socket says 172.21.15.116:33064) 2024-03-24T09:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f04752ff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0470100710 0x7f047019e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.243+0000 7f0474afe640 1 -- 172.21.15.116:0/3996530872 learned_addr learned my addr 172.21.15.116:0/3996530872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0474afe640 1 -- 172.21.15.116:0/3996530872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0470100710 msgr2=0x7f047019e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0474afe640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0470100710 0x7f047019e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0474afe640 1 -- 172.21.15.116:0/3996530872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470105b10 msgr2=0x7f04701a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0474afe640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470105b10 0x7f04701a2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0474afe640 1 -- 172.21.15.116:0/3996530872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0468009900 con 0x7f0470101030 2024-03-24T09:29:49.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0475b00640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470105b10 0x7f04701a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:49.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0474afe640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 0x7f047019e770 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f046000e9f0 tx=0x7f046000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f04667fc640 1 -- 172.21.15.116:0/3996530872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460009700 con 0x7f0470101030 2024-03-24T09:29:49.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f04667fc640 1 -- 172.21.15.116:0/3996530872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f046000cd80 con 0x7f0470101030 2024-03-24T09:29:49.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04701a3700 con 0x7f0470101030 2024-03-24T09:29:49.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f04667fc640 1 -- 172.21.15.116:0/3996530872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460018840 con 0x7f0470101030 2024-03-24T09:29:49.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.244+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04701aaf80 con 0x7f0470101030 2024-03-24T09:29:49.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.246+0000 7f04667fc640 1 -- 172.21.15.116:0/3996530872 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f04600189a0 con 0x7f0470101030 2024-03-24T09:29:49.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 Processor -- start 2024-03-24T09:29:49.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04667fc640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f045c0a22b0 0x7f045c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 -- start start 2024-03-24T09:29:49.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863cd760 0x5646863cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f0475b00640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f045c0a22b0 0x7f045c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863d6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04667fc640 1 -- 172.21.15.116:0/3996530872 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0460014070 con 0x7f0470101030 2024-03-24T09:29:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56468633b530 con 0x5646863d6c60 2024-03-24T09:29:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564686338a60 con 0x5646863cd760 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f047a217740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564686334240 con 0x5646863ce080 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f0475b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04752ff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863cd760 0x5646863cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f0474afe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863d6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f0475b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33068/0 (socket says 172.21.15.116:33068) 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f0475b00640 1 -- 172.21.15.116:0/3944671683 learned_addr learned my addr 172.21.15.116:0/3944671683 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04752ff640 1 -- 172.21.15.116:0/3944671683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 msgr2=0x5646863d6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04752ff640 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863d6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04752ff640 1 -- 172.21.15.116:0/3944671683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 msgr2=0x5646863d9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04752ff640 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f04752ff640 1 -- 172.21.15.116:0/3944671683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646863328c0 con 0x5646863cd760 2024-03-24T09:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.247+0000 7f0475b00640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f045c0a22b0 0x7f045c0a4770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f04680345e0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.248+0000 7f04752ff640 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863cd760 0x5646863cdb40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f046c009c00 tx=0x7f046c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e4f24e1e2d502c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.248+0000 7f045b7fe640 1 -- 172.21.15.116:0/3944671683 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c004230 con 0x5646863cd760 2024-03-24T09:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.248+0000 7f045b7fe640 1 -- 172.21.15.116:0/3944671683 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f046c009e70 con 0x5646863cd760 2024-03-24T09:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.248+0000 7f045b7fe640 1 -- 172.21.15.116:0/3944671683 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c004b90 con 0x5646863cd760 2024-03-24T09:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- 172.21.15.116:0/3944671683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863cd760 msgr2=0x5646863cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863cd760 0x5646863cdb40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f046c009c00 tx=0x7f046c011810 comp rx=0 tx=0).stop 2024-03-24T09:29:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- 172.21.15.116:0/3944671683 shutdown_connections 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863d6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- 172.21.15.116:0/3944671683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863cd760 0x5646863cdb40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- 172.21.15.116:0/3944671683 >> 172.21.15.116:0/3944671683 conn(0x5646863c7e90 msgr2=0x5646863c8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- 172.21.15.116:0/3944671683 shutdown_connections 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- 172.21.15.116:0/3944671683 wait complete. 2024-03-24T09:29:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- start start 2024-03-24T09:29:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 Processor -- start 2024-03-24T09:29:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863d3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5646862d8f80 con 0x5646863d6c60 2024-03-24T09:29:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5646860074b0 con 0x5646863ce080 2024-03-24T09:29:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.249+0000 7f047a217740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5646862b9ad0 con 0x5646863cd760 2024-03-24T09:29:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f047a217740 10 client.0 _ll_register_callbacks cb 0x5646862ecb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f04752ff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863d3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0474afe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33074/0 (socket says 172.21.15.116:33074) 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f04752ff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863d3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:34438/0 (socket says 172.21.15.116:34438) 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 -- 172.21.15.116:0/1498088294 learned_addr learned my addr 172.21.15.116:0/1498088294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 -- 172.21.15.116:0/1498088294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 msgr2=0x5646863d3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863d3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 -- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 msgr2=0x5646863d4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 -- 172.21.15.116:0/1498088294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046c009830 con 0x5646863d6c60 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0474afe640 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0475b00640 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d4640 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0468036000 tx=0x7f046802c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104cf96119dd0b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0458ff9640 1 -- 172.21.15.116:0/1498088294 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468032a90 con 0x5646863d6c60 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0458ff9640 1 -- 172.21.15.116:0/1498088294 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f0468034870 con 0x5646863d6c60 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5646863587a0 con 0x5646863d6c60 2024-03-24T09:29:49.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f0458ff9640 1 -- 172.21.15.116:0/1498088294 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f0468032bf0 con 0x5646863d6c60 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f047a217740 10 client.17883 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.250+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5646862ea6d0 con 0x5646863d6c60 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f0458ff9640 1 -- 172.21.15.116:0/1498088294 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5646863587a0 con 0x5646863d6c60 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f0458ff9640 1 -- 172.21.15.116:0/1498088294 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f0468053020 con 0x5646863d6c60 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f047a217740 10 client.17883 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f047a217740 10 client.17883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f047a217740 20 client.17883 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f047a217740 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5646863e6d20 0x5646863e9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f047a217740 4 client.17883 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56468632a1f0 con 0x5646863e6d20 2024-03-24T09:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f04752ff640 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5646863e6d20 0x5646863e9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f0458ff9640 10 client.17883 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.251+0000 7f04752ff640 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5646863e6d20 0x5646863e9140 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=793bf3e8b92680f4 server_cookie=5659b497dd595b1c in_seq=0 out_seq=0 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f0458ff9640 1 -- 172.21.15.116:0/1498088294 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56468632a1f0 con 0x5646863e6d20 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f0458ff9640 10 client.17883 handle_command_reply: tid=0 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 client.17883 shutdown 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 msgr2=0x5646863d4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d4640 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0468036000 tx=0x7f046802c180 comp rx=0 tx=0).stop 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 shutdown_connections 2024-03-24T09:29:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863d4640 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5646863e6d20 0x5646863e9140 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f0458ff9640 0 client.17883 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 --2- 172.21.15.116:0/1498088294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863d3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 >> 172.21.15.116:0/1498088294 conn(0x5646863c7e90 msgr2=0x5646863c9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 shutdown_connections 2024-03-24T09:29:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 1 -- 172.21.15.116:0/1498088294 wait complete. 2024-03-24T09:29:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.252+0000 7f047a217740 20 client.17883 trim_cache size 0 max 16384 2024-03-24T09:29:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.269+0000 7f047a217740 1 Processor -- start 2024-03-24T09:29:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.269+0000 7f047a217740 1 -- start start 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f047a217740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863e2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f047a217740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f047a217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863eea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f047a217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5646863c8e90 con 0x5646863d6c60 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f047a217740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5646863e97a0 con 0x5646863ce080 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f047a217740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5646863d2110 con 0x5646863cd760 2024-03-24T09:29:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863eea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863eea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33086/0 (socket says 172.21.15.116:33086) 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 -- 172.21.15.116:0/3239316829 learned_addr learned my addr 172.21.15.116:0/3239316829 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0474afe640 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f04752ff640 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863e2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 -- 172.21.15.116:0/3239316829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 msgr2=0x5646863e2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863e2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 -- 172.21.15.116:0/3239316829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 msgr2=0x5646863d4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 -- 172.21.15.116:0/3239316829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646862cec60 con 0x5646863d6c60 2024-03-24T09:29:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f0475b00640 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863eea30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f046802c6c0 tx=0x7f04680095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816c3ec24d69ba60 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f045affd640 1 -- 172.21.15.116:0/3239316829 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046804d820 con 0x5646863d6c60 2024-03-24T09:29:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f045affd640 1 -- 172.21.15.116:0/3239316829 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04680324a0 con 0x5646863d6c60 2024-03-24T09:29:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.270+0000 7f045affd640 1 -- 172.21.15.116:0/3239316829 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468003e50 con 0x5646863d6c60 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 -- 172.21.15.116:0/3239316829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 msgr2=0x5646863eea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863eea30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f046802c6c0 tx=0x7f04680095c0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 -- 172.21.15.116:0/3239316829 shutdown_connections 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863d6c60 0x5646863eea30 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863ce080 0x5646863d4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 --2- 172.21.15.116:0/3239316829 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863cd760 0x5646863e2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 -- 172.21.15.116:0/3239316829 >> 172.21.15.116:0/3239316829 conn(0x5646863c7e90 msgr2=0x5646863c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 -- 172.21.15.116:0/3239316829 shutdown_connections 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 -- 172.21.15.116:0/3239316829 wait complete. 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 -- start start 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.271+0000 7f047a217740 1 Processor -- start 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 0x5646863de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863ded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863d6c60 0x5646863dac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5646863db2d0 con 0x5646863cd760 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5646863db440 con 0x5646863d6c60 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5646863db600 con 0x5646863ce080 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 10 client.0 _ll_register_callbacks cb 0x7f0468003ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 0x5646863de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 0x5646863de7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33100/0 (socket says 172.21.15.116:33100) 2024-03-24T09:29:49.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 -- 172.21.15.116:0/3782590141 learned_addr learned my addr 172.21.15.116:0/3782590141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f0474afe640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f0475b00640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863d6c60 0x5646863dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 -- 172.21.15.116:0/3782590141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 msgr2=0x5646863ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863ded30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 -- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863d6c60 msgr2=0x5646863dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863d6c60 0x5646863dac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 -- 172.21.15.116:0/3782590141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646862ea6d0 con 0x5646863cd760 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f0475b00640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863d6c60 0x5646863dac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f04752ff640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 0x5646863de7f0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f046c011d10 tx=0x7f046c009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345e2f9855958499 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f0458ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f0458ff9640 1 -- 172.21.15.116:0/3782590141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c005240 con 0x5646863cd760 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f0458ff9640 1 -- 172.21.15.116:0/3782590141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f046c0053a0 con 0x5646863cd760 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.272+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f046c009e70 con 0x5646863cd760 2024-03-24T09:29:49.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f0458ff9640 1 -- 172.21.15.116:0/3782590141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f046c005560 con 0x5646863cd760 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 10 client.17889 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5646862f4970 con 0x5646863cd760 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f0458ff9640 1 -- 172.21.15.116:0/3782590141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f046c009e70 con 0x5646863cd760 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f0458ff9640 1 -- 172.21.15.116:0/3782590141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f046c003af0 con 0x5646863cd760 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 10 client.17889 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:49.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 20 client.17889 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56468644e1d0 0x5646864505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 4 client.17889 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564686033860 con 0x56468644e1d0 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f0474afe640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56468644e1d0 0x5646864505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f0458ff9640 10 client.17889 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.273+0000 7f0474afe640 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56468644e1d0 0x5646864505f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8db95baed1b5793 server_cookie=12b6bed86c7c48ee in_seq=0 out_seq=0 2024-03-24T09:29:49.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f0458ff9640 1 -- 172.21.15.116:0/3782590141 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564686033860 con 0x56468644e1d0 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f0458ff9640 10 client.17889 handle_command_reply: tid=0 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 client.17889 shutdown 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 msgr2=0x5646863de7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 0x5646863de7f0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f046c011d10 tx=0x7f046c009620 comp rx=0 tx=0).stop 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 shutdown_connections 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56468644e1d0 0x5646864505f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f0458ff9640 0 client.17889 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5646863d6c60 0x5646863dac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.274+0000 7f047a217740 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5646863ce080 0x5646863ded30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f047a217740 1 --2- 172.21.15.116:0/3782590141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5646863cd760 0x5646863de7f0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 >> 172.21.15.116:0/3782590141 conn(0x5646863c7e90 msgr2=0x5646863c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 shutdown_connections 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f047a217740 1 -- 172.21.15.116:0/3782590141 wait complete. 2024-03-24T09:29:49.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f047a217740 20 client.17889 trim_cache size 0 max 16384 2024-03-24T09:29:49.292 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:29:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:29:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "e457fa4c-996a-401f-973e-1f79cdbfba9e", 2024-03-24T09:29:49.292 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:29:49.292 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:29:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f045c0a22b0 msgr2=0x7f045c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f045c0a22b0 0x7f045c0a4770 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f04680345e0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 msgr2=0x7f047019e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 0x7f047019e770 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f046000e9f0 tx=0x7f046000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 shutdown_connections 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0470105b10 0x7f04701a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470101030 0x7f047019e770 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f045c0a22b0 0x7f045c0a4770 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 --2- 172.21.15.116:0/3996530872 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0470100710 0x7f047019e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 >> 172.21.15.116:0/3996530872 conn(0x7f0470076990 msgr2=0x7f04701034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 shutdown_connections 2024-03-24T09:29:49.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.275+0000 7f0477563640 1 -- 172.21.15.116:0/3996530872 wait complete. 2024-03-24T09:29:49.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:29:49.296 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:29:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:49 smithi117 ceph-mon[35788]: pgmap v236: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:49.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:49 smithi116 ceph-mon[32207]: pgmap v236: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:49.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 -- 172.21.15.116:0/1995045533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c0fff90 msgr2=0x7f407c100370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 --2- 172.21.15.116:0/1995045533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c0fff90 0x7f407c100370 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f406c009900 tx=0x7f406c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 -- 172.21.15.116:0/1995045533 shutdown_connections 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 --2- 172.21.15.116:0/1995045533 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c105060 0x7f407c10cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 --2- 172.21.15.116:0/1995045533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f407c100940 0x7f407c104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 --2- 172.21.15.116:0/1995045533 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c0fff90 0x7f407c100370 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 -- 172.21.15.116:0/1995045533 >> 172.21.15.116:0/1995045533 conn(0x7f407c076b30 msgr2=0x7f407c0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 -- 172.21.15.116:0/1995045533 shutdown_connections 2024-03-24T09:29:49.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 -- 172.21.15.116:0/1995045533 wait complete. 2024-03-24T09:29:49.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 Processor -- start 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.419+0000 7f40823f4640 1 -- start start 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f40823f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c0fff90 0x7f407c19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f40823f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 0x7f407c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f40823f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f407c196a20 0x7f407c196ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f40823f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f407c197410 con 0x7f407c105060 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f40823f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f407c197580 con 0x7f407c196a20 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f40823f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f407c197740 con 0x7f407c0fff90 2024-03-24T09:29:49.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 0x7f407c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 0x7f407c19cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33126/0 (socket says 172.21.15.116:33126) 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 -- 172.21.15.116:0/429869479 learned_addr learned my addr 172.21.15.116:0/429869479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f4080991640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f407c196a20 0x7f407c196ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 -- 172.21.15.116:0/429869479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c0fff90 msgr2=0x7f407c19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407bfff640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c0fff90 0x7f407c19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c0fff90 0x7f407c19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 -- 172.21.15.116:0/429869479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f407c196a20 msgr2=0x7f407c196ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f407c196a20 0x7f407c196ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407b7fe640 1 -- 172.21.15.116:0/429869479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406c009560 con 0x7f407c105060 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.420+0000 7f407bfff640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c0fff90 0x7f407c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.421+0000 7f407b7fe640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 0x7f407c19cdb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9c0 tx=0x7f407000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.421+0000 7f40797fa640 1 -- 172.21.15.116:0/429869479 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070009700 con 0x7f407c105060 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.421+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407c1a3730 con 0x7f407c105060 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.421+0000 7f40797fa640 1 -- 172.21.15.116:0/429869479 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f407000cd80 con 0x7f407c105060 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.421+0000 7f40797fa640 1 -- 172.21.15.116:0/429869479 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070010430 con 0x7f407c105060 2024-03-24T09:29:49.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.421+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f407c1a3c40 con 0x7f407c105060 2024-03-24T09:29:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.423+0000 7f4084fd5740 1 Processor -- start 2024-03-24T09:29:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.423+0000 7f4084fd5740 1 -- start start 2024-03-24T09:29:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f40797fa640 1 -- 172.21.15.116:0/429869479 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f4070004f50 con 0x7f407c105060 2024-03-24T09:29:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 0x55bf97ef42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef47e0 0x55bf97eff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97eff8e0 0x55bf97f01cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4084fd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf97e18f30 con 0x55bf97ef3ec0 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4084fd5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf97e62810 con 0x55bf97ef47e0 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4084fd5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf97e5fd40 con 0x55bf97eff8e0 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 0x55bf97ef42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 0x55bf97ef42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33128/0 (socket says 172.21.15.116:33128) 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f4080991640 1 -- 172.21.15.116:0/3724611041 learned_addr learned my addr 172.21.15.116:0/3724611041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f407bfff640 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef47e0 0x55bf97eff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.424+0000 7f40797fa640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f40640a22b0 0x7f40640a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 -- 172.21.15.116:0/3724611041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97eff8e0 msgr2=0x55bf97f01cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97eff8e0 0x55bf97f01cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 -- 172.21.15.116:0/3724611041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef47e0 msgr2=0x55bf97eff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef47e0 0x55bf97eff3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 -- 172.21.15.116:0/3724611041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf97e1cea0 con 0x55bf97ef3ec0 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f40640a22b0 0x7f40640a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f4080991640 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 0x55bf97ef42a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f40740159a0 tx=0x7f4074015e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539ef495831c9016 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f40797fa640 1 -- 172.21.15.116:0/429869479 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f4070014070 con 0x7f407c105060 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f40627fc640 1 -- 172.21.15.116:0/3724611041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407400fe40 con 0x55bf97ef3ec0 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f40627fc640 1 -- 172.21.15.116:0/3724611041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f407401f8b0 con 0x55bf97ef3ec0 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.425+0000 7f40627fc640 1 -- 172.21.15.116:0/3724611041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407400ea90 con 0x55bf97ef3ec0 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 -- 172.21.15.116:0/3724611041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 msgr2=0x55bf97ef42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4080991640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f40640a22b0 0x7f40640a4770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4074000f80 tx=0x7f407400ebf0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 0x55bf97ef42a0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f40740159a0 tx=0x7f4074015e60 comp rx=0 tx=0).stop 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 -- 172.21.15.116:0/3724611041 shutdown_connections 2024-03-24T09:29:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97eff8e0 0x55bf97f01cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef47e0 0x55bf97eff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3724611041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97ef3ec0 0x55bf97ef42a0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 -- 172.21.15.116:0/3724611041 >> 172.21.15.116:0/3724611041 conn(0x55bf97eee900 msgr2=0x55bf97eeece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 -- 172.21.15.116:0/3724611041 shutdown_connections 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.426+0000 7f4084fd5740 1 -- 172.21.15.116:0/3724611041 wait complete. 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 -- start start 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 Processor -- start 2024-03-24T09:29:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97efaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97efafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97efb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf97e5b520 con 0x55bf97eff8e0 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf97b2e430 con 0x55bf97ef3ec0 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf97aeae00 con 0x55bf97ef47e0 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4084fd5740 10 client.0 _ll_register_callbacks cb 0x55bf97df9d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97efb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97efb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33138/0 (socket says 172.21.15.116:33138) 2024-03-24T09:29:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f4080991640 1 -- 172.21.15.116:0/3476173115 learned_addr learned my addr 172.21.15.116:0/3476173115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f407bfff640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97efaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.427+0000 7f407b7fe640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97efafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4080991640 1 -- 172.21.15.116:0/3476173115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 msgr2=0x55bf97efafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4080991640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97efafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4080991640 1 -- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 msgr2=0x55bf97efaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4080991640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97efaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4080991640 1 -- 172.21.15.116:0/3476173115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4074009c50 con 0x55bf97eff8e0 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f407bfff640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97efaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:29:49.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4080991640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97efb520 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f407400e3d0 tx=0x7f407402a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7799ef0c57d4420a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f405b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f405b7fe640 1 -- 172.21.15.116:0/3476173115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407400e450 con 0x55bf97eff8e0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f405b7fe640 1 -- 172.21.15.116:0/3476173115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f407401fa20 con 0x55bf97eff8e0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bf97b5a7e0 con 0x55bf97eff8e0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f405b7fe640 1 -- 172.21.15.116:0/3476173115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4074009ed0 con 0x55bf97eff8e0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4084fd5740 10 client.17901 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.428+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bf97e59ba0 con 0x55bf97eff8e0 2024-03-24T09:29:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f405b7fe640 1 -- 172.21.15.116:0/3476173115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bf97b5a7e0 con 0x55bf97eff8e0 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f405b7fe640 1 -- 172.21.15.116:0/3476173115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f407402f040 con 0x55bf97eff8e0 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f4084fd5740 10 client.17901 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f4084fd5740 10 client.17901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f4084fd5740 20 client.17901 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f0db40 0x55bf97f0ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f4084fd5740 4 client.17901 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf97b97720 con 0x55bf97f0db40 2024-03-24T09:29:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.429+0000 7f407bfff640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f0db40 0x55bf97f0ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.430+0000 7f407bfff640 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f0db40 0x55bf97f0ff60 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81652e9eae1ce596 server_cookie=80647b99884d961d in_seq=0 out_seq=0 2024-03-24T09:29:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.430+0000 7f405b7fe640 10 client.17901 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f405b7fe640 1 -- 172.21.15.116:0/3476173115 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf97b97720 con 0x55bf97f0db40 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f405b7fe640 10 client.17901 handle_command_reply: tid=0 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 client.17901 shutdown 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 msgr2=0x55bf97efb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97efb520 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f407400e3d0 tx=0x7f407402a070 comp rx=0 tx=0).stop 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 shutdown_connections 2024-03-24T09:29:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97efb520 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97efafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f0db40 0x55bf97f0ff60 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3476173115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97efaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 >> 172.21.15.116:0/3476173115 conn(0x55bf97eee900 msgr2=0x55bf97ef0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f405b7fe640 0 client.17901 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 shutdown_connections 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.431+0000 7f4084fd5740 1 -- 172.21.15.116:0/3476173115 wait complete. 2024-03-24T09:29:49.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.432+0000 7f4084fd5740 20 client.17901 trim_cache size 0 max 16384 2024-03-24T09:29:49.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 Processor -- start 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 -- start start 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97eff1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97eedb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f15830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf97de0430 con 0x55bf97eff8e0 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf97f0a680 con 0x55bf97ef3ec0 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4084fd5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf97ef3b00 con 0x55bf97ef47e0 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f15830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f15830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33144/0 (socket says 172.21.15.116:33144) 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f4080991640 1 -- 172.21.15.116:0/3536268007 learned_addr learned my addr 172.21.15.116:0/3536268007 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f407bfff640 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97eff1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.455+0000 7f407b7fe640 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97eedb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4080991640 1 -- 172.21.15.116:0/3536268007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 msgr2=0x55bf97eedb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4080991640 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97eedb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4080991640 1 -- 172.21.15.116:0/3536268007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 msgr2=0x55bf97eff1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4080991640 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97eff1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4080991640 1 -- 172.21.15.116:0/3536268007 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf97df5b20 con 0x55bf97eff8e0 2024-03-24T09:29:49.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4080991640 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f15830 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4074018040 tx=0x7f4074015e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5f9da6b5506815 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4061ffb640 1 -- 172.21.15.116:0/3536268007 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074027cb0 con 0x55bf97eff8e0 2024-03-24T09:29:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4061ffb640 1 -- 172.21.15.116:0/3536268007 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4074019040 con 0x55bf97eff8e0 2024-03-24T09:29:49.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.456+0000 7f4061ffb640 1 -- 172.21.15.116:0/3536268007 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407401fa20 con 0x55bf97eff8e0 2024-03-24T09:29:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 -- 172.21.15.116:0/3536268007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 msgr2=0x55bf97f15830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f15830 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f4074018040 tx=0x7f4074015e60 comp rx=0 tx=0).stop 2024-03-24T09:29:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 -- 172.21.15.116:0/3536268007 shutdown_connections 2024-03-24T09:29:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f15830 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97eedb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3536268007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97eff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 -- 172.21.15.116:0/3536268007 >> 172.21.15.116:0/3536268007 conn(0x55bf97eee900 msgr2=0x55bf97eef900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 -- 172.21.15.116:0/3536268007 shutdown_connections 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 -- 172.21.15.116:0/3536268007 wait complete. 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.457+0000 7f4084fd5740 1 -- start start 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 Processor -- start 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97f08e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97f04180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf97f04dc0 con 0x55bf97eff8e0 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf97f04f30 con 0x55bf97ef3ec0 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf97f7b670 con 0x55bf97ef47e0 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4084fd5740 10 client.0 _ll_register_callbacks cb 0x7f4070005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f407bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97f08e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33156/0 (socket says 172.21.15.116:33156) 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f407bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97f08e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:34128/0 (socket says 172.21.15.116:34128) 2024-03-24T09:29:49.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 -- 172.21.15.116:0/3142076060 learned_addr learned my addr 172.21.15.116:0/3142076060 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 -- 172.21.15.116:0/3142076060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 msgr2=0x55bf97f04180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97f04180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 -- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 msgr2=0x55bf97f08e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97f08e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.458+0000 7f4080991640 1 -- 172.21.15.116:0/3142076060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf97e59ba0 con 0x55bf97eff8e0 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f4080991640 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f046c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4074009560 tx=0x7f4074005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318758bd138913bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f405b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f405b7fe640 1 -- 172.21.15.116:0/3142076060 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407400ea50 con 0x55bf97eff8e0 2024-03-24T09:29:49.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f405b7fe640 1 -- 172.21.15.116:0/3142076060 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9554+0+0 (secure 0 0 0) 0x7f407401fa20 con 0x55bf97eff8e0 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f407401f8b0 con 0x55bf97eff8e0 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f405b7fe640 1 -- 172.21.15.116:0/3142076060 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f4074009eb0 con 0x55bf97eff8e0 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f4084fd5740 10 client.17907 fetch_fsmap learned FSMap version 35 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bf97e1b700 con 0x55bf97eff8e0 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.459+0000 7f405b7fe640 1 -- 172.21.15.116:0/3142076060 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f407401f8b0 con 0x55bf97eff8e0 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f405b7fe640 1 -- 172.21.15.116:0/3142076060 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f4074015020 con 0x55bf97eff8e0 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f4084fd5740 10 client.17907 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:29:49.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f4084fd5740 10 client.17907 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f4084fd5740 20 client.17907 populate_metadata read hostname 'smithi116' 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f7c990 0x55bf97f7edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f4084fd5740 4 client.17907 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bf97be6890 con 0x55bf97f7c990 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.460+0000 7f407bfff640 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f7c990 0x55bf97f7edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f407bfff640 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f7c990 0x55bf97f7edb0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8b6c3916aebcdd0 server_cookie=7678ddd1987414d3 in_seq=0 out_seq=0 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f405b7fe640 10 client.17907 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f405b7fe640 1 -- 172.21.15.116:0/3142076060 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bf97be6890 con 0x55bf97f7c990 2024-03-24T09:29:49.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f405b7fe640 10 client.17907 handle_command_reply: tid=0 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 client.17907 shutdown 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 msgr2=0x55bf97f046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f046c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4074009560 tx=0x7f4074005cf0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 shutdown_connections 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf97eff8e0 0x55bf97f046c0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf97ef47e0 0x55bf97f04180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf97f7c990 0x55bf97f7edb0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 --2- 172.21.15.116:0/3142076060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf97ef3ec0 0x55bf97f08e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 >> 172.21.15.116:0/3142076060 conn(0x55bf97eee900 msgr2=0x55bf97ef0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f405b7fe640 0 client.17907 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:29:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 shutdown_connections 2024-03-24T09:29:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 1 -- 172.21.15.116:0/3142076060 wait complete. 2024-03-24T09:29:49.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.461+0000 7f4084fd5740 20 client.17907 trim_cache size 0 max 16384 2024-03-24T09:29:49.470 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:29:49.470 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:29:49.470 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout: "e457fa4c-996a-401f-973e-1f79cdbfba9e": { 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "e457fa4c-996a-401f-973e-1f79cdbfba9e", 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:29:49.471 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:29:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f40640a22b0 msgr2=0x7f40640a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f40640a22b0 0x7f40640a4770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4074000f80 tx=0x7f407400ebf0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 msgr2=0x7f407c19cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:29:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 0x7f407c19cdb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f407000e9c0 tx=0x7f407000ee80 comp rx=0 tx=0).stop 2024-03-24T09:29:49.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 shutdown_connections 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f407c196a20 0x7f407c196ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f407c105060 0x7f407c19cdb0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f40640a22b0 0x7f40640a4770 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 --2- 172.21.15.116:0/429869479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f407c0fff90 0x7f407c19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 >> 172.21.15.116:0/429869479 conn(0x7f407c076b30 msgr2=0x7f407c1034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 shutdown_connections 2024-03-24T09:29:49.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:29:49.462+0000 7f40823f4640 1 -- 172.21.15.116:0/429869479 wait complete. 2024-03-24T09:29:49.475 INFO:tasks.cephfs.filesystem:scrub status for tag:e457fa4c-996a-401f-973e-1f79cdbfba9e - {'path': '/', 'tag': 'e457fa4c-996a-401f-973e-1f79cdbfba9e', 'options': 'recursive,force'} 2024-03-24T09:29:49.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:49 smithi143 ceph-mon[35965]: pgmap v236: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:29:50.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:29:50.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:50.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:50.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:50.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:50.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:50 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:29:50.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:50 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:29:50.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:50.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:50.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:50.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:50.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:50.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:50 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:51 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:51 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:29:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:51 smithi117 ceph-mon[35788]: pgmap v237: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:51 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:51 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:51 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:29:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:51 smithi116 ceph-mon[32207]: pgmap v237: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:51 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:51 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:51 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:29:51.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:51 smithi143 ceph-mon[35965]: pgmap v237: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:29:51.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:51.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:51 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:29:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:53 smithi117 ceph-mon[35788]: pgmap v238: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:53 smithi116 ceph-mon[32207]: pgmap v238: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:53 smithi143 ceph-mon[35965]: pgmap v238: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:55 smithi117 ceph-mon[35788]: pgmap v239: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:55 smithi116 ceph-mon[32207]: pgmap v239: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:55 smithi143 ceph-mon[35965]: pgmap v239: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:57 smithi117 ceph-mon[35788]: pgmap v240: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:57.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:57 smithi116 ceph-mon[32207]: pgmap v240: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:57 smithi143 ceph-mon[35965]: pgmap v240: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:29:59.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:29:59 smithi143 ceph-mon[35965]: pgmap v241: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:59.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:29:59 smithi117 ceph-mon[35788]: pgmap v241: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:29:59.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:29:59 smithi116 ceph-mon[32207]: pgmap v241: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:00.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:00 smithi143 ceph-mon[35965]: overall HEALTH_OK 2024-03-24T09:30:00.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:00 smithi117 ceph-mon[35788]: overall HEALTH_OK 2024-03-24T09:30:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:00 smithi116 ceph-mon[32207]: overall HEALTH_OK 2024-03-24T09:30:01.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:01 smithi143 ceph-mon[35965]: pgmap v242: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:01 smithi117 ceph-mon[35788]: pgmap v242: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:01 smithi116 ceph-mon[32207]: pgmap v242: 129 pgs: 129 active+clean; 152 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:03.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:03 smithi116 ceph-mon[32207]: pgmap v243: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:30:03.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:03 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:03.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:03 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:03.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:03 smithi143 ceph-mon[35965]: pgmap v243: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:30:03.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:03 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:03.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:03 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:03.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:03 smithi117 ceph-mon[35788]: pgmap v243: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:30:03.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:03 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:03.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:03 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:04.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:04 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:04.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:04 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:04.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:04 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:04.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:04 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:04.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:04 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:04.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:04 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:04.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:04 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:04.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:04 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:04.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:04 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:05.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:05 smithi143 ceph-mon[35965]: pgmap v244: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:30:05.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:05 smithi117 ceph-mon[35788]: pgmap v244: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:30:05.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:05 smithi116 ceph-mon[32207]: pgmap v244: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:30:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:07 smithi143 ceph-mon[35965]: pgmap v245: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:30:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:07 smithi117 ceph-mon[35788]: pgmap v245: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:30:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:07 smithi116 ceph-mon[32207]: pgmap v245: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:30:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:09 smithi143 ceph-mon[35965]: pgmap v246: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:30:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:09 smithi143 ceph-mon[35965]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T09:30:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:09 smithi117 ceph-mon[35788]: pgmap v246: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:30:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:09 smithi117 ceph-mon[35788]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T09:30:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:09 smithi116 ceph-mon[32207]: pgmap v246: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:30:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:09 smithi116 ceph-mon[32207]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T09:30:11.482 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:11 smithi117 ceph-mon[35788]: pgmap v248: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:30:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:11 smithi143 ceph-mon[35965]: pgmap v248: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:30:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:11 smithi116 ceph-mon[32207]: pgmap v248: 129 pgs: 129 active+clean; 152 MiB data, 908 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:30:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:13 smithi143 ceph-mon[35965]: pgmap v249: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:13 smithi117 ceph-mon[35788]: pgmap v249: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:13 smithi116 ceph-mon[32207]: pgmap v249: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:15.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:15 smithi143 ceph-mon[35965]: pgmap v250: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:15.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:15 smithi143 ceph-mon[35965]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T09:30:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:15 smithi117 ceph-mon[35788]: pgmap v250: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:15.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:15 smithi117 ceph-mon[35788]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T09:30:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:15 smithi116 ceph-mon[32207]: pgmap v250: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:15 smithi116 ceph-mon[32207]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T09:30:17.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:17 smithi143 ceph-mon[35965]: pgmap v252: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:30:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:17 smithi117 ceph-mon[35788]: pgmap v252: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:30:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:17 smithi116 ceph-mon[32207]: pgmap v252: 129 pgs: 129 active+clean; 150 MiB data, 878 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:30:18.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:18 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:30:18.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:18 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:30:18.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:18 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:30:19.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:19 smithi143 ceph-mon[35965]: pgmap v253: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2024-03-24T09:30:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:19 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:30:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:19 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:30:19.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:19 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:30:19.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:19 smithi117 ceph-mon[35788]: pgmap v253: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2024-03-24T09:30:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:19 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:30:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:19 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:30:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:19 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:30:19.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:19 smithi116 ceph-mon[32207]: pgmap v253: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 219 B/s wr, 11 op/s 2024-03-24T09:30:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:19 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:30:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:19 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:30:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:19 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:30:19.476 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:30:19.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:30:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 -- 172.21.15.116:0/2228750915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec941005c0 msgr2=0x7fec94100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/2228750915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec941005c0 0x7fec94100a40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fec90009930 tx=0x7fec9002bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 -- 172.21.15.116:0/2228750915 shutdown_connections 2024-03-24T09:30:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/2228750915 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec94100f80 0x7fec9410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/2228750915 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec941005c0 0x7fec94100a40 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/2228750915 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec94106660 0x7fec94106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 -- 172.21.15.116:0/2228750915 >> 172.21.15.116:0/2228750915 conn(0x7fec94076990 msgr2=0x7fec94076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 -- 172.21.15.116:0/2228750915 shutdown_connections 2024-03-24T09:30:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.588+0000 7fec9b9e2640 1 -- 172.21.15.116:0/2228750915 wait complete. 2024-03-24T09:30:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 Processor -- start 2024-03-24T09:30:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 -- start start 2024-03-24T09:30:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec941005c0 0x7fec9419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 0x7fec9419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec94106660 0x7fec94197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec94198080 con 0x7fec94100f80 2024-03-24T09:30:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec941981f0 con 0x7fec94106660 2024-03-24T09:30:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9b9e2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fec941983b0 con 0x7fec941005c0 2024-03-24T09:30:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 0x7fec9419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 0x7fec9419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58800/0 (socket says 172.21.15.116:58800) 2024-03-24T09:30:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 -- 172.21.15.116:0/571429661 learned_addr learned my addr 172.21.15.116:0/571429661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec9977e640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec941005c0 0x7fec9419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec99f7f640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec94106660 0x7fec94197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 -- 172.21.15.116:0/571429661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec941005c0 msgr2=0x7fec9419d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec941005c0 0x7fec9419d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 -- 172.21.15.116:0/571429661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec94106660 msgr2=0x7fec94197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec94106660 0x7fec94197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 -- 172.21.15.116:0/571429661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec90009560 con 0x7fec94100f80 2024-03-24T09:30:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.589+0000 7fec98f7d640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 0x7fec9419ddc0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fec90009230 tx=0x7fec90005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.590+0000 7fec8a7fc640 1 -- 172.21.15.116:0/571429661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90004350 con 0x7fec94100f80 2024-03-24T09:30:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.590+0000 7fec8a7fc640 1 -- 172.21.15.116:0/571429661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec9002de90 con 0x7fec94100f80 2024-03-24T09:30:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.590+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec94198630 con 0x7fec94100f80 2024-03-24T09:30:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.590+0000 7fec8a7fc640 1 -- 172.21.15.116:0/571429661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90005220 con 0x7fec94100f80 2024-03-24T09:30:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.590+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec941a4920 con 0x7fec94100f80 2024-03-24T09:30:19.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.592+0000 7fec8a7fc640 1 -- 172.21.15.116:0/571429661 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fec90003c30 con 0x7fec94100f80 2024-03-24T09:30:19.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.592+0000 7fec9e5c0740 1 Processor -- start 2024-03-24T09:30:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.592+0000 7fec9e5c0740 1 -- start start 2024-03-24T09:30:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.592+0000 7fec8a7fc640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec800a2230 0x7fec800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.592+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 0x55b0398a35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a3af0 0x55b0398ac190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0397c7f30 con 0x55b0398a31d0 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b039811810 con 0x55b0398a3af0 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b03980ed40 con 0x55b0398ac6d0 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec99f7f640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec800a2230 0x7fec800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec8a7fc640 1 -- 172.21.15.116:0/571429661 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fec900f02e0 con 0x7fec94100f80 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 0x55b0398a35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 0x55b0398a35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58806/0 (socket says 172.21.15.116:58806) 2024-03-24T09:30:19.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 -- 172.21.15.116:0/2926654290 learned_addr learned my addr 172.21.15.116:0/2926654290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:19.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec98f7d640 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a3af0 0x55b0398ac190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec99f7f640 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 -- 172.21.15.116:0/2926654290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 msgr2=0x55b0398aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 -- 172.21.15.116:0/2926654290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a3af0 msgr2=0x55b0398ac190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a3af0 0x55b0398ac190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 -- 172.21.15.116:0/2926654290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0397cbea0 con 0x55b0398a31d0 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec99f7f640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec800a2230 0x7fec800a46f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec8c005fd0 tx=0x7fec8c013b60 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec9977e640 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 0x55b0398a35b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fec84009c60 tx=0x7fec84011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322b757c3daf6240 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec7f7fe640 1 -- 172.21.15.116:0/2926654290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84004220 con 0x55b0398a31d0 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec7f7fe640 1 -- 172.21.15.116:0/2926654290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec84004380 con 0x55b0398a31d0 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.593+0000 7fec7f7fe640 1 -- 172.21.15.116:0/2926654290 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84004c20 con 0x55b0398a31d0 2024-03-24T09:30:19.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 -- 172.21.15.116:0/2926654290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 msgr2=0x55b0398a35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 0x55b0398a35b0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fec84009c60 tx=0x7fec84011810 comp rx=0 tx=0).stop 2024-03-24T09:30:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 -- 172.21.15.116:0/2926654290 shutdown_connections 2024-03-24T09:30:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a3af0 0x55b0398ac190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/2926654290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a31d0 0x55b0398a35b0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 -- 172.21.15.116:0/2926654290 >> 172.21.15.116:0/2926654290 conn(0x55b03989d900 msgr2=0x55b03989dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 -- 172.21.15.116:0/2926654290 shutdown_connections 2024-03-24T09:30:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 -- 172.21.15.116:0/2926654290 wait complete. 2024-03-24T09:30:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 -- start start 2024-03-24T09:30:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.594+0000 7fec9e5c0740 1 Processor -- start 2024-03-24T09:30:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398a9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aa1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b03980a520 con 0x55b0398a3af0 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0394dd430 con 0x55b0398a31d0 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b039499e00 con 0x55b0398ac6d0 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9e5c0740 10 client.0 _ll_register_callbacks cb 0x55b0397a8d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9977e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58810/0 (socket says 172.21.15.116:58810) 2024-03-24T09:30:19.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9977e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398a9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:35370/0 (socket says 172.21.15.116:35370) 2024-03-24T09:30:19.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 -- 172.21.15.116:0/1604535627 learned_addr learned my addr 172.21.15.116:0/1604535627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:19.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec99f7f640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aa1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 -- 172.21.15.116:0/1604535627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 msgr2=0x55b0398aa1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aa1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 -- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 msgr2=0x55b0398a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 -- 172.21.15.116:0/1604535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec84009890 con 0x55b0398a3af0 2024-03-24T09:30:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec9977e640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec98f7d640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a9c60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fec900bc960 tx=0x7fec900bc3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c6381a660551c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.595+0000 7fec7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec7cff9640 1 -- 172.21.15.116:0/1604535627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec900c4ee0 con 0x55b0398a3af0 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec7cff9640 1 -- 172.21.15.116:0/1604535627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fec900c0b20 con 0x55b0398a3af0 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b0395097e0 con 0x55b0398a3af0 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec7cff9640 1 -- 172.21.15.116:0/1604535627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec900c5170 con 0x55b0398a3af0 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 10 client.17919 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b039808ba0 con 0x55b0398a3af0 2024-03-24T09:30:19.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec7cff9640 1 -- 172.21.15.116:0/1604535627 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b0395097e0 con 0x55b0398a3af0 2024-03-24T09:30:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec7cff9640 1 -- 172.21.15.116:0/1604535627 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec9003c070 con 0x55b0398a3af0 2024-03-24T09:30:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 10 client.17919 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 10 client.17919 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 20 client.17919 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b0398bc940 0x55b0398bed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 4 client.17919 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:19.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b039546720 con 0x55b0398bc940 2024-03-24T09:30:19.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.596+0000 7fec9977e640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b0398bc940 0x55b0398bed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.597+0000 7fec7cff9640 10 client.17919 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:19.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.597+0000 7fec9977e640 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b0398bc940 0x55b0398bed70 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f29274991003ca2e server_cookie=cd8f1fd5a8acd78e in_seq=0 out_seq=0 2024-03-24T09:30:19.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec7cff9640 1 -- 172.21.15.116:0/1604535627 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b039546720 con 0x55b0398bc940 2024-03-24T09:30:19.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec7cff9640 10 client.17919 handle_command_reply: tid=0 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 client.17919 shutdown 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 msgr2=0x55b0398a9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a9c60 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fec900bc960 tx=0x7fec900bc3f0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 shutdown_connections 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398ac6d0 0x55b0398aa1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a9c60 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b0398bc940 0x55b0398bed70 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/1604535627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec7cff9640 0 client.17919 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:19.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 >> 172.21.15.116:0/1604535627 conn(0x55b03989d900 msgr2=0x55b03989cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 shutdown_connections 2024-03-24T09:30:19.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 1 -- 172.21.15.116:0/1604535627 wait complete. 2024-03-24T09:30:19.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.598+0000 7fec9e5c0740 20 client.17919 trim_cache size 0 max 16384 2024-03-24T09:30:19.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 Processor -- start 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 -- start start 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a3af0 0x55b0398a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 0x55b0398c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b03978f430 con 0x55b0398ac6d0 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0398a25a0 con 0x55b0398a31d0 2024-03-24T09:30:19.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.615+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b0398991d0 con 0x55b0398a3af0 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 0x55b0398c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 0x55b0398c4650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58820/0 (socket says 172.21.15.116:58820) 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 -- 172.21.15.116:0/3835067818 learned_addr learned my addr 172.21.15.116:0/3835067818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec9977e640 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec98f7d640 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a3af0 0x55b0398a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 -- 172.21.15.116:0/3835067818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a3af0 msgr2=0x55b0398a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a3af0 0x55b0398a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 -- 172.21.15.116:0/3835067818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 msgr2=0x55b0398b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 -- 172.21.15.116:0/3835067818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0397a4b20 con 0x55b0398ac6d0 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec99f7f640 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 0x55b0398c4650 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00fb20 tx=0x7fec8c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97cc33715f5c5a8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec7effd640 1 -- 172.21.15.116:0/3835067818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c004b20 con 0x55b0398ac6d0 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec7effd640 1 -- 172.21.15.116:0/3835067818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec8c025070 con 0x55b0398ac6d0 2024-03-24T09:30:19.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.616+0000 7fec7effd640 1 -- 172.21.15.116:0/3835067818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c005260 con 0x55b0398ac6d0 2024-03-24T09:30:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 -- 172.21.15.116:0/3835067818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 msgr2=0x55b0398c4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 0x55b0398c4650 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00fb20 tx=0x7fec8c009500 comp rx=0 tx=0).stop 2024-03-24T09:30:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 -- 172.21.15.116:0/3835067818 shutdown_connections 2024-03-24T09:30:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398ac6d0 0x55b0398c4650 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a3af0 0x55b0398a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/3835067818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398a31d0 0x55b0398b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 -- 172.21.15.116:0/3835067818 >> 172.21.15.116:0/3835067818 conn(0x55b03989d900 msgr2=0x55b03989cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 -- 172.21.15.116:0/3835067818 shutdown_connections 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 -- 172.21.15.116:0/3835067818 wait complete. 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 -- start start 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.617+0000 7fec9e5c0740 1 Processor -- start 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a31d0 0x55b0398a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398ac6d0 0x55b039921c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0398a7740 con 0x55b0398a3af0 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b0399221f0 con 0x55b0398ac6d0 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b0399223b0 con 0x55b0398a31d0 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 10 client.0 _ll_register_callbacks cb 0x7fec90009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec98f7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec98f7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58830/0 (socket says 172.21.15.116:58830) 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec98f7d640 1 -- 172.21.15.116:0/4279670949 learned_addr learned my addr 172.21.15.116:0/4279670949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398ac6d0 0x55b039921c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9977e640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a31d0 0x55b0398a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 -- 172.21.15.116:0/4279670949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a31d0 msgr2=0x55b0398a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a31d0 0x55b0398a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 -- 172.21.15.116:0/4279670949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 msgr2=0x55b0398a6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:19.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 -- 172.21.15.116:0/4279670949 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0397ca700 con 0x55b0398ac6d0 2024-03-24T09:30:19.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec98f7d640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:19.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec99f7f640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398ac6d0 0x55b039921c80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00faf0 tx=0x7fec8c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d54a1b049346e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec7cff9640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:30:19.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec7cff9640 1 -- 172.21.15.116:0/4279670949 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c003970 con 0x55b0398ac6d0 2024-03-24T09:30:19.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec7cff9640 1 -- 172.21.15.116:0/4279670949 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fec8c003ad0 con 0x55b0398ac6d0 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.618+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec84004380 con 0x55b0398ac6d0 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec7cff9640 1 -- 172.21.15.116:0/4279670949 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec8c003cb0 con 0x55b0398ac6d0 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 10 client.27740 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b0397b0e90 con 0x55b0398ac6d0 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec7cff9640 1 -- 172.21.15.116:0/4279670949 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec84004380 con 0x55b0398ac6d0 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec7cff9640 1 -- 172.21.15.116:0/4279670949 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec8c004320 con 0x55b0398ac6d0 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 10 client.27740 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:19.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 10 client.27740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 20 client.27740 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b039923ba0 0x55b039925f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 4 client.27740 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b039595890 con 0x55b039923ba0 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9977e640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b039923ba0 0x55b039925f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec7cff9640 10 client.27740 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.619+0000 7fec9977e640 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b039923ba0 0x55b039925f90 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35f860efb3679009 server_cookie=783c9e4c4688ab31 in_seq=0 out_seq=0 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec7cff9640 1 -- 172.21.15.116:0/4279670949 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b039595890 con 0x55b039923ba0 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec7cff9640 10 client.27740 handle_command_reply: tid=0 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 client.27740 shutdown 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398ac6d0 msgr2=0x55b039921c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398ac6d0 0x55b039921c80 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec8c00faf0 tx=0x7fec8c004a40 comp rx=0 tx=0).stop 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 shutdown_connections 2024-03-24T09:30:19.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55b039923ba0 0x55b039925f90 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b0398ac6d0 0x55b039921c80 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec7cff9640 0 client.27740 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0398a3af0 0x55b0398a6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 --2- 172.21.15.116:0/4279670949 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b0398a31d0 0x55b0398a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 >> 172.21.15.116:0/4279670949 conn(0x55b03989d900 msgr2=0x55b03989ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 shutdown_connections 2024-03-24T09:30:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 1 -- 172.21.15.116:0/4279670949 wait complete. 2024-03-24T09:30:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9e5c0740 20 client.27740 trim_cache size 0 max 16384 2024-03-24T09:30:19.639 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:30:19.639 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:30:19.639 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:30:19.639 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:30:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec800a2230 msgr2=0x7fec800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec800a2230 0x7fec800a46f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fec8c005fd0 tx=0x7fec8c013b60 comp rx=0 tx=0).stop 2024-03-24T09:30:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 msgr2=0x7fec9419ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.620+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 0x7fec9419ddc0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fec90009230 tx=0x7fec90005dd0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 shutdown_connections 2024-03-24T09:30:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec94106660 0x7fec94197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec94100f80 0x7fec9419ddc0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec800a2230 0x7fec800a46f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 --2- 172.21.15.116:0/571429661 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec941005c0 0x7fec9419d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 >> 172.21.15.116:0/571429661 conn(0x7fec94076990 msgr2=0x7fec940fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 shutdown_connections 2024-03-24T09:30:19.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.621+0000 7fec9b9e2640 1 -- 172.21.15.116:0/571429661 wait complete. 2024-03-24T09:30:19.643 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:30:19.643 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:30:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 -- 172.21.15.116:0/3747070089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 msgr2=0x7f205010e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 --2- 172.21.15.116:0/3747070089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 0x7f205010e570 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f204c009c10 tx=0x7f204c02bef0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 -- 172.21.15.116:0/3747070089 shutdown_connections 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 --2- 172.21.15.116:0/3747070089 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 0x7f205010e570 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 --2- 172.21.15.116:0/3747070089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f2050101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 --2- 172.21.15.116:0/3747070089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f2050107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 -- 172.21.15.116:0/3747070089 >> 172.21.15.116:0/3747070089 conn(0x7f20500fd1b0 msgr2=0x7f20500ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 -- 172.21.15.116:0/3747070089 shutdown_connections 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.744+0000 7f20583c9640 1 -- 172.21.15.116:0/3747070089 wait complete. 2024-03-24T09:30:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 Processor -- start 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 -- start start 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f205019d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 0x7f205019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f2050197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20501980c0 con 0x7f20501016a0 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2050198230 con 0x7f20501076b0 2024-03-24T09:30:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f20583c9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f20501983f0 con 0x7f2050102060 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f205019d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f205019d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58842/0 (socket says 172.21.15.116:58842) 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 -- 172.21.15.116:0/3477636882 learned_addr learned my addr 172.21.15.116:0/3477636882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2055964640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 0x7f205019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056966640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f2050197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 -- 172.21.15.116:0/3477636882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 msgr2=0x7f205019de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 0x7f205019de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 -- 172.21.15.116:0/3477636882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 msgr2=0x7f2050197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f2050197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 -- 172.21.15.116:0/3477636882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2048009560 con 0x7f20501016a0 2024-03-24T09:30:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.745+0000 7f2056165640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f205019d8c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f204800e9c0 tx=0x7f204800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.746+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204800cde0 con 0x7f20501016a0 2024-03-24T09:30:19.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.746+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204c009840 con 0x7f20501016a0 2024-03-24T09:30:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.746+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f20480044d0 con 0x7f20501016a0 2024-03-24T09:30:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.746+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048010430 con 0x7f20501016a0 2024-03-24T09:30:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.746+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20501988c0 con 0x7f20501016a0 2024-03-24T09:30:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.747+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2050068950 con 0x7f20501016a0 2024-03-24T09:30:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.748+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2048004000 con 0x7f20501016a0 2024-03-24T09:30:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.748+0000 7f20477fe640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f203c0a2230 0x7f203c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.748+0000 7f2055964640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f203c0a2230 0x7f203c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.748+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f2048014070 con 0x7f20501016a0 2024-03-24T09:30:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.749+0000 7f2055964640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f203c0a2230 0x7f203c0a46f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20501008e0 tx=0x7f2040005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.752+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2048097080 con 0x7f20501016a0 2024-03-24T09:30:19.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.966+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2050109cd0 con 0x7f20501016a0 2024-03-24T09:30:19.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.967+0000 7f20477fe640 1 -- 172.21.15.116:0/3477636882 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f204809a740 con 0x7f20501016a0 2024-03-24T09:30:19.968 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:30:19.968 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:30:19.968 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:30:19.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f203c0a2230 msgr2=0x7f203c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f203c0a2230 0x7f203c0a46f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20501008e0 tx=0x7f2040005f90 comp rx=0 tx=0).stop 2024-03-24T09:30:19.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 msgr2=0x7f205019d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:19.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f205019d8c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f204800e9c0 tx=0x7f204800ee80 comp rx=0 tx=0).stop 2024-03-24T09:30:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 shutdown_connections 2024-03-24T09:30:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20501076b0 0x7f2050197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f203c0a2230 0x7f203c0a46f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2050102060 0x7f205019de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 --2- 172.21.15.116:0/3477636882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20501016a0 0x7f205019d8c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:19.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 >> 172.21.15.116:0/3477636882 conn(0x7f20500fd1b0 msgr2=0x7f20500fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:19.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 shutdown_connections 2024-03-24T09:30:19.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:19.969+0000 7f20583c9640 1 -- 172.21.15.116:0/3477636882 wait complete. 2024-03-24T09:30:19.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:30:20.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 -- 172.21.15.116:0/1408798386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8102060 msgr2=0x7f94c810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 --2- 172.21.15.116:0/1408798386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8102060 0x7f94c810e570 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009cd0 tx=0x7f94c002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 -- 172.21.15.116:0/1408798386 shutdown_connections 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 --2- 172.21.15.116:0/1408798386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8102060 0x7f94c810e570 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 --2- 172.21.15.116:0/1408798386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 0x7f94c8101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 --2- 172.21.15.116:0/1408798386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94c81076b0 0x7f94c8107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 -- 172.21.15.116:0/1408798386 >> 172.21.15.116:0/1408798386 conn(0x7f94c8078b10 msgr2=0x7f94c8078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 -- 172.21.15.116:0/1408798386 shutdown_connections 2024-03-24T09:30:20.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.085+0000 7f94cc84e640 1 -- 172.21.15.116:0/1408798386 wait complete. 2024-03-24T09:30:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 Processor -- start 2024-03-24T09:30:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 -- start start 2024-03-24T09:30:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 0x7f94c819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94c8102060 0x7f94c819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 0x7f94c8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8198060 con 0x7f94c81076b0 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81981d0 con 0x7f94c8102060 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94cc84e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8198390 con 0x7f94c81016a0 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 0x7f94c8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c659c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 0x7f94c819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 0x7f94c8197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58866/0 (socket says 172.21.15.116:58866) 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 -- 172.21.15.116:0/2552143129 learned_addr learned my addr 172.21.15.116:0/2552143129 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94c8102060 0x7f94c819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 -- 172.21.15.116:0/2552143129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 msgr2=0x7f94c819d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 0x7f94c819d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 -- 172.21.15.116:0/2552143129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94c8102060 msgr2=0x7f94c819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94c8102060 0x7f94c819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c6d9d640 1 -- 172.21.15.116:0/2552143129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c0009900 con 0x7f94c81076b0 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.086+0000 7f94c659c640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 0x7f94c819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.087+0000 7f94c6d9d640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 0x7f94c8197a10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f94c0004af0 tx=0x7f94c0032f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.087+0000 7f94b77fe640 1 -- 172.21.15.116:0/2552143129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c00325d0 con 0x7f94c81076b0 2024-03-24T09:30:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.087+0000 7f94b77fe640 1 -- 172.21.15.116:0/2552143129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94c0003970 con 0x7f94c81076b0 2024-03-24T09:30:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.087+0000 7f94b77fe640 1 -- 172.21.15.116:0/2552143129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c00046b0 con 0x7f94c81076b0 2024-03-24T09:30:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.087+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c8198610 con 0x7f94c81076b0 2024-03-24T09:30:20.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.087+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c81a4900 con 0x7f94c81076b0 2024-03-24T09:30:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.089+0000 7f94b77fe640 1 -- 172.21.15.116:0/2552143129 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f94c003b040 con 0x7f94c81076b0 2024-03-24T09:30:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.089+0000 7f94cf42f740 1 Processor -- start 2024-03-24T09:30:20.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 -- start start 2024-03-24T09:30:20.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111f3f0 0x563c1111f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c111283b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c111288f0 0x563c1112ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c1108d840 con 0x563c1111fd10 2024-03-24T09:30:20.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563c1108ad70 con 0x563c1111f3f0 2024-03-24T09:30:20.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94cf42f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563c11086550 con 0x563c111288f0 2024-03-24T09:30:20.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94b77fe640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94b00a2230 0x7f94b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c111283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111f3f0 0x563c1111f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c111283b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58876/0 (socket says 172.21.15.116:58876) 2024-03-24T09:30:20.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c659c640 1 -- 172.21.15.116:0/694474272 learned_addr learned my addr 172.21.15.116:0/694474272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c6d9d640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94b00a2230 0x7f94b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c111288f0 0x563c1112ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94b77fe640 1 -- 172.21.15.116:0/2552143129 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f94c00f0cc0 con 0x7f94c81076b0 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 -- 172.21.15.116:0/694474272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c111288f0 msgr2=0x563c1112ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c111288f0 0x563c1112ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 -- 172.21.15.116:0/694474272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 msgr2=0x563c111283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c111283b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 -- 172.21.15.116:0/694474272 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c1101bde0 con 0x563c1111f3f0 2024-03-24T09:30:20.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c659c640 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c111283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c6d9d640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94b00a2230 0x7f94b00a46f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f94c00b98a0 tx=0x7f94c008f400 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.090+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111f3f0 0x563c1111f7d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f94c81039a0 tx=0x7f94b8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33b3b857e98404f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94b4ff9640 1 -- 172.21.15.116:0/694474272 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8004050 con 0x563c1111f3f0 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94b4ff9640 1 -- 172.21.15.116:0/694474272 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94b800f790 con 0x563c1111f3f0 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94b4ff9640 1 -- 172.21.15.116:0/694474272 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b80038e0 con 0x563c1111f3f0 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 -- 172.21.15.116:0/694474272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111f3f0 msgr2=0x563c1111f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111f3f0 0x563c1111f7d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f94c81039a0 tx=0x7f94b8004730 comp rx=0 tx=0).stop 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 -- 172.21.15.116:0/694474272 shutdown_connections 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c111288f0 0x563c1112ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c111283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 --2- 172.21.15.116:0/694474272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111f3f0 0x563c1111f7d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 -- 172.21.15.116:0/694474272 >> 172.21.15.116:0/694474272 conn(0x563c11119b20 msgr2=0x563c11119f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 -- 172.21.15.116:0/694474272 shutdown_connections 2024-03-24T09:30:20.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.091+0000 7f94cf42f740 1 -- 172.21.15.116:0/694474272 wait complete. 2024-03-24T09:30:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 -- start start 2024-03-24T09:30:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 Processor -- start 2024-03-24T09:30:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c11125990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11125ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11126410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c10d59430 con 0x563c1111fd10 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563c10d15e00 con 0x563c111288f0 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563c1100b590 con 0x563c1111f3f0 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94cf42f740 10 client.0 _ll_register_callbacks cb 0x563c1107afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11125ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c5d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c11125990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11125ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58890/0 (socket says 172.21.15.116:58890) 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c5d9b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c11125990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:35538/0 (socket says 172.21.15.116:35538) 2024-03-24T09:30:20.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 -- 172.21.15.116:0/3161625572 learned_addr learned my addr 172.21.15.116:0/3161625572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11126410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 -- 172.21.15.116:0/3161625572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 msgr2=0x563c11125990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c11125990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 -- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 msgr2=0x563c11126410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11126410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.092+0000 7f94c659c640 1 -- 172.21.15.116:0/3161625572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b8019040 con 0x563c1111fd10 2024-03-24T09:30:20.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94c659c640 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11125ed0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f94bc012d20 tx=0x7f94bc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4278fa6d228cff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94adffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94adffb640 1 -- 172.21.15.116:0/3161625572 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc017070 con 0x563c1111fd10 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94adffb640 1 -- 172.21.15.116:0/3161625572 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f94bc012110 con 0x563c1111fd10 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563c10d857e0 con 0x563c1111fd10 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94adffb640 1 -- 172.21.15.116:0/3161625572 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f94bc004030 con 0x563c1111fd10 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 10 client.17943 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c11084bd0 con 0x563c1111fd10 2024-03-24T09:30:20.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94adffb640 1 -- 172.21.15.116:0/3161625572 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x563c10d857e0 con 0x563c1111fd10 2024-03-24T09:30:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94adffb640 1 -- 172.21.15.116:0/3161625572 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f94bc010430 con 0x563c1111fd10 2024-03-24T09:30:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 10 client.17943 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 10 client.17943 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:30:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 20 client.17943 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c11138b90 0x563c1113afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 4 client.17943 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c10dc2720 con 0x563c11138b90 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.093+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c11138b90 0x563c1113afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c11138b90 0x563c1113afb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69bb9cba11703077 server_cookie=38c1a0317cf26053 in_seq=0 out_seq=0 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94adffb640 10 client.17943 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94adffb640 1 -- 172.21.15.116:0/3161625572 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c10dc2720 con 0x563c11138b90 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94adffb640 10 client.17943 handle_command_reply: tid=0 2024-03-24T09:30:20.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 client.17943 shutdown 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 msgr2=0x563c11125ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11125ed0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f94bc012d20 tx=0x7f94bc00f790 comp rx=0 tx=0).stop 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 shutdown_connections 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11126410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c11138b90 0x563c1113afb0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11125ed0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 --2- 172.21.15.116:0/3161625572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c11125990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94adffb640 0 client.17943 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:20.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 >> 172.21.15.116:0/3161625572 conn(0x563c11119b20 msgr2=0x563c1111b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.094+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 shutdown_connections 2024-03-24T09:30:20.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.095+0000 7f94cf42f740 1 -- 172.21.15.116:0/3161625572 wait complete. 2024-03-24T09:30:20.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.095+0000 7f94cf42f740 20 client.17943 trim_cache size 0 max 16384 2024-03-24T09:30:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 Processor -- start 2024-03-24T09:30:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 -- start start 2024-03-24T09:30:20.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11126230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11140880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c11120790 con 0x563c1111fd10 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563c11123f00 con 0x563c111288f0 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94cf42f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563c1111a4a0 con 0x563c1111f3f0 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11126230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11126230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58892/0 (socket says 172.21.15.116:58892) 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 -- 172.21.15.116:0/1137105235 learned_addr learned my addr 172.21.15.116:0/1137105235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 -- 172.21.15.116:0/1137105235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 msgr2=0x563c111280c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111280c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 -- 172.21.15.116:0/1137105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 msgr2=0x563c11140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 -- 172.21.15.116:0/1137105235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c11020c80 con 0x563c1111fd10 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94c659c640 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11126230 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00a010 tx=0x7f94bc005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98760f637c02c531 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94affff640 1 -- 172.21.15.116:0/1137105235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc017040 con 0x563c1111fd10 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94affff640 1 -- 172.21.15.116:0/1137105235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94bc0055c0 con 0x563c1111fd10 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.108+0000 7f94affff640 1 -- 172.21.15.116:0/1137105235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc010a30 con 0x563c1111fd10 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 -- 172.21.15.116:0/1137105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 msgr2=0x563c11126230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11126230 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00a010 tx=0x7f94bc005850 comp rx=0 tx=0).stop 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 -- 172.21.15.116:0/1137105235 shutdown_connections 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c111288f0 0x563c11140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c1111fd10 0x563c11126230 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 --2- 172.21.15.116:0/1137105235 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111280c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 -- 172.21.15.116:0/1137105235 >> 172.21.15.116:0/1137105235 conn(0x563c11119b20 msgr2=0x563c1111ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 -- 172.21.15.116:0/1137105235 shutdown_connections 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 -- 172.21.15.116:0/1137105235 wait complete. 2024-03-24T09:30:20.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 -- start start 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.109+0000 7f94cf42f740 1 Processor -- start 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111307e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111fd10 0x563c1112d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 0x563c1112dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563c11130ee0 con 0x563c111288f0 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563c11131050 con 0x563c1111fd10 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x563c1112e2b0 con 0x563c1111f3f0 2024-03-24T09:30:20.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 10 client.0 _ll_register_callbacks cb 0x7f94bc009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 0x563c1112dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111fd10 0x563c1112d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 0x563c1112dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58902/0 (socket says 172.21.15.116:58902) 2024-03-24T09:30:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111fd10 0x563c1112d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:35442/0 (socket says 172.21.15.116:35442) 2024-03-24T09:30:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 -- 172.21.15.116:0/2502029419 learned_addr learned my addr 172.21.15.116:0/2502029419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111307e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 -- 172.21.15.116:0/2502029419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 msgr2=0x563c111307e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111307e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 -- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111fd10 msgr2=0x563c1112d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111fd10 0x563c1112d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 -- 172.21.15.116:0/2502029419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc021040 con 0x563c111288f0 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94c5d9b640 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 0x563c1112dcf0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f94b8000c00 tx=0x7f94b8021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4094f974e91b1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94adffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94adffb640 1 -- 172.21.15.116:0/2502029419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b800fd80 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b800f790 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94adffb640 1 -- 172.21.15.116:0/2502029419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f94b8005020 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94adffb640 1 -- 172.21.15.116:0/2502029419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f94b80051e0 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 10 client.17949 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.110+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c11046920 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94adffb640 1 -- 172.21.15.116:0/2502029419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f94b800f790 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94adffb640 1 -- 172.21.15.116:0/2502029419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f94b801d070 con 0x563c111288f0 2024-03-24T09:30:20.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 10 client.17949 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 10 client.17949 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 20 client.17949 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c111a00f0 0x563c111a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 4 client.17949 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c10e11890 con 0x563c111a00f0 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94c659c640 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c111a00f0 0x563c111a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94adffb640 10 client.17949 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:20.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94c659c640 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c111a00f0 0x563c111a24e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e725bbefa017a0f7 server_cookie=92bee267990aca7c in_seq=0 out_seq=0 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94adffb640 1 -- 172.21.15.116:0/2502029419 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c10e11890 con 0x563c111a00f0 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94adffb640 10 client.17949 handle_command_reply: tid=0 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 1 client.17949 shutdown 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 msgr2=0x563c1112dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.111+0000 7f94cf42f740 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 0x563c1112dcf0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f94b8000c00 tx=0x7f94b8021040 comp rx=0 tx=0).stop 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 shutdown_connections 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x563c111a00f0 0x563c111a24e0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563c111288f0 0x563c1112dcf0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563c1111fd10 0x563c1112d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94adffb640 0 client.17949 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 --2- 172.21.15.116:0/2502029419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x563c1111f3f0 0x563c111307e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 >> 172.21.15.116:0/2502029419 conn(0x563c11119b20 msgr2=0x563c1111b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 shutdown_connections 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 1 -- 172.21.15.116:0/2502029419 wait complete. 2024-03-24T09:30:20.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cf42f740 20 client.17949 trim_cache size 0 max 16384 2024-03-24T09:30:20.131 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94b00a2230 msgr2=0x7f94b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94b00a2230 0x7f94b00a46f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f94c00b98a0 tx=0x7f94c008f400 comp rx=0 tx=0).stop 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 msgr2=0x7f94c8197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 0x7f94c8197a10 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f94c0004af0 tx=0x7f94c0032f30 comp rx=0 tx=0).stop 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 shutdown_connections 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c81076b0 0x7f94c8197a10 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94c8102060 0x7f94c819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94b00a2230 0x7f94b00a46f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 --2- 172.21.15.116:0/2552143129 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94c81016a0 0x7f94c819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 >> 172.21.15.116:0/2552143129 conn(0x7f94c8078b10 msgr2=0x7f94c81041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 shutdown_connections 2024-03-24T09:30:20.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.112+0000 7f94cc84e640 1 -- 172.21.15.116:0/2552143129 wait complete. 2024-03-24T09:30:20.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:30:20.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.237+0000 7f7eaed56640 1 -- 172.21.15.116:0/4111875621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100ff0 msgr2=0x7f7ea810d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.237+0000 7f7eaed56640 1 --2- 172.21.15.116:0/4111875621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100ff0 0x7f7ea810d500 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009cd0 tx=0x7f7ea002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- 172.21.15.116:0/4111875621 shutdown_connections 2024-03-24T09:30:20.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 --2- 172.21.15.116:0/4111875621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100ff0 0x7f7ea810d500 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 --2- 172.21.15.116:0/4111875621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8100630 0x7f7ea8100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 --2- 172.21.15.116:0/4111875621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ea8106640 0x7f7ea8106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- 172.21.15.116:0/4111875621 >> 172.21.15.116:0/4111875621 conn(0x7f7ea8077770 msgr2=0x7f7ea8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- 172.21.15.116:0/4111875621 shutdown_connections 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- 172.21.15.116:0/4111875621 wait complete. 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 Processor -- start 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- start start 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 0x7f7ea80fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ea8100ff0 0x7f7ea80fdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8106640 0x7f7ea80fe030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea81a6050 con 0x7f7ea8100630 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea81a61c0 con 0x7f7ea8100ff0 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eaed56640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea81a6380 con 0x7f7ea8106640 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eacaf2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 0x7f7ea80fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eacaf2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 0x7f7ea80fd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58930/0 (socket says 172.21.15.116:58930) 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.238+0000 7f7eacaf2640 1 -- 172.21.15.116:0/353886770 learned_addr learned my addr 172.21.15.116:0/353886770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8106640 0x7f7ea80fe030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ea8100ff0 0x7f7ea80fdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eacaf2640 1 -- 172.21.15.116:0/353886770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8106640 msgr2=0x7f7ea80fe030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8106640 0x7f7ea80fe030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eacaf2640 1 -- 172.21.15.116:0/353886770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ea8100ff0 msgr2=0x7f7ea80fdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ea8100ff0 0x7f7ea80fdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eacaf2640 1 -- 172.21.15.116:0/353886770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e98009490 con 0x7f7ea8100630 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 0x7f7ea80fd5b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800ea60 tx=0x7f7e9800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7ea5ffb640 1 -- 172.21.15.116:0/353886770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9800cde0 con 0x7f7ea8100630 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7ea5ffb640 1 -- 172.21.15.116:0/353886770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e980044d0 con 0x7f7ea8100630 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea0009900 con 0x7f7ea8100630 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7ea5ffb640 1 -- 172.21.15.116:0/353886770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e98010690 con 0x7f7ea8100630 2024-03-24T09:30:20.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.239+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea81a69e0 con 0x7f7ea8100630 2024-03-24T09:30:20.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.241+0000 7f7ea5ffb640 1 -- 172.21.15.116:0/353886770 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7e98003e40 con 0x7f7ea8100630 2024-03-24T09:30:20.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ea5ffb640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7e900a2260 0x7f7e900a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 Processor -- start 2024-03-24T09:30:20.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 -- start start 2024-03-24T09:30:20.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7e900a2260 0x7f7e900a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ea5ffb640 1 -- 172.21.15.116:0/353886770 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f7e98014070 con 0x7f7ea8100630 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f999f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99aace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609f990d840 con 0x5609f99a88f0 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609f990ad70 con 0x5609f999f3f0 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5609f9906550 con 0x5609f999fd10 2024-03-24T09:30:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ead2f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99aace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eacaf2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f999f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ea7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ead2f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99aace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58938/0 (socket says 172.21.15.116:58938) 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ead2f3640 1 -- 172.21.15.116:0/2843459289 learned_addr learned my addr 172.21.15.116:0/2843459289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7e900a2260 0x7f7e900a4720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7ea80feff0 tx=0x7f7e9c005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eacaf2640 1 -- 172.21.15.116:0/2843459289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 msgr2=0x5609f99a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eacaf2640 1 -- 172.21.15.116:0/2843459289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 msgr2=0x5609f99aace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99aace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7eacaf2640 1 -- 172.21.15.116:0/2843459289 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609f989bde0 con 0x5609f999f3f0 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.242+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f999f7d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7e980cbbe0 tx=0x7f7e9809aab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f02c88d6370d00d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7e8effd640 1 -- 172.21.15.116:0/2843459289 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9809efd0 con 0x5609f999f3f0 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7e8effd640 1 -- 172.21.15.116:0/2843459289 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e980a3620 con 0x5609f999f3f0 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7e8effd640 1 -- 172.21.15.116:0/2843459289 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e980a3190 con 0x5609f999f3f0 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2843459289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 msgr2=0x5609f999f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f999f7d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7e980cbbe0 tx=0x7f7e9809aab0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2843459289 shutdown_connections 2024-03-24T09:30:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99aace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2843459289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f999f7d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.243+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2843459289 >> 172.21.15.116:0/2843459289 conn(0x5609f9999b20 msgr2=0x5609f9999f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2843459289 shutdown_connections 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2843459289 wait complete. 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 -- start start 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 Processor -- start 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999f3f0 0x5609f99a5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 0x5609f99a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99a6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609f95d9430 con 0x5609f999fd10 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609f9595e00 con 0x5609f99a88f0 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5609f988b590 con 0x5609f999f3f0 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eb1a0d740 10 client.0 _ll_register_callbacks cb 0x5609f98fafc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7ea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 0x5609f99a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7ea7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 0x5609f99a5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58950/0 (socket says 172.21.15.116:58950) 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7ea7fff640 1 -- 172.21.15.116:0/2557403521 learned_addr learned my addr 172.21.15.116:0/2557403521 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.244+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999f3f0 0x5609f99a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7ea7fff640 1 -- 172.21.15.116:0/2557403521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999f3f0 msgr2=0x5609f99a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999f3f0 0x5609f99a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7ea7fff640 1 -- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 msgr2=0x5609f99a6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99a6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7ea7fff640 1 -- 172.21.15.116:0/2557403521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea81a69e0 con 0x5609f999fd10 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 0x5609f99a5ed0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c01c980 tx=0x7f7e9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a49f774731b201 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7e87fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7e87fff640 1 -- 172.21.15.116:0/2557403521 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c021070 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7e87fff640 1 -- 172.21.15.116:0/2557403521 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f7e9c004020 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5609f96057e0 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7e87fff640 1 -- 172.21.15.116:0/2557403521 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7e9c004220 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7eb1a0d740 10 client.17961 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5609f9904bd0 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7e87fff640 1 -- 172.21.15.116:0/2557403521 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5609f96057e0 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7e87fff640 1 -- 172.21.15.116:0/2557403521 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7e9c0035f0 con 0x5609f999fd10 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7eb1a0d740 10 client.17961 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7eb1a0d740 10 client.17961 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:30:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.245+0000 7f7eb1a0d740 20 client.17961 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.246+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f99bae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.246+0000 7f7eb1a0d740 4 client.17961 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.246+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609f9642720 con 0x5609f99b8a10 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.246+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f99bae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.246+0000 7f7e87fff640 10 client.17961 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.246+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f99bae30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=69777f01a28a2f5c server_cookie=ae81fb68f53c278b in_seq=0 out_seq=0 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7e87fff640 1 -- 172.21.15.116:0/2557403521 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609f9642720 con 0x5609f99b8a10 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7e87fff640 10 client.17961 handle_command_reply: tid=0 2024-03-24T09:30:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 client.17961 shutdown 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 msgr2=0x5609f99a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 0x5609f99a5ed0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c01c980 tx=0x7f7e9c004730 comp rx=0 tx=0).stop 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 shutdown_connections 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f99bae30 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99a6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999fd10 0x5609f99a5ed0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7e87fff640 0 client.17961 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/2557403521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999f3f0 0x5609f99a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 >> 172.21.15.116:0/2557403521 conn(0x5609f9999b20 msgr2=0x5609f999b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 shutdown_connections 2024-03-24T09:30:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/2557403521 wait complete. 2024-03-24T09:30:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.247+0000 7f7eb1a0d740 20 client.17961 trim_cache size 0 max 16384 2024-03-24T09:30:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 Processor -- start 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 -- start start 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f99b4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99c0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609f999ab20 con 0x5609f99a88f0 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609f99b54d0 con 0x5609f999f3f0 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5609f9999650 con 0x5609f999fd10 2024-03-24T09:30:20.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7ead2f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99c0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7eacaf2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f99b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7ead2f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99c0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58960/0 (socket says 172.21.15.116:58960) 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7ead2f3640 1 -- 172.21.15.116:0/3007880622 learned_addr learned my addr 172.21.15.116:0/3007880622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.260+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7ead2f3640 1 -- 172.21.15.116:0/3007880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 msgr2=0x5609f99a6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7ead2f3640 1 -- 172.21.15.116:0/3007880622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 msgr2=0x5609f99b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f99b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7ead2f3640 1 -- 172.21.15.116:0/3007880622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609f98a0c80 con 0x5609f99a88f0 2024-03-24T09:30:20.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99c0740 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009cd0 tx=0x7f7ea0032480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6583b093bcc5d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7e8e7fc640 1 -- 172.21.15.116:0/3007880622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0034440 con 0x5609f99a88f0 2024-03-24T09:30:20.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7e8e7fc640 1 -- 172.21.15.116:0/3007880622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7ea00345a0 con 0x5609f99a88f0 2024-03-24T09:30:20.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7e8e7fc640 1 -- 172.21.15.116:0/3007880622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0003c60 con 0x5609f99a88f0 2024-03-24T09:30:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/3007880622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 msgr2=0x5609f99c0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.261+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99c0740 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009cd0 tx=0x7f7ea0032480 comp rx=0 tx=0).stop 2024-03-24T09:30:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/3007880622 shutdown_connections 2024-03-24T09:30:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f99a88f0 0x5609f99c0740 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99a6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/3007880622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f999f3f0 0x5609f99b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/3007880622 >> 172.21.15.116:0/3007880622 conn(0x5609f9999b20 msgr2=0x5609f9998cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/3007880622 shutdown_connections 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/3007880622 wait complete. 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- start start 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 Processor -- start 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 0x5609f99b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99ad780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5609f99b0eb0 con 0x5609f999f3f0 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5609f99b1020 con 0x5609f99a88f0 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5609f99ae200 con 0x5609f999fd10 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eb1a0d740 10 client.0 _ll_register_callbacks cb 0x7f7e9c012d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eacaf2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 0x5609f99b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eacaf2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 0x5609f99b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58962/0 (socket says 172.21.15.116:58962) 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7eacaf2640 1 -- 172.21.15.116:0/951857276 learned_addr learned my addr 172.21.15.116:0/951857276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7ea7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.262+0000 7f7ead2f3640 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eacaf2640 1 -- 172.21.15.116:0/951857276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 msgr2=0x5609f99ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eacaf2640 1 -- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 msgr2=0x5609f99adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eacaf2640 1 -- 172.21.15.116:0/951857276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea0030070 con 0x5609f999f3f0 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eacaf2640 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 0x5609f99b07b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7e9801f000 tx=0x7f7e9809b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b08a32bfc00752a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7e87fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7e87fff640 1 -- 172.21.15.116:0/951857276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e980b8040 con 0x5609f999f3f0 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7e87fff640 1 -- 172.21.15.116:0/951857276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f7e980035f0 con 0x5609f999f3f0 2024-03-24T09:30:20.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e980a3620 con 0x5609f999f3f0 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7e87fff640 1 -- 172.21.15.116:0/951857276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7e98099180 con 0x5609f999f3f0 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 10 client.17967 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5609f98c6920 con 0x5609f999f3f0 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7e87fff640 1 -- 172.21.15.116:0/951857276 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7e98089080 con 0x5609f999f3f0 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7e87fff640 1 -- 172.21.15.116:0/951857276 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7e980a3190 con 0x5609f999f3f0 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 10 client.17967 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 10 client.17967 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 20 client.17967 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f9a21d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 4 client.17967 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.263+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5609f9691890 con 0x5609f99b8a10 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f9a21d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7e87fff640 10 client.17967 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7ea7fff640 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f9a21d90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=96e37f6887f7b406 server_cookie=5fe24170fee5269d in_seq=0 out_seq=0 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7e87fff640 1 -- 172.21.15.116:0/951857276 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5609f9691890 con 0x5609f99b8a10 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7e87fff640 10 client.17967 handle_command_reply: tid=0 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 client.17967 shutdown 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 msgr2=0x5609f99b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 0x5609f99b07b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7e9801f000 tx=0x7f7e9809b000 comp rx=0 tx=0).stop 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 shutdown_connections 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x5609f99b8a10 0x5609f9a21d90 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5609f99a88f0 0x5609f99adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5609f999fd10 0x5609f99ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7e87fff640 0 client.17967 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 --2- 172.21.15.116:0/951857276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5609f999f3f0 0x5609f99b07b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 >> 172.21.15.116:0/951857276 conn(0x5609f9999b20 msgr2=0x5609f999ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 shutdown_connections 2024-03-24T09:30:20.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 1 -- 172.21.15.116:0/951857276 wait complete. 2024-03-24T09:30:20.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.264+0000 7f7eb1a0d740 20 client.17967 trim_cache size 0 max 16384 2024-03-24T09:30:20.275 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7e900a2260 msgr2=0x7f7e900a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7e900a2260 0x7f7e900a4720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7ea80feff0 tx=0x7f7e9c005f90 comp rx=0 tx=0).stop 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 msgr2=0x7f7ea80fd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 0x7f7ea80fd5b0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800ea60 tx=0x7f7e9800ef20 comp rx=0 tx=0).stop 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 shutdown_connections 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7e900a2260 0x7f7e900a4720 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ea8106640 0x7f7ea80fe030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ea8100ff0 0x7f7ea80fdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 --2- 172.21.15.116:0/353886770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ea8100630 0x7f7ea80fd5b0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 >> 172.21.15.116:0/353886770 conn(0x7f7ea8077770 msgr2=0x7f7ea806a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 shutdown_connections 2024-03-24T09:30:20.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.265+0000 7f7eaed56640 1 -- 172.21.15.116:0/353886770 wait complete. 2024-03-24T09:30:20.277 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:20 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3477636882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:30:20.278 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:30:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:20 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3477636882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:30:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.381+0000 7f8c73bb6640 1 -- 172.21.15.116:0/691071126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 msgr2=0x7f8c6c068910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.381+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/691071126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c068910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009900 tx=0x7f8c6802bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- 172.21.15.116:0/691071126 shutdown_connections 2024-03-24T09:30:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/691071126 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/691071126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c068e50 0x7f8c6c0692b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/691071126 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c068910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- 172.21.15.116:0/691071126 >> 172.21.15.116:0/691071126 conn(0x7f8c6c077710 msgr2=0x7f8c6c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- 172.21.15.116:0/691071126 shutdown_connections 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- 172.21.15.116:0/691071126 wait complete. 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 Processor -- start 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- start start 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c068e50 0x7f8c6c19e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c1a2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c1a30d0 con 0x7f8c6c1041d0 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c1a3240 con 0x7f8c6c068e50 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c73bb6640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8c6c1a3400 con 0x7f8c6c106660 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.382+0000 7f8c71151640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c19e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58982/0 (socket says 172.21.15.116:58982) 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 -- 172.21.15.116:0/4290769433 learned_addr learned my addr 172.21.15.116:0/4290769433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c72153640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c1a2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71952640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c068e50 0x7f8c6c19e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 -- 172.21.15.116:0/4290769433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 msgr2=0x7f8c6c1a2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c1a2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 -- 172.21.15.116:0/4290769433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c068e50 msgr2=0x7f8c6c19e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c068e50 0x7f8c6c19e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 -- 172.21.15.116:0/4290769433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c68009560 con 0x7f8c6c1041d0 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c72153640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c1a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71952640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c068e50 0x7f8c6c19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c71151640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c19e750 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00e9f0 tx=0x7f8c5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c62ffd640 1 -- 172.21.15.116:0/4290769433 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c009700 con 0x7f8c6c1041d0 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c6c1a36e0 con 0x7f8c6c1041d0 2024-03-24T09:30:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c62ffd640 1 -- 172.21.15.116:0/4290769433 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c5c00cd80 con 0x7f8c6c1041d0 2024-03-24T09:30:20.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c62ffd640 1 -- 172.21.15.116:0/4290769433 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c018840 con 0x7f8c6c1041d0 2024-03-24T09:30:20.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.383+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6c1aaf60 con 0x7f8c6c1041d0 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.386+0000 7f8c62ffd640 1 -- 172.21.15.116:0/4290769433 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f8c5c0189a0 con 0x7f8c6c1041d0 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.386+0000 7f8c76791740 1 Processor -- start 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.386+0000 7f8c76791740 1 -- start start 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c76791740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669acb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c76791740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669acbd10 0x55b669ad43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c76791740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669ad48f0 0x55b669ad6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c76791740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b669a39840 con 0x55b669ad48f0 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c76791740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b669a36d70 con 0x55b669acbd10 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c76791740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b669a32550 con 0x55b669acb3f0 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669acb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c71151640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669ad48f0 0x55b669ad6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c71952640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669acbd10 0x55b669ad43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c71952640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669acbd10 0x55b669ad43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:35534/0 (socket says 172.21.15.116:35534) 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669acb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:35642/0 (socket says 172.21.15.116:35642) 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c71151640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669ad48f0 0x55b669ad6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58986/0 (socket says 172.21.15.116:58986) 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c71952640 1 -- 172.21.15.116:0/1687500478 learned_addr learned my addr 172.21.15.116:0/1687500478 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c62ffd640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8c580a2230 0x7f8c580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 -- 172.21.15.116:0/1687500478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669acbd10 msgr2=0x55b669ad43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669acbd10 0x55b669ad43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 -- 172.21.15.116:0/1687500478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669ad48f0 msgr2=0x55b669ad6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669ad48f0 0x55b669ad6ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 -- 172.21.15.116:0/1687500478 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6699c7de0 con 0x55b669acb3f0 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c62ffd640 1 -- 172.21.15.116:0/4290769433 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8c5c014070 con 0x7f8c6c1041d0 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8c580a2230 0x7f8c580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.387+0000 7f8c72153640 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669acb7d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8c64005fd0 tx=0x7f8c64011c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1b690e56aee6862 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c57fff640 1 -- 172.21.15.116:0/1687500478 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6401bad0 con 0x55b669acb3f0 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c57fff640 1 -- 172.21.15.116:0/1687500478 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c6401bc30 con 0x55b669acb3f0 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c57fff640 1 -- 172.21.15.116:0/1687500478 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c6400e3b0 con 0x55b669acb3f0 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 -- 172.21.15.116:0/1687500478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 msgr2=0x55b669acb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c72153640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8c580a2230 0x7f8c580a46f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8c64023000 tx=0x7f8c64024000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669acb7d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8c64005fd0 tx=0x7f8c64011c90 comp rx=0 tx=0).stop 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 -- 172.21.15.116:0/1687500478 shutdown_connections 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669ad48f0 0x55b669ad6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669acbd10 0x55b669ad43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 --2- 172.21.15.116:0/1687500478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669acb7d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 -- 172.21.15.116:0/1687500478 >> 172.21.15.116:0/1687500478 conn(0x55b669ac5b20 msgr2=0x55b669ac5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 -- 172.21.15.116:0/1687500478 shutdown_connections 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.388+0000 7f8c76791740 1 -- 172.21.15.116:0/1687500478 wait complete. 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 -- start start 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 Processor -- start 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ad1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669ad1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad2400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b669705430 con 0x55b669acb3f0 2024-03-24T09:30:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b6696c1e00 con 0x55b669ad48f0 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b6699b7590 con 0x55b669acbd10 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c71952640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ad1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c76791740 10 client.0 _ll_register_callbacks cb 0x55b669a26fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c71952640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ad1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59000/0 (socket says 172.21.15.116:59000) 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c71952640 1 -- 172.21.15.116:0/2149351767 learned_addr learned my addr 172.21.15.116:0/2149351767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c72153640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c71151640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669ad1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c71952640 1 -- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 msgr2=0x55b669ad1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.389+0000 7f8c71952640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669ad1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c71952640 1 -- 172.21.15.116:0/2149351767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 msgr2=0x55b669ad2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c71952640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c71952640 1 -- 172.21.15.116:0/2149351767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c64011440 con 0x55b669acb3f0 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c71952640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ad1980 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f8c68005940 tx=0x7f8c68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d1b12a12477436 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c557fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c557fa640 1 -- 172.21.15.116:0/2149351767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68004a70 con 0x55b669acb3f0 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b6697317e0 con 0x55b669acb3f0 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c557fa640 1 -- 172.21.15.116:0/2149351767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8c680035f0 con 0x55b669acb3f0 2024-03-24T09:30:20.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c557fa640 1 -- 172.21.15.116:0/2149351767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8c680037b0 con 0x55b669acb3f0 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 10 client.17979 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b669a30bd0 con 0x55b669acb3f0 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c557fa640 1 -- 172.21.15.116:0/2149351767 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55b6697317e0 con 0x55b669acb3f0 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c557fa640 1 -- 172.21.15.116:0/2149351767 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8c6803c040 con 0x55b669acb3f0 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 10 client.17979 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 10 client.17979 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 20 client.17979 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669ae4a40 0x55b669ae6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 4 client.17979 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:20.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.390+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b66976e720 con 0x55b669ae4a40 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.391+0000 7f8c71151640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669ae4a40 0x55b669ae6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.391+0000 7f8c71151640 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669ae4a40 0x55b669ae6e60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f4b11c0f5cb53ba server_cookie=e1c343e7ae34d121 in_seq=0 out_seq=0 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.391+0000 7f8c557fa640 10 client.17979 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c557fa640 1 -- 172.21.15.116:0/2149351767 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b66976e720 con 0x55b669ae4a40 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c557fa640 10 client.17979 handle_command_reply: tid=0 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 client.17979 shutdown 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 msgr2=0x55b669ad1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ad1980 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f8c68005940 tx=0x7f8c68004270 comp rx=0 tx=0).stop 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 shutdown_connections 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669ae4a40 0x55b669ae6e60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c557fa640 0 client.17979 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669ad1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 --2- 172.21.15.116:0/2149351767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ad1980 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 >> 172.21.15.116:0/2149351767 conn(0x55b669ac5b20 msgr2=0x55b669ac7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 shutdown_connections 2024-03-24T09:30:20.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 1 -- 172.21.15.116:0/2149351767 wait complete. 2024-03-24T09:30:20.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.392+0000 7f8c76791740 20 client.17979 trim_cache size 0 max 16384 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 Processor -- start 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 -- start start 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ac4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669aec770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669adc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b669ac6b20 con 0x55b669acb3f0 2024-03-24T09:30:20.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b669ad6ce0 con 0x55b669ad48f0 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c76791740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b669acadc0 con 0x55b669acbd10 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ac4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ac4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59010/0 (socket says 172.21.15.116:59010) 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 -- 172.21.15.116:0/2368331819 learned_addr learned my addr 172.21.15.116:0/2368331819 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71151640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669aec770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c72153640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669adc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 -- 172.21.15.116:0/2368331819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 msgr2=0x55b669aec770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669aec770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 -- 172.21.15.116:0/2368331819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 msgr2=0x55b669adc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669adc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71952640 1 -- 172.21.15.116:0/2368331819 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6699ccc80 con 0x55b669acb3f0 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c72153640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669adc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.405+0000 7f8c71151640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669aec770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c71952640 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ac4f00 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f8c68006010 tx=0x7f8c68005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3516ffe81dd66c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c577fe640 1 -- 172.21.15.116:0/2368331819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68005d50 con 0x55b669acb3f0 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c577fe640 1 -- 172.21.15.116:0/2368331819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c68004a70 con 0x55b669acb3f0 2024-03-24T09:30:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c577fe640 1 -- 172.21.15.116:0/2368331819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c68034cd0 con 0x55b669acb3f0 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 -- 172.21.15.116:0/2368331819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 msgr2=0x55b669ac4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ac4f00 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f8c68006010 tx=0x7f8c68005b50 comp rx=0 tx=0).stop 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 -- 172.21.15.116:0/2368331819 shutdown_connections 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669adc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acbd10 0x55b669aec770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 --2- 172.21.15.116:0/2368331819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acb3f0 0x55b669ac4f00 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 -- 172.21.15.116:0/2368331819 >> 172.21.15.116:0/2368331819 conn(0x55b669ac5b20 msgr2=0x55b669ac75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 -- 172.21.15.116:0/2368331819 shutdown_connections 2024-03-24T09:30:20.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.406+0000 7f8c76791740 1 -- 172.21.15.116:0/2368331819 wait complete. 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 -- start start 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 Processor -- start 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669adc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 0x55b669adc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b669ad8f30 con 0x55b669acbd10 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b669ad90a0 con 0x55b669ad48f0 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55b669ad9260 con 0x55b669acb3f0 2024-03-24T09:30:20.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c76791740 10 client.0 _ll_register_callbacks cb 0x7f8c68009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 0x55b669adc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 0x55b669adc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59026/0 (socket says 172.21.15.116:59026) 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71952640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669adc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 -- 172.21.15.116:0/617109833 learned_addr learned my addr 172.21.15.116:0/617109833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c72153640 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 -- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 msgr2=0x55b669adc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669adc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 -- 172.21.15.116:0/617109833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 msgr2=0x55b669ad8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.407+0000 7f8c71151640 1 -- 172.21.15.116:0/617109833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c6802ce70 con 0x55b669acbd10 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c71151640 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 0x55b669adc990 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c09ab40 tx=0x7f8c5c0ce970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40582247b3091d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c557fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c557fa640 1 -- 172.21.15.116:0/617109833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c09e040 con 0x55b669acbd10 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c557fa640 1 -- 172.21.15.116:0/617109833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f8c5c0a3530 con 0x55b669acbd10 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c6401bc30 con 0x55b669acbd10 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c557fa640 1 -- 172.21.15.116:0/617109833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8c5c010b50 con 0x55b669acbd10 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 10 client.17985 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b6699f2920 con 0x55b669acbd10 2024-03-24T09:30:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c557fa640 1 -- 172.21.15.116:0/617109833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8c6401bc30 con 0x55b669acbd10 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c557fa640 1 -- 172.21.15.116:0/617109833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8c5c0ce040 con 0x55b669acbd10 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 10 client.17985 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 10 client.17985 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 20 client.17985 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669b4bf90 0x55b669b4e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 4 client.17985 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6697bd890 con 0x55b669b4bf90 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.408+0000 7f8c71952640 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669b4bf90 0x55b669b4e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c557fa640 10 client.17985 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c71952640 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669b4bf90 0x55b669b4e3b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3661c353bf372d5d server_cookie=675b55d81d560ab3 in_seq=0 out_seq=0 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c557fa640 1 -- 172.21.15.116:0/617109833 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6697bd890 con 0x55b669b4bf90 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c557fa640 10 client.17985 handle_command_reply: tid=0 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 client.17985 shutdown 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 msgr2=0x55b669adc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 0x55b669adc990 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c09ab40 tx=0x7f8c5c0ce970 comp rx=0 tx=0).stop 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 shutdown_connections 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55b669b4bf90 0x55b669b4e3b0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b669ad48f0 0x55b669ad8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c557fa640 0 client.17985 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b669acbd10 0x55b669adc990 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 --2- 172.21.15.116:0/617109833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55b669acb3f0 0x55b669adc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 >> 172.21.15.116:0/617109833 conn(0x55b669ac5b20 msgr2=0x55b669ad3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 shutdown_connections 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 1 -- 172.21.15.116:0/617109833 wait complete. 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.409+0000 7f8c76791740 20 client.17985 trim_cache size 0 max 16384 2024-03-24T09:30:20.420 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8c580a2230 msgr2=0x7f8c580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8c580a2230 0x7f8c580a46f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8c64023000 tx=0x7f8c64024000 comp rx=0 tx=0).stop 2024-03-24T09:30:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 msgr2=0x7f8c6c19e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c19e750 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00e9f0 tx=0x7f8c5c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 shutdown_connections 2024-03-24T09:30:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8c580a2230 0x7f8c580a46f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8c6c106660 0x7f8c6c1a2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8c6c1041d0 0x7f8c6c19e750 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 --2- 172.21.15.116:0/4290769433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c6c068e50 0x7f8c6c19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 >> 172.21.15.116:0/4290769433 conn(0x7f8c6c077710 msgr2=0x7f8c6c101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 shutdown_connections 2024-03-24T09:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.410+0000 7f8c73bb6640 1 -- 172.21.15.116:0/4290769433 wait complete. 2024-03-24T09:30:20.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:30:20.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:20 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3477636882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:30:20.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 -- 172.21.15.116:0/2609318984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 msgr2=0x7f942810e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 --2- 172.21.15.116:0/2609318984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f942810e1c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f9420009be0 tx=0x7f942002bf80 comp rx=0 tx=0).stop 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 -- 172.21.15.116:0/2609318984 shutdown_connections 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 --2- 172.21.15.116:0/2609318984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f942810e1c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 --2- 172.21.15.116:0/2609318984 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94281032a0 0x7f942810b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 --2- 172.21.15.116:0/2609318984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9428102980 0x7f9428102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 -- 172.21.15.116:0/2609318984 >> 172.21.15.116:0/2609318984 conn(0x7f9428077770 msgr2=0x7f9428077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 -- 172.21.15.116:0/2609318984 shutdown_connections 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.526+0000 7f943031e640 1 -- 172.21.15.116:0/2609318984 wait complete. 2024-03-24T09:30:20.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 Processor -- start 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 -- start start 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9428102980 0x7f942819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94281032a0 0x7f942819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f9428197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9428198100 con 0x7f942810bd50 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9428198270 con 0x7f9428102980 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f943031e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9428198430 con 0x7f94281032a0 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f9428197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f9428197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59048/0 (socket says 172.21.15.116:59048) 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 -- 172.21.15.116:0/174600770 learned_addr learned my addr 172.21.15.116:0/174600770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e0ba640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9428102980 0x7f942819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942d8b9640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94281032a0 0x7f942819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 -- 172.21.15.116:0/174600770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94281032a0 msgr2=0x7f942819ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94281032a0 0x7f942819ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 -- 172.21.15.116:0/174600770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9428102980 msgr2=0x7f942819d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9428102980 0x7f942819d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 -- 172.21.15.116:0/174600770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9420009840 con 0x7f942810bd50 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.527+0000 7f942e8bb640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f9428197ab0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f94200096f0 tx=0x7f9420033c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.528+0000 7f941f7fe640 1 -- 172.21.15.116:0/174600770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420033e30 con 0x7f942810bd50 2024-03-24T09:30:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.528+0000 7f941f7fe640 1 -- 172.21.15.116:0/174600770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9420003b80 con 0x7f942810bd50 2024-03-24T09:30:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.528+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94281986b0 con 0x7f942810bd50 2024-03-24T09:30:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.528+0000 7f941f7fe640 1 -- 172.21.15.116:0/174600770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420034430 con 0x7f942810bd50 2024-03-24T09:30:20.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.528+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94281a4950 con 0x7f942810bd50 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.530+0000 7f941f7fe640 1 -- 172.21.15.116:0/174600770 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9420003cf0 con 0x7f942810bd50 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.530+0000 7f9432fdb740 1 Processor -- start 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.530+0000 7f9432fdb740 1 -- start start 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.530+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f63f0 0x5605ea9f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605eaa01ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f9432fdb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5605ea964840 con 0x5605ea9ff8f0 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f9432fdb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5605ea961d70 con 0x5605ea9f6d10 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f9432fdb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ea95d550 con 0x5605ea9f63f0 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f941f7fe640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94140a2230 0x7f94140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605eaa01ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9ff3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:35568/0 (socket says 172.21.15.116:35568) 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 -- 172.21.15.116:0/3737513333 learned_addr learned my addr 172.21.15.116:0/3737513333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e8bb640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94140a2230 0x7f94140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f63f0 0x5605ea9f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f941f7fe640 1 -- 172.21.15.116:0/174600770 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f94200f0fe0 con 0x7f942810bd50 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 -- 172.21.15.116:0/3737513333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f63f0 msgr2=0x5605ea9f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f63f0 0x5605ea9f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 -- 172.21.15.116:0/3737513333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 msgr2=0x5605eaa01ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605eaa01ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 -- 172.21.15.116:0/3737513333 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ea8f2de0 con 0x5605ea9f6d10 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e8bb640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94140a2230 0x7f94140a46f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f94200ee4e0 tx=0x7f94200b2b60 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f942e0ba640 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9ff3b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9424009c90 tx=0x7f9424011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c385f41790a3ee57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f941cbfe640 1 -- 172.21.15.116:0/3737513333 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424004250 con 0x5605ea9f6d10 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f941cbfe640 1 -- 172.21.15.116:0/3737513333 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9424009e40 con 0x5605ea9f6d10 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.531+0000 7f941cbfe640 1 -- 172.21.15.116:0/3737513333 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424019670 con 0x5605ea9f6d10 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 -- 172.21.15.116:0/3737513333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 msgr2=0x5605ea9ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9ff3b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9424009c90 tx=0x7f9424011810 comp rx=0 tx=0).stop 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 -- 172.21.15.116:0/3737513333 shutdown_connections 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605eaa01ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9ff3b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3737513333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f63f0 0x5605ea9f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 -- 172.21.15.116:0/3737513333 >> 172.21.15.116:0/3737513333 conn(0x5605ea9f0b20 msgr2=0x5605ea9f0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 -- 172.21.15.116:0/3737513333 shutdown_connections 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 -- 172.21.15.116:0/3737513333 wait complete. 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.532+0000 7f9432fdb740 1 -- start start 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 Processor -- start 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f63f0 0x5605ea9fc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605ea9fceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5605ea630430 con 0x5605ea9ff8f0 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5605ea5ece00 con 0x5605ea9f63f0 2024-03-24T09:30:20.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ea8e2590 con 0x5605ea9f6d10 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9432fdb740 10 client.0 _ll_register_callbacks cb 0x5605ea951fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59064/0 (socket says 172.21.15.116:59064) 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 -- 172.21.15.116:0/4129356687 learned_addr learned my addr 172.21.15.116:0/4129356687 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942e0ba640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605ea9fceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f63f0 0x5605ea9fc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 -- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 msgr2=0x5605ea9fceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605ea9fceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 -- 172.21.15.116:0/4129356687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f63f0 msgr2=0x5605ea9fc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f63f0 0x5605ea9fc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 -- 172.21.15.116:0/4129356687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94240098f0 con 0x5605ea9ff8f0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942e0ba640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605ea9fceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f942d8b9640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fd3f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9418009f40 tx=0x7f9418004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac52d31231f5f727 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.533+0000 7f9411ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9411ffb640 1 -- 172.21.15.116:0/4129356687 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418019030 con 0x5605ea9ff8f0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9411ffb640 1 -- 172.21.15.116:0/4129356687 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f9418004030 con 0x5605ea9ff8f0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5605ea65c7e0 con 0x5605ea9ff8f0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9411ffb640 1 -- 172.21.15.116:0/4129356687 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f94180041f0 con 0x5605ea9ff8f0 2024-03-24T09:30:20.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 10 client.17997 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5605ea95bbd0 con 0x5605ea9ff8f0 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9411ffb640 1 -- 172.21.15.116:0/4129356687 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5605ea65c7e0 con 0x5605ea9ff8f0 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9411ffb640 1 -- 172.21.15.116:0/4129356687 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9418012830 con 0x5605ea9ff8f0 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 10 client.17997 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 10 client.17997 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 20 client.17997 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa0fa20 0x5605eaa11e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 4 client.17997 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:20.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5605ea699720 con 0x5605eaa0fa20 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.534+0000 7f942e0ba640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa0fa20 0x5605eaa11e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9411ffb640 10 client.17997 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f942e0ba640 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa0fa20 0x5605eaa11e40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f9df1b536598a1d server_cookie=6c8d8f31d133b20c in_seq=0 out_seq=0 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9411ffb640 1 -- 172.21.15.116:0/4129356687 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5605ea699720 con 0x5605eaa0fa20 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9411ffb640 10 client.17997 handle_command_reply: tid=0 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 client.17997 shutdown 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 msgr2=0x5605ea9fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fd3f0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f9418009f40 tx=0x7f9418004730 comp rx=0 tx=0).stop 2024-03-24T09:30:20.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 shutdown_connections 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa0fa20 0x5605eaa11e40 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fd3f0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605ea9fceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 --2- 172.21.15.116:0/4129356687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f63f0 0x5605ea9fc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9411ffb640 0 client.17997 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 >> 172.21.15.116:0/4129356687 conn(0x5605ea9f0b20 msgr2=0x5605ea9f2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.535+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 shutdown_connections 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.536+0000 7f9432fdb740 1 -- 172.21.15.116:0/4129356687 wait complete. 2024-03-24T09:30:20.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.536+0000 7f9432fdb740 20 client.17997 trim_cache size 0 max 16384 2024-03-24T09:30:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 Processor -- start 2024-03-24T09:30:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 -- start start 2024-03-24T09:30:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9ff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605eaa0b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9ff8f0 0x5605eaa17750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5605ea9f0650 con 0x5605ea9f63f0 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5605eaa0f870 con 0x5605ea9ff8f0 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9432fdb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605ea699720 con 0x5605ea9f6d10 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9ff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9ff0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59080/0 (socket says 172.21.15.116:59080) 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 -- 172.21.15.116:0/3494367004 learned_addr learned my addr 172.21.15.116:0/3494367004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942e0ba640 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605eaa0b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9ff8f0 0x5605eaa17750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 -- 172.21.15.116:0/3494367004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 msgr2=0x5605eaa0b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605eaa0b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 -- 172.21.15.116:0/3494367004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9ff8f0 msgr2=0x5605eaa17750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9ff8f0 0x5605eaa17750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 -- 172.21.15.116:0/3494367004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605ea8f7c80 con 0x5605ea9f63f0 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9ff0a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9418016040 tx=0x7f9418004670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b4fdc5e22f60e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9413fff640 1 -- 172.21.15.116:0/3494367004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418019070 con 0x5605ea9f63f0 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9413fff640 1 -- 172.21.15.116:0/3494367004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9418009530 con 0x5605ea9f63f0 2024-03-24T09:30:20.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.549+0000 7f9413fff640 1 -- 172.21.15.116:0/3494367004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418005040 con 0x5605ea9f63f0 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 -- 172.21.15.116:0/3494367004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 msgr2=0x5605ea9ff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9ff0a0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f9418016040 tx=0x7f9418004670 comp rx=0 tx=0).stop 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 -- 172.21.15.116:0/3494367004 shutdown_connections 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9ff8f0 0x5605eaa17750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9f6d10 0x5605eaa0b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3494367004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9ff0a0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 -- 172.21.15.116:0/3494367004 >> 172.21.15.116:0/3494367004 conn(0x5605ea9f0b20 msgr2=0x5605ea9f70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 -- 172.21.15.116:0/3494367004 shutdown_connections 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 -- 172.21.15.116:0/3494367004 wait complete. 2024-03-24T09:30:20.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 -- start start 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.550+0000 7f9432fdb740 1 Processor -- start 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9f96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9f9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5605eaa74e20 con 0x5605ea9f63f0 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5605eaa74f90 con 0x5605ea9f6d10 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5605eaa75150 con 0x5605ea9ff8f0 2024-03-24T09:30:20.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 10 client.0 _ll_register_callbacks cb 0x7f9418005890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9f96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942e0ba640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9f9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9f96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59082/0 (socket says 172.21.15.116:59082) 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 -- 172.21.15.116:0/3941551204 learned_addr learned my addr 172.21.15.116:0/3941551204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 -- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9ff8f0 msgr2=0x5605ea9fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 -- 172.21.15.116:0/3941551204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 msgr2=0x5605ea9f9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9f9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 -- 172.21.15.116:0/3941551204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605ea95bbd0 con 0x5605ea9f63f0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f942d8b9640 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9f96e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f941800a010 tx=0x7f9418009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5233496c018ed95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9411ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9411ffb640 1 -- 172.21.15.116:0/3941551204 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9418019070 con 0x5605ea9f63f0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9411ffb640 1 -- 172.21.15.116:0/3941551204 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f9418000e40 con 0x5605ea9f63f0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9424009e40 con 0x5605ea9f63f0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9411ffb640 1 -- 172.21.15.116:0/3941551204 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9418005240 con 0x5605ea9f63f0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 10 client.18003 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.551+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5605ea91d920 con 0x5605ea9f63f0 2024-03-24T09:30:20.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9411ffb640 1 -- 172.21.15.116:0/3941551204 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9424009e40 con 0x5605ea9f63f0 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9411ffb640 1 -- 172.21.15.116:0/3941551204 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9418049070 con 0x5605ea9f63f0 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 10 client.18003 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 10 client.18003 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 20 client.18003 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa764e0 0x5605eaa788d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 4 client.18003 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5605ea6e8890 con 0x5605eaa764e0 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f942e0ba640 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa764e0 0x5605eaa788d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9411ffb640 10 client.18003 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f942e0ba640 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa764e0 0x5605eaa788d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e0f9e866ea36956 server_cookie=40c319df0ea3e07d in_seq=0 out_seq=0 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9411ffb640 1 -- 172.21.15.116:0/3941551204 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5605ea6e8890 con 0x5605eaa764e0 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9411ffb640 10 client.18003 handle_command_reply: tid=0 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 1 client.18003 shutdown 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 msgr2=0x5605ea9f96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.552+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9f96e0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f941800a010 tx=0x7f9418009c70 comp rx=0 tx=0).stop 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 shutdown_connections 2024-03-24T09:30:20.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5605eaa764e0 0x5605eaa788d0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5605ea9ff8f0 0x5605ea9fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5605ea9f6d10 0x5605ea9f9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9411ffb640 0 client.18003 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 --2- 172.21.15.116:0/3941551204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5605ea9f63f0 0x5605ea9f96e0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 >> 172.21.15.116:0/3941551204 conn(0x5605ea9f0b20 msgr2=0x5605ea9fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 shutdown_connections 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 1 -- 172.21.15.116:0/3941551204 wait complete. 2024-03-24T09:30:20.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f9432fdb740 20 client.18003 trim_cache size 0 max 16384 2024-03-24T09:30:20.564 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:20.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94140a2230 msgr2=0x7f94140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94140a2230 0x7f94140a46f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f94200ee4e0 tx=0x7f94200b2b60 comp rx=0 tx=0).stop 2024-03-24T09:30:20.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 msgr2=0x7f9428197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f9428197ab0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f94200096f0 tx=0x7f9420033c40 comp rx=0 tx=0).stop 2024-03-24T09:30:20.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 shutdown_connections 2024-03-24T09:30:20.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f942810bd50 0x7f9428197ab0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f94140a2230 0x7f94140a46f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94281032a0 0x7f942819ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 --2- 172.21.15.116:0/174600770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9428102980 0x7f942819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 >> 172.21.15.116:0/174600770 conn(0x7f9428077770 msgr2=0x7f94280fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 shutdown_connections 2024-03-24T09:30:20.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.553+0000 7f943031e640 1 -- 172.21.15.116:0/174600770 wait complete. 2024-03-24T09:30:20.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:30:20.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.669+0000 7f44b2609640 1 -- 172.21.15.116:0/1286667325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 msgr2=0x7f44ac10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.669+0000 7f44b2609640 1 --2- 172.21.15.116:0/1286667325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac10e550 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009cd0 tx=0x7f44a402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 -- 172.21.15.116:0/1286667325 shutdown_connections 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 --2- 172.21.15.116:0/1286667325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac10e550 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 --2- 172.21.15.116:0/1286667325 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac0688f0 0x7f44ac068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 --2- 172.21.15.116:0/1286667325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac107690 0x7f44ac107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 -- 172.21.15.116:0/1286667325 >> 172.21.15.116:0/1286667325 conn(0x7f44ac077710 msgr2=0x7f44ac077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 -- 172.21.15.116:0/1286667325 shutdown_connections 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 -- 172.21.15.116:0/1286667325 wait complete. 2024-03-24T09:30:20.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 Processor -- start 2024-03-24T09:30:20.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 -- start start 2024-03-24T09:30:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac0688f0 0x7f44ac1005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac107690 0x7f44ac101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.670+0000 7f44b2609640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac1a6000 con 0x7f44ac0692b0 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44b2609640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac1a6170 con 0x7f44ac0688f0 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44b2609640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac1a6330 con 0x7f44ac107690 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac100b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59106/0 (socket says 172.21.15.116:59106) 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2428405871 learned_addr learned my addr 172.21.15.116:0/2428405871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44abfff640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac0688f0 0x7f44ac1005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac107690 0x7f44ac101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2428405871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac107690 msgr2=0x7f44ac101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac107690 0x7f44ac101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2428405871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac0688f0 msgr2=0x7f44ac1005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac0688f0 0x7f44ac1005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2428405871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a4009900 con 0x7f44ac0692b0 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac100b10 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f44a000e9f0 tx=0x7f44a000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44a97fa640 1 -- 172.21.15.116:0/2428405871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0009720 con 0x7f44ac0692b0 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44a97fa640 1 -- 172.21.15.116:0/2428405871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a000cd80 con 0x7f44ac0692b0 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac1a6610 con 0x7f44ac0692b0 2024-03-24T09:30:20.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44a97fa640 1 -- 172.21.15.116:0/2428405871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0010430 con 0x7f44ac0692b0 2024-03-24T09:30:20.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.671+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac1a6ad0 con 0x7f44ac0692b0 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44a97fa640 1 -- 172.21.15.116:0/2428405871 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f44a0005020 con 0x7f44ac0692b0 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 Processor -- start 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 -- start start 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a547d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a5d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a5fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5642629c2840 con 0x564262a5d8f0 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5642629bfd70 con 0x564262a54d10 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b51e1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5642629bb550 con 0x564262a543f0 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a5fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.674+0000 7f44b0ba6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44b0ba6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:35730/0 (socket says 172.21.15.116:35730) 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44b0ba6640 1 -- 172.21.15.116:0/3431298046 learned_addr learned my addr 172.21.15.116:0/3431298046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44abfff640 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a5d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44a97fa640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44940a2230 0x7f44940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44ab7fe640 1 -- 172.21.15.116:0/3431298046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 msgr2=0x564262a547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a547d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44ab7fe640 1 -- 172.21.15.116:0/3431298046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 msgr2=0x564262a5d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a5d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44ab7fe640 1 -- 172.21.15.116:0/3431298046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564262950de0 con 0x564262a5d8f0 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44abfff640 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a5d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44940a2230 0x7f44940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44a97fa640 1 -- 172.21.15.116:0/2428405871 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f44a0014070 con 0x7f44ac0692b0 2024-03-24T09:30:20.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a5fce0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f44a00cbb40 tx=0x7f44a0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e6e3dfe55b308c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44927fc640 1 -- 172.21.15.116:0/3431298046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a00046c0 con 0x564262a5d8f0 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44927fc640 1 -- 172.21.15.116:0/3431298046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a009b4e0 con 0x564262a5d8f0 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44940a2230 0x7f44940a46f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44a4005e30 tx=0x7f44a402e040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.675+0000 7f44927fc640 1 -- 172.21.15.116:0/3431298046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a009ef30 con 0x564262a5d8f0 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 -- 172.21.15.116:0/3431298046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 msgr2=0x564262a5fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a5fce0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f44a00cbb40 tx=0x7f44a0021000 comp rx=0 tx=0).stop 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 -- 172.21.15.116:0/3431298046 shutdown_connections 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a5fce0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a5d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3431298046 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 -- 172.21.15.116:0/3431298046 >> 172.21.15.116:0/3431298046 conn(0x564262a4eb20 msgr2=0x564262a4ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 -- 172.21.15.116:0/3431298046 shutdown_connections 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 -- 172.21.15.116:0/3431298046 wait complete. 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.676+0000 7f44b51e1740 1 -- start start 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 Processor -- start 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a543f0 0x564262a5a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 0x564262a5ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a5d8f0 0x564262a5b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56426268e430 con 0x564262a54d10 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56426264ae00 con 0x564262a543f0 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564262940590 con 0x564262a5d8f0 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b51e1740 10 client.0 _ll_register_callbacks cb 0x5642629affc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 0x564262a5ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 0x564262a5ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59112/0 (socket says 172.21.15.116:59112) 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44abfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a543f0 0x564262a5a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2852292027 learned_addr learned my addr 172.21.15.116:0/2852292027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a5d8f0 0x564262a5b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2852292027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a5d8f0 msgr2=0x564262a5b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a5d8f0 0x564262a5b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a543f0 msgr2=0x564262a5a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a543f0 0x564262a5a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 -- 172.21.15.116:0/2852292027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac1a6ad0 con 0x564262a54d10 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44abfff640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a543f0 0x564262a5a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 0x564262a5ae40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f44a0097240 tx=0x7f44a009b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7647ee7effd7af48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f448b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f448b7fe640 1 -- 172.21.15.116:0/2852292027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a00046c0 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.677+0000 7f448b7fe640 1 -- 172.21.15.116:0/2852292027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f44a00a3580 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5642626ba7e0 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f448b7fe640 1 -- 172.21.15.116:0/2852292027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f44a00aeb80 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 10 client.18015 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5642629b9bd0 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f448b7fe640 1 -- 172.21.15.116:0/2852292027 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5642626ba7e0 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f448b7fe640 1 -- 172.21.15.116:0/2852292027 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f44a00d6070 con 0x564262a54d10 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 10 client.18015 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 10 client.18015 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:20.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 20 client.18015 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262a6d630 0x564262a6fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 4 client.18015 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642626f7720 con 0x564262a6d630 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44abfff640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262a6d630 0x564262a6fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f44abfff640 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262a6d630 0x564262a6fa50 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52f9d17c526a9f33 server_cookie=b1244ba0c7358c6 in_seq=0 out_seq=0 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.678+0000 7f448b7fe640 10 client.18015 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.679+0000 7f448b7fe640 1 -- 172.21.15.116:0/2852292027 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642626f7720 con 0x564262a6d630 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.679+0000 7f448b7fe640 10 client.18015 handle_command_reply: tid=0 2024-03-24T09:30:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.679+0000 7f44b51e1740 1 client.18015 shutdown 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 msgr2=0x564262a5ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 0x564262a5ae40 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f44a0097240 tx=0x7f44a009b650 comp rx=0 tx=0).stop 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 shutdown_connections 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a5d8f0 0x564262a5b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a54d10 0x564262a5ae40 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262a6d630 0x564262a6fa50 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2852292027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a543f0 0x564262a5a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f448b7fe640 0 client.18015 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 >> 172.21.15.116:0/2852292027 conn(0x564262a4eb20 msgr2=0x564262a50870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 shutdown_connections 2024-03-24T09:30:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 1 -- 172.21.15.116:0/2852292027 wait complete. 2024-03-24T09:30:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.680+0000 7f44b51e1740 20 client.18015 trim_cache size 0 max 16384 2024-03-24T09:30:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 Processor -- start 2024-03-24T09:30:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 -- start start 2024-03-24T09:30:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a69120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a75360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a65740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564262a3fe20 con 0x564262a5d8f0 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564262a4f6f0 con 0x564262a54d10 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b51e1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564262a700d0 con 0x564262a543f0 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a65740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a65740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59124/0 (socket says 172.21.15.116:59124) 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 -- 172.21.15.116:0/3316678481 learned_addr learned my addr 172.21.15.116:0/3316678481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a75360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44abfff640 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a69120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 -- 172.21.15.116:0/3316678481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 msgr2=0x564262a69120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a69120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 -- 172.21.15.116:0/3316678481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 msgr2=0x564262a75360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a75360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 -- 172.21.15.116:0/3316678481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564262955c80 con 0x564262a5d8f0 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a65740 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009ca0 tx=0x7f44a4042070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed71f479730e9d51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f4491ffb640 1 -- 172.21.15.116:0/3316678481 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4034ea0 con 0x564262a5d8f0 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f4491ffb640 1 -- 172.21.15.116:0/3316678481 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a4032a80 con 0x564262a5d8f0 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.693+0000 7f4491ffb640 1 -- 172.21.15.116:0/3316678481 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a404c420 con 0x564262a5d8f0 2024-03-24T09:30:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 -- 172.21.15.116:0/3316678481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 msgr2=0x564262a65740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a65740 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009ca0 tx=0x7f44a4042070 comp rx=0 tx=0).stop 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 -- 172.21.15.116:0/3316678481 shutdown_connections 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a5d8f0 0x564262a65740 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a54d10 0x564262a75360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 --2- 172.21.15.116:0/3316678481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a543f0 0x564262a69120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 -- 172.21.15.116:0/3316678481 >> 172.21.15.116:0/3316678481 conn(0x564262a4eb20 msgr2=0x564262a5a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 -- 172.21.15.116:0/3316678481 shutdown_connections 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 -- 172.21.15.116:0/3316678481 wait complete. 2024-03-24T09:30:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.694+0000 7f44b51e1740 1 -- start start 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 Processor -- start 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 0x564262a61840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a54d10 0x564262a61d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a5d8f0 0x564262ad2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564262ad2ed0 con 0x564262a543f0 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564262ad3010 con 0x564262a5d8f0 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x564262ad31a0 con 0x564262a54d10 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b51e1740 10 client.0 _ll_register_callbacks cb 0x7f449c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:20.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 0x564262a61840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 0x564262a61840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59130/0 (socket says 172.21.15.116:59130) 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 -- 172.21.15.116:0/2098927154 learned_addr learned my addr 172.21.15.116:0/2098927154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a54d10 0x564262a61d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44b0ba6640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a5d8f0 0x564262ad2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 -- 172.21.15.116:0/2098927154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a54d10 msgr2=0x564262a61d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a54d10 0x564262a61d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 -- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a5d8f0 msgr2=0x564262ad2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a5d8f0 0x564262ad2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 -- 172.21.15.116:0/2098927154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a4030070 con 0x564262a543f0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f44abfff640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 0x564262a61840 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x564262a70300 tx=0x7f449c012cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe8c7227733ffb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f448b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.695+0000 7f448b7fe640 1 -- 172.21.15.116:0/2098927154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c027070 con 0x564262a543f0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f448b7fe640 1 -- 172.21.15.116:0/2098927154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f449c01f8a0 con 0x564262a543f0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44a009b4e0 con 0x564262a543f0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f448b7fe640 1 -- 172.21.15.116:0/2098927154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f449c017040 con 0x564262a543f0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 10 client.18021 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56426297b920 con 0x564262a543f0 2024-03-24T09:30:20.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f448b7fe640 1 -- 172.21.15.116:0/2098927154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f44a009b4e0 con 0x564262a543f0 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f448b7fe640 1 -- 172.21.15.116:0/2098927154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f449c02a020 con 0x564262a543f0 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 10 client.18021 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 10 client.18021 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 20 client.18021 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262ad4a70 0x564262ad6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 4 client.18021 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564262746890 con 0x564262ad4a70 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262ad4a70 0x564262ad6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:20.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f44ab7fe640 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262ad4a70 0x564262ad6e90 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fe2bedc3f376a31 server_cookie=c258a6ad8aee4e7e in_seq=0 out_seq=0 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.696+0000 7f448b7fe640 10 client.18021 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f448b7fe640 1 -- 172.21.15.116:0/2098927154 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564262746890 con 0x564262ad4a70 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f448b7fe640 10 client.18021 handle_command_reply: tid=0 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 client.18021 shutdown 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 msgr2=0x564262a61840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 0x564262a61840 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x564262a70300 tx=0x7f449c012cf0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 shutdown_connections 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x564262ad4a70 0x564262ad6e90 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564262a5d8f0 0x564262ad2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f448b7fe640 0 client.18021 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x564262a54d10 0x564262a61d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 --2- 172.21.15.116:0/2098927154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564262a543f0 0x564262a61840 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 >> 172.21.15.116:0/2098927154 conn(0x564262a4eb20 msgr2=0x564262a5a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 shutdown_connections 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 1 -- 172.21.15.116:0/2098927154 wait complete. 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b51e1740 20 client.18021 trim_cache size 0 max 16384 2024-03-24T09:30:20.707 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44940a2230 msgr2=0x7f44940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b2609640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44940a2230 0x7f44940a46f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44a4005e30 tx=0x7f44a402e040 comp rx=0 tx=0).stop 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 msgr2=0x7f44ac100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.697+0000 7f44b2609640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac100b10 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f44a000e9f0 tx=0x7f44a000edf0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 shutdown_connections 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f44940a2230 0x7f44940a46f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f44ac107690 0x7f44ac101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44ac0692b0 0x7f44ac100b10 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 --2- 172.21.15.116:0/2428405871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44ac0688f0 0x7f44ac1005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 >> 172.21.15.116:0/2428405871 conn(0x7f44ac077710 msgr2=0x7f44ac0fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 shutdown_connections 2024-03-24T09:30:20.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:20.698+0000 7f44b2609640 1 -- 172.21.15.116:0/2428405871 wait complete. 2024-03-24T09:30:21.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:21 smithi117 ceph-mon[35788]: pgmap v254: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:21.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:21 smithi116 ceph-mon[32207]: pgmap v254: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:21 smithi143 ceph-mon[35965]: pgmap v254: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:30:21.711 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:30:21.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:30:21.711 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:30:21.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 -- 172.21.15.116:0/3459367106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 msgr2=0x7ff38010e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 --2- 172.21.15.116:0/3459367106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff38010e150 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7ff378009ca0 tx=0x7ff37802c180 comp rx=0 tx=0).stop 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 -- 172.21.15.116:0/3459367106 shutdown_connections 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 --2- 172.21.15.116:0/3459367106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff38010e150 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 --2- 172.21.15.116:0/3459367106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 0x7ff38010b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 --2- 172.21.15.116:0/3459367106 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff380102950 0x7ff380102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 -- 172.21.15.116:0/3459367106 >> 172.21.15.116:0/3459367106 conn(0x7ff380076990 msgr2=0x7ff380076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 -- 172.21.15.116:0/3459367106 shutdown_connections 2024-03-24T09:30:21.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 -- 172.21.15.116:0/3459367106 wait complete. 2024-03-24T09:30:21.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 Processor -- start 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.828+0000 7ff385c9a640 1 -- start start 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff380102950 0x7ff38019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 0x7ff38019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff380197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff380198070 con 0x7ff38010bd20 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801981e0 con 0x7ff380103270 2024-03-24T09:30:21.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801983a0 con 0x7ff380102950 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff380197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff380197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59150/0 (socket says 172.21.15.116:59150) 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 -- 172.21.15.116:0/2595919084 learned_addr learned my addr 172.21.15.116:0/2595919084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37effd640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 0x7ff38019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff380102950 0x7ff38019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 -- 172.21.15.116:0/2595919084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff380102950 msgr2=0x7ff38019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff380102950 0x7ff38019d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 -- 172.21.15.116:0/2595919084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 msgr2=0x7ff38019ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 0x7ff38019ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 -- 172.21.15.116:0/2595919084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff374009490 con 0x7ff38010bd20 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37effd640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 0x7ff38019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37ffff640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff380197a20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff37802c790 tx=0x7ff378034930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37cff9640 1 -- 172.21.15.116:0/2595919084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff378003d80 con 0x7ff38010bd20 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff378009900 con 0x7ff38010bd20 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37cff9640 1 -- 172.21.15.116:0/2595919084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff378032b30 con 0x7ff38010bd20 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.829+0000 7ff37cff9640 1 -- 172.21.15.116:0/2595919084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff378004670 con 0x7ff38010bd20 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.830+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3801a4730 con 0x7ff38010bd20 2024-03-24T09:30:21.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff37cff9640 1 -- 172.21.15.116:0/2595919084 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7ff37803b040 con 0x7ff38010bd20 2024-03-24T09:30:21.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 Processor -- start 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 -- start start 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed22b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff37cff9640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff3680a2230 0x7ff3680a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed2b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed2e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55591ec90530 con 0x55591ed23080 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55591ec8da60 con 0x55591ed22760 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.832+0000 7ff388877740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55591ec89240 con 0x55591ed2bc60 2024-03-24T09:30:21.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff3680a2230 0x7ff3680a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed2b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed2e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed2b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59154/0 (socket says 172.21.15.116:59154) 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37effd640 1 -- 172.21.15.116:0/493667595 learned_addr learned my addr 172.21.15.116:0/493667595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37effd640 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed22b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37cff9640 1 -- 172.21.15.116:0/2595919084 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7ff3780f0d50 con 0x7ff38010bd20 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 -- 172.21.15.116:0/493667595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 msgr2=0x55591ed22b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed22b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 -- 172.21.15.116:0/493667595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 msgr2=0x55591ed2b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed2b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 -- 172.21.15.116:0/493667595 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55591ec878c0 con 0x55591ed2bc60 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff3680a2230 0x7ff3680a46f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff380198b60 tx=0x7ff370005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff37ffff640 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed2e050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff3780ee140 tx=0x7ff3780095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40b9d41bcd42ab70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff365ffb640 1 -- 172.21.15.116:0/493667595 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3780ca590 con 0x55591ed2bc60 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff365ffb640 1 -- 172.21.15.116:0/493667595 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff3780cfe10 con 0x55591ed2bc60 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.833+0000 7ff365ffb640 1 -- 172.21.15.116:0/493667595 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3780d1070 con 0x55591ed2bc60 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 -- 172.21.15.116:0/493667595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 msgr2=0x55591ed2e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed2e050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff3780ee140 tx=0x7ff3780095c0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 -- 172.21.15.116:0/493667595 shutdown_connections 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed2e050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed2b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 --2- 172.21.15.116:0/493667595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed22b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 -- 172.21.15.116:0/493667595 >> 172.21.15.116:0/493667595 conn(0x55591ed1ce90 msgr2=0x55591ed1d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 -- 172.21.15.116:0/493667595 shutdown_connections 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 -- 172.21.15.116:0/493667595 wait complete. 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.834+0000 7ff388877740 1 -- start start 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 Processor -- start 2024-03-24T09:30:21.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed28c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed29140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed29680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55591ec2df80 con 0x55591ed23080 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55591e95c4b0 con 0x55591ed22760 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55591ec0ead0 con 0x55591ed2bc60 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff388877740 10 client.0 _ll_register_callbacks cb 0x55591ec41b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed29140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed28c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed29140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59156/0 (socket says 172.21.15.116:59156) 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37effd640 1 -- 172.21.15.116:0/1382099524 learned_addr learned my addr 172.21.15.116:0/1382099524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37ffff640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed29680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 -- 172.21.15.116:0/1382099524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 msgr2=0x55591ed29680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed29680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 -- 172.21.15.116:0/1382099524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 msgr2=0x55591ed29140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed29140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 -- 172.21.15.116:0/1382099524 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3801a4730 con 0x55591ed22760 2024-03-24T09:30:21.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37effd640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed29140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed28c00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff370003920 tx=0x7ff370004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bb52eb6f948ed09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.835+0000 7ff35effd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff35effd640 1 -- 172.21.15.116:0/1382099524 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff370003ea0 con 0x55591ed22760 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff35effd640 1 -- 172.21.15.116:0/1382099524 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7ff37000ecc0 con 0x55591ed22760 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55591ecad7a0 con 0x55591ed22760 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff35effd640 1 -- 172.21.15.116:0/1382099524 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff37000ee80 con 0x55591ed22760 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 10 client.27848 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55591ec3f6d0 con 0x55591ed22760 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff35effd640 1 -- 172.21.15.116:0/1382099524 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55591ecad7a0 con 0x55591ed22760 2024-03-24T09:30:21.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff35effd640 1 -- 172.21.15.116:0/1382099524 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff370024020 con 0x55591ed22760 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 10 client.27848 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 10 client.27848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 20 client.27848 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591ed3bb30 0x55591ed3df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 4 client.27848 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55591ec7f1f0 con 0x55591ed3bb30 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.836+0000 7ff37effd640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591ed3bb30 0x55591ed3df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.837+0000 7ff35effd640 10 client.27848 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:21.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.837+0000 7ff37effd640 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591ed3bb30 0x55591ed3df70 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d80b7d06d450271f server_cookie=b998197efb062746 in_seq=0 out_seq=0 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.837+0000 7ff35effd640 1 -- 172.21.15.116:0/1382099524 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55591ec7f1f0 con 0x55591ed3bb30 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.837+0000 7ff35effd640 10 client.27848 handle_command_reply: tid=0 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.837+0000 7ff388877740 1 client.27848 shutdown 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 msgr2=0x55591ed28c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed28c00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff370003920 tx=0x7ff370004730 comp rx=0 tx=0).stop 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 shutdown_connections 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed29680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed29140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591ed3bb30 0x55591ed3df70 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 --2- 172.21.15.116:0/1382099524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed28c00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 >> 172.21.15.116:0/1382099524 conn(0x55591ed1ce90 msgr2=0x55591ed1ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff35effd640 0 client.27848 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 shutdown_connections 2024-03-24T09:30:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 1 -- 172.21.15.116:0/1382099524 wait complete. 2024-03-24T09:30:21.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.838+0000 7ff388877740 20 client.27848 trim_cache size 0 max 16384 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.855+0000 7ff388877740 1 Processor -- start 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 -- start start 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed372b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed43880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed294a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55591ed1dc90 con 0x55591ed23080 2024-03-24T09:30:21.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55591ed377f0 con 0x55591ed22760 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff388877740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55591ed221a0 con 0x55591ed2bc60 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed43880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed43880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59168/0 (socket says 172.21.15.116:59168) 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 -- 172.21.15.116:0/3444542576 learned_addr learned my addr 172.21.15.116:0/3444542576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed372b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37ffff640 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed294a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 -- 172.21.15.116:0/3444542576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 msgr2=0x55591ed294a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed294a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 -- 172.21.15.116:0/3444542576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 msgr2=0x55591ed372b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed372b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 -- 172.21.15.116:0/3444542576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55591ec23c60 con 0x55591ed23080 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed372b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff37effd640 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed43880 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff374013010 tx=0x7ff374009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f931cb5103730a29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff3657fa640 1 -- 172.21.15.116:0/3444542576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374016040 con 0x55591ed23080 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff3657fa640 1 -- 172.21.15.116:0/3444542576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff374026070 con 0x55591ed23080 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.856+0000 7ff3657fa640 1 -- 172.21.15.116:0/3444542576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374004a30 con 0x55591ed23080 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 -- 172.21.15.116:0/3444542576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 msgr2=0x55591ed43880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed43880 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7ff374013010 tx=0x7ff374009770 comp rx=0 tx=0).stop 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 -- 172.21.15.116:0/3444542576 shutdown_connections 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed294a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed23080 0x55591ed43880 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 --2- 172.21.15.116:0/3444542576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed22760 0x55591ed372b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 -- 172.21.15.116:0/3444542576 >> 172.21.15.116:0/3444542576 conn(0x55591ed1ce90 msgr2=0x55591ed1bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 -- 172.21.15.116:0/3444542576 shutdown_connections 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 -- 172.21.15.116:0/3444542576 wait complete. 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.857+0000 7ff388877740 1 -- start start 2024-03-24T09:30:21.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 Processor -- start 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 0x55591ed338e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed23080 0x55591ed308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed30df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55591ed33e70 con 0x55591ed22760 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55591ed33fe0 con 0x55591ed23080 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55591ed31330 con 0x55591ed2bc60 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff388877740 10 client.0 _ll_register_callbacks cb 0x7ff3700035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 0x55591ed338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 0x55591ed338e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59182/0 (socket says 172.21.15.116:59182) 2024-03-24T09:30:21.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 -- 172.21.15.116:0/4049565459 learned_addr learned my addr 172.21.15.116:0/4049565459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed30df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37effd640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed23080 0x55591ed308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 -- 172.21.15.116:0/4049565459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 msgr2=0x55591ed30df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed30df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 -- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed23080 msgr2=0x55591ed308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed23080 0x55591ed308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 -- 172.21.15.116:0/4049565459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55591ec3f6d0 con 0x55591ed22760 2024-03-24T09:30:21.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37effd640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed23080 0x55591ed308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37ffff640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed30df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff37f7fe640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 0x55591ed338e0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff3700118d0 tx=0x7ff370011950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc34149e0d9ea481 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.858+0000 7ff35effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff35effd640 1 -- 172.21.15.116:0/4049565459 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37000ed50 con 0x55591ed22760 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff35effd640 1 -- 172.21.15.116:0/4049565459 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7ff370034430 con 0x55591ed22760 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3780cfe10 con 0x55591ed22760 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff35effd640 1 -- 172.21.15.116:0/4049565459 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3700345f0 con 0x55591ed22760 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 10 client.18039 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:21.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7ff374019040 con 0x55591ed22760 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff35effd640 1 -- 172.21.15.116:0/4049565459 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7ff370045020 con 0x55591ed22760 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff35effd640 1 -- 172.21.15.116:0/4049565459 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7ff370024040 con 0x55591ed22760 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 10 client.18039 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 10 client.18039 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 20 client.18039 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591eda30c0 0x55591eda54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 4 client.18039 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55591e988860 con 0x55591eda30c0 2024-03-24T09:30:21.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.859+0000 7ff37effd640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591eda30c0 0x55591eda54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.860+0000 7ff35effd640 10 client.18039 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:21.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.860+0000 7ff37effd640 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591eda30c0 0x55591eda54e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a5c150bdd761d5 server_cookie=2bf3094091fd5e86 in_seq=0 out_seq=0 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff35effd640 1 -- 172.21.15.116:0/4049565459 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55591e988860 con 0x55591eda30c0 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff35effd640 10 client.18039 handle_command_reply: tid=0 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 client.18039 shutdown 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 msgr2=0x55591ed338e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 0x55591ed338e0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7ff3700118d0 tx=0x7ff370011950 comp rx=0 tx=0).stop 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 shutdown_connections 2024-03-24T09:30:21.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55591ed2bc60 0x55591ed30df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55591eda30c0 0x55591eda54e0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55591ed23080 0x55591ed308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 --2- 172.21.15.116:0/4049565459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55591ed22760 0x55591ed338e0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff35effd640 0 client.18039 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 >> 172.21.15.116:0/4049565459 conn(0x55591ed1ce90 msgr2=0x55591ed1e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 shutdown_connections 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.861+0000 7ff388877740 1 -- 172.21.15.116:0/4049565459 wait complete. 2024-03-24T09:30:21.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff388877740 20 client.18039 trim_cache size 0 max 16384 2024-03-24T09:30:21.872 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:30:21.872 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:30:21.872 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "ff9fd113-a7fd-4faf-8816-2438b2f652df", 2024-03-24T09:30:21.872 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:30:21.872 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff3680a2230 msgr2=0x7ff3680a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff3680a2230 0x7ff3680a46f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff380198b60 tx=0x7ff370005f90 comp rx=0 tx=0).stop 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 msgr2=0x7ff380197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff380197a20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7ff37802c790 tx=0x7ff378034930 comp rx=0 tx=0).stop 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 shutdown_connections 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff38010bd20 0x7ff380197a20 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff380103270 0x7ff38019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7ff3680a2230 0x7ff3680a46f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 --2- 172.21.15.116:0/2595919084 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff380102950 0x7ff38019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 >> 172.21.15.116:0/2595919084 conn(0x7ff380076990 msgr2=0x7ff3800fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 shutdown_connections 2024-03-24T09:30:21.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.862+0000 7ff385c9a640 1 -- 172.21.15.116:0/2595919084 wait complete. 2024-03-24T09:30:21.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:30:21.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:30:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 -- 172.21.15.116:0/702111139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154102050 msgr2=0x7f015410e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 --2- 172.21.15.116:0/702111139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154102050 0x7f015410e560 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f014c009cd0 tx=0x7f014c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 -- 172.21.15.116:0/702111139 shutdown_connections 2024-03-24T09:30:21.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 --2- 172.21.15.116:0/702111139 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154102050 0x7f015410e560 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 --2- 172.21.15.116:0/702111139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0154101690 0x7f0154101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 --2- 172.21.15.116:0/702111139 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f01541076a0 0x7f0154107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 -- 172.21.15.116:0/702111139 >> 172.21.15.116:0/702111139 conn(0x7f01540fd1a0 msgr2=0x7f01540ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 -- 172.21.15.116:0/702111139 shutdown_connections 2024-03-24T09:30:21.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.993+0000 7f0159c8d640 1 -- 172.21.15.116:0/702111139 wait complete. 2024-03-24T09:30:21.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.994+0000 7f0159c8d640 1 Processor -- start 2024-03-24T09:30:21.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.994+0000 7f0159c8d640 1 -- start start 2024-03-24T09:30:21.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0159c8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 0x7f0154199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0159c8d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0154102050 0x7f015419a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0159c8d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01541076a0 0x7f015419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0159c8d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f015419ed20 con 0x7f0154101690 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0159c8d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f015419ee90 con 0x7f01541076a0 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0159c8d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f015419f050 con 0x7f0154102050 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 0x7f0154199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 0x7f0154199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59194/0 (socket says 172.21.15.116:59194) 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 -- 172.21.15.116:0/3961599544 learned_addr learned my addr 172.21.15.116:0/3961599544 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0152ffd640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0154102050 0x7f015419a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f0153fff640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01541076a0 0x7f015419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:21.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 -- 172.21.15.116:0/3961599544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0154102050 msgr2=0x7f015419a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0154102050 0x7f015419a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 -- 172.21.15.116:0/3961599544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01541076a0 msgr2=0x7f015419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01541076a0 0x7f015419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 -- 172.21.15.116:0/3961599544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014c009900 con 0x7f0154101690 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.995+0000 7f01537fe640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 0x7f0154199e60 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f014400d770 tx=0x7f014400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.996+0000 7f0150ff9640 1 -- 172.21.15.116:0/3961599544 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144009c30 con 0x7f0154101690 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.996+0000 7f0150ff9640 1 -- 172.21.15.116:0/3961599544 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f014400de80 con 0x7f0154101690 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.996+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015419f330 con 0x7f0154101690 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.996+0000 7f0150ff9640 1 -- 172.21.15.116:0/3961599544 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144004e00 con 0x7f0154101690 2024-03-24T09:30:21.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.996+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01541a6bb0 con 0x7f0154101690 2024-03-24T09:30:22.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.998+0000 7f0150ff9640 1 -- 172.21.15.116:0/3961599544 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f0144010460 con 0x7f0154101690 2024-03-24T09:30:22.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 Processor -- start 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 -- start start 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f0150ff9640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f013c0a2230 0x7f013c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 0x5626c84065b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c840f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c8411ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5626c832af30 con 0x5626c84061d0 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5626c8374810 con 0x5626c840f6d0 2024-03-24T09:30:22.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f015c86a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626c8371d40 con 0x5626c8406af0 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f0153fff640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f013c0a2230 0x7f013c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f01537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 0x5626c84065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f0150ff9640 1 -- 172.21.15.116:0/3961599544 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f01440cdf70 con 0x7f0154101690 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f01537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 0x5626c84065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59196/0 (socket says 172.21.15.116:59196) 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f01537fe640 1 -- 172.21.15.116:0/1438905112 learned_addr learned my addr 172.21.15.116:0/1438905112 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f0153fff640 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c8411ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:21.999+0000 7f0152ffd640 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c840f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f01537fe640 1 -- 172.21.15.116:0/1438905112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 msgr2=0x5626c840f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f01537fe640 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c840f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f01537fe640 1 -- 172.21.15.116:0/1438905112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 msgr2=0x5626c8411ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f01537fe640 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c8411ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f01537fe640 1 -- 172.21.15.116:0/1438905112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626c832eea0 con 0x5626c84061d0 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f0153fff640 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c8411ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f01537fe640 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 0x5626c84065b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f0144096f20 tx=0x7f01440961b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4781d9a71deaaf38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f0153fff640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f013c0a2230 0x7f013c0a46f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f014c006010 tx=0x7f014c005d80 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f0139ffb640 1 -- 172.21.15.116:0/1438905112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144077b70 con 0x5626c84061d0 2024-03-24T09:30:22.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f0139ffb640 1 -- 172.21.15.116:0/1438905112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0144077990 con 0x5626c84061d0 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.000+0000 7f0139ffb640 1 -- 172.21.15.116:0/1438905112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014409e6a0 con 0x5626c84061d0 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 -- 172.21.15.116:0/1438905112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 msgr2=0x5626c84065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 0x5626c84065b0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f0144096f20 tx=0x7f01440961b0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 -- 172.21.15.116:0/1438905112 shutdown_connections 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c8411ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c840f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 --2- 172.21.15.116:0/1438905112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c84061d0 0x5626c84065b0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 -- 172.21.15.116:0/1438905112 >> 172.21.15.116:0/1438905112 conn(0x5626c8400900 msgr2=0x5626c8400ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 -- 172.21.15.116:0/1438905112 shutdown_connections 2024-03-24T09:30:22.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 -- 172.21.15.116:0/1438905112 wait complete. 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.001+0000 7f015c86a740 1 -- start start 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 Processor -- start 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 0x5626c840c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840f6d0 0x5626c840cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840d2a0 0x5626c841b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5626c836d520 con 0x5626c8406af0 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5626c8040430 con 0x5626c840d2a0 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626c7ffce00 con 0x5626c840f6d0 2024-03-24T09:30:22.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f01537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 0x5626c840c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f01537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 0x5626c840c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59198/0 (socket says 172.21.15.116:59198) 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f01537fe640 1 -- 172.21.15.116:0/2683559410 learned_addr learned my addr 172.21.15.116:0/2683559410 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f0152ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840f6d0 0x5626c840cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f0153fff640 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840d2a0 0x5626c841b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.002+0000 7f015c86a740 10 client.0 _ll_register_callbacks cb 0x5626c830bd90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f01537fe640 1 -- 172.21.15.116:0/2683559410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840f6d0 msgr2=0x5626c840cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f01537fe640 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840f6d0 0x5626c840cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f01537fe640 1 -- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840d2a0 msgr2=0x5626c841b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f01537fe640 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840d2a0 0x5626c841b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f01537fe640 1 -- 172.21.15.116:0/2683559410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01541a6bb0 con 0x5626c8406af0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f01537fe640 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 0x5626c840c820 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f014400d740 tx=0x7f014409a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566a31f15242638c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f0132ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f0132ffd640 1 -- 172.21.15.116:0/2683559410 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014409ac50 con 0x5626c8406af0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f0132ffd640 1 -- 172.21.15.116:0/2683559410 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f01440a2cf0 con 0x5626c8406af0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5626c806c7e0 con 0x5626c8406af0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f0132ffd640 1 -- 172.21.15.116:0/2683559410 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f01440b3070 con 0x5626c8406af0 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.003+0000 7f015c86a740 10 client.18051 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:22.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626c836bba0 con 0x5626c8406af0 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f0132ffd640 1 -- 172.21.15.116:0/2683559410 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5626c806c7e0 con 0x5626c8406af0 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f0132ffd640 1 -- 172.21.15.116:0/2683559410 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f01440999a0 con 0x5626c8406af0 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 10 client.18051 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 10 client.18051 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 20 client.18051 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c841fc00 0x5626c8422040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 4 client.18051 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626c80a9720 con 0x5626c841fc00 2024-03-24T09:30:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.004+0000 7f0152ffd640 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c841fc00 0x5626c8422040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.005+0000 7f0152ffd640 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c841fc00 0x5626c8422040 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed1afc65d8f8ecd5 server_cookie=1a42b2463881f878 in_seq=0 out_seq=0 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.005+0000 7f0132ffd640 10 client.18051 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f0132ffd640 1 -- 172.21.15.116:0/2683559410 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5626c80a9720 con 0x5626c841fc00 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f0132ffd640 10 client.18051 handle_command_reply: tid=0 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 client.18051 shutdown 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 msgr2=0x5626c840c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 0x5626c840c820 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f014400d740 tx=0x7f014409a5d0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 shutdown_connections 2024-03-24T09:30:22.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c841fc00 0x5626c8422040 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840d2a0 0x5626c841b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840f6d0 0x5626c840cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 --2- 172.21.15.116:0/2683559410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c8406af0 0x5626c840c820 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f0132ffd640 0 client.18051 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 >> 172.21.15.116:0/2683559410 conn(0x5626c8400900 msgr2=0x5626c83ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 shutdown_connections 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 1 -- 172.21.15.116:0/2683559410 wait complete. 2024-03-24T09:30:22.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.006+0000 7f015c86a740 20 client.18051 trim_cache size 0 max 16384 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 Processor -- start 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 -- start start 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c8427910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 0x5626c841af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c841b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5626c82f2430 con 0x5626c840d2a0 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5626c840d7e0 con 0x5626c840f6d0 2024-03-24T09:30:22.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f015c86a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626c84226c0 con 0x5626c8406af0 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 0x5626c841af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 0x5626c841af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59214/0 (socket says 172.21.15.116:59214) 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 -- 172.21.15.116:0/190978602 learned_addr learned my addr 172.21.15.116:0/190978602 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0153fff640 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c841b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f01537fe640 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c8427910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 -- 172.21.15.116:0/190978602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 msgr2=0x5626c8427910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c8427910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 -- 172.21.15.116:0/190978602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 msgr2=0x5626c841b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.030+0000 7f0152ffd640 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c841b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.031+0000 7f0152ffd640 1 -- 172.21.15.116:0/190978602 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626c8307b20 con 0x5626c840d2a0 2024-03-24T09:30:22.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.031+0000 7f0152ffd640 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 0x5626c841af60 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x5626c840c080 tx=0x7f0148005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587858b92831f378 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:22.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.031+0000 7f01397fa640 1 -- 172.21.15.116:0/190978602 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148017030 con 0x5626c840d2a0 2024-03-24T09:30:22.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.031+0000 7f01397fa640 1 -- 172.21.15.116:0/190978602 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f0148018040 con 0x5626c840d2a0 2024-03-24T09:30:22.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.031+0000 7f01397fa640 1 -- 172.21.15.116:0/190978602 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014801e390 con 0x5626c840d2a0 2024-03-24T09:30:22.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:22 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:30:22.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:22 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:22.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:22 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:22.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:22 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:22.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 -- 172.21.15.116:0/190978602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 msgr2=0x5626c841af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:22.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 0x5626c841af60 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x5626c840c080 tx=0x7f0148005f90 comp rx=0 tx=0).stop 2024-03-24T09:30:22.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 -- 172.21.15.116:0/190978602 shutdown_connections 2024-03-24T09:30:22.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c840f6d0 0x5626c841b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840d2a0 0x5626c841af60 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 --2- 172.21.15.116:0/190978602 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c8406af0 0x5626c8427910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 -- 172.21.15.116:0/190978602 >> 172.21.15.116:0/190978602 conn(0x5626c8400900 msgr2=0x5626c8407010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 -- 172.21.15.116:0/190978602 shutdown_connections 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.032+0000 7f015c86a740 1 -- 172.21.15.116:0/190978602 wait complete. 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 -- start start 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 Processor -- start 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c8406af0 0x5626c8417880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840d2a0 0x5626c8409130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 0x5626c84096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5626c8409db0 con 0x5626c840f6d0 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5626c8409f20 con 0x5626c8406af0 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5626c8484f40 con 0x5626c840d2a0 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f015c86a740 10 client.0 _ll_register_callbacks cb 0x7f0148009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.033+0000 7f0153fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 0x5626c84096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c8406af0 0x5626c8417880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f0153fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 0x5626c84096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59230/0 (socket says 172.21.15.116:59230) 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f0153fff640 1 -- 172.21.15.116:0/3356377546 learned_addr learned my addr 172.21.15.116:0/3356377546 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:22.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f0152ffd640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840d2a0 0x5626c8409130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 -- 172.21.15.116:0/3356377546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840d2a0 msgr2=0x5626c8409130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840d2a0 0x5626c8409130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 -- 172.21.15.116:0/3356377546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 msgr2=0x5626c84096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 0x5626c84096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 -- 172.21.15.116:0/3356377546 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0148009e30 con 0x5626c8406af0 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f0152ffd640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840d2a0 0x5626c8409130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f0153fff640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 0x5626c84096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f01537fe640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c8406af0 0x5626c8417880 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0144096f20 tx=0x7f0144005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f9a9963df2f3709 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:22.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.034+0000 7f0132ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:30:22.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.036+0000 7f0132ffd640 1 -- 172.21.15.116:0/3356377546 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0144099eb0 con 0x5626c8406af0 2024-03-24T09:30:22.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.036+0000 7f0132ffd640 1 -- 172.21.15.116:0/3356377546 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f014409ac50 con 0x5626c8406af0 2024-03-24T09:30:22.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.036+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0144077990 con 0x5626c8406af0 2024-03-24T09:30:22.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f0132ffd640 1 -- 172.21.15.116:0/3356377546 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f01440a2cf0 con 0x5626c8406af0 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 10 client.27872 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5626c832d700 con 0x5626c8406af0 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f0132ffd640 1 -- 172.21.15.116:0/3356377546 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f014409e6a0 con 0x5626c8406af0 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f0132ffd640 1 -- 172.21.15.116:0/3356377546 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f01440cb530 con 0x5626c8406af0 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 10 client.27872 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 10 client.27872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 20 client.27872 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c8486410 0x5626c8488830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:22.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 4 client.27872 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:30:22.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.044+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f0144077990 con 0x5626c8486410 2024-03-24T09:30:22.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.045+0000 7f0152ffd640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c8486410 0x5626c8488830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:22.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.045+0000 7f0132ffd640 10 client.27872 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:22.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.045+0000 7f0152ffd640 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c8486410 0x5626c8488830 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d90e3c1506e8c79 server_cookie=76e986b1df6fc7ac in_seq=0 out_seq=0 2024-03-24T09:30:22.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.045+0000 7f0132ffd640 1 -- 172.21.15.116:0/3356377546 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f0144077990 con 0x5626c8486410 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.045+0000 7f0132ffd640 10 client.27872 handle_command_reply: tid=0 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.045+0000 7f015c86a740 1 client.27872 shutdown 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c8406af0 msgr2=0x5626c8417880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c8406af0 0x5626c8417880 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0144096f20 tx=0x7f0144005b20 comp rx=0 tx=0).stop 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 shutdown_connections 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626c840f6d0 0x5626c84096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5626c840d2a0 0x5626c8409130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5626c8486410 0x5626c8488830 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 --2- 172.21.15.116:0/3356377546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5626c8406af0 0x5626c8417880 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f0132ffd640 0 client.27872 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 >> 172.21.15.116:0/3356377546 conn(0x5626c8400900 msgr2=0x5626c840b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 shutdown_connections 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 1 -- 172.21.15.116:0/3356377546 wait complete. 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.046+0000 7f015c86a740 20 client.27872 trim_cache size 0 max 16384 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:30:22.049 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: "ff9fd113-a7fd-4faf-8816-2438b2f652df": { 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "ff9fd113-a7fd-4faf-8816-2438b2f652df", 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:30:22.050 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:30:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f013c0a2230 msgr2=0x7f013c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f013c0a2230 0x7f013c0a46f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f014c006010 tx=0x7f014c005d80 comp rx=0 tx=0).stop 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 msgr2=0x7f0154199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 0x7f0154199e60 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f014400d770 tx=0x7f014400dc30 comp rx=0 tx=0).stop 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 shutdown_connections 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01541076a0 0x7f015419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f013c0a2230 0x7f013c0a46f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0154102050 0x7f015419a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 --2- 172.21.15.116:0/3961599544 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154101690 0x7f0154199e60 secure :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f014400d770 tx=0x7f014400dc30 comp rx=0 tx=0).stop 2024-03-24T09:30:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 >> 172.21.15.116:0/3961599544 conn(0x7f01540fd1a0 msgr2=0x7f015410cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 shutdown_connections 2024-03-24T09:30:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:22.047+0000 7f0159c8d640 1 -- 172.21.15.116:0/3961599544 wait complete. 2024-03-24T09:30:22.060 INFO:tasks.cephfs.filesystem:scrub status for tag:ff9fd113-a7fd-4faf-8816-2438b2f652df - {'path': '/', 'tag': 'ff9fd113-a7fd-4faf-8816-2438b2f652df', 'options': 'recursive,force'} 2024-03-24T09:30:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:22 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:30:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:22 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:22 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:22 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:22.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:22 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:30:22.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:22 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:22.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:22 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:22.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:22 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: pgmap v255: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:23 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: pgmap v255: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:23 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: pgmap v255: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:30:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:23.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:23 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:25 smithi117 ceph-mon[35788]: pgmap v256: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:30:25.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:25 smithi116 ceph-mon[32207]: pgmap v256: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:30:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:25 smithi143 ceph-mon[35965]: pgmap v256: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:30:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:27 smithi117 ceph-mon[35788]: pgmap v257: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2024-03-24T09:30:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:27 smithi116 ceph-mon[32207]: pgmap v257: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2024-03-24T09:30:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:27 smithi143 ceph-mon[35965]: pgmap v257: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 89 B/s wr, 8 op/s 2024-03-24T09:30:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:29 smithi117 ceph-mon[35788]: pgmap v258: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:30:29.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:29 smithi116 ceph-mon[32207]: pgmap v258: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:30:29.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:29 smithi143 ceph-mon[35965]: pgmap v258: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:30:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:31 smithi117 ceph-mon[35788]: pgmap v259: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:31 smithi116 ceph-mon[32207]: pgmap v259: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:31.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:31 smithi143 ceph-mon[35965]: pgmap v259: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:33 smithi117 ceph-mon[35788]: pgmap v260: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:33.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:33 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:33.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:33 smithi116 ceph-mon[32207]: pgmap v260: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:33.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:33 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:33 smithi143 ceph-mon[35965]: pgmap v260: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:33 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:34.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:34.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:34 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:30:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:34.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:34 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:30:34.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:34.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:34.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:34.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:34 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:30:35.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:35 smithi117 ceph-mon[35788]: pgmap v261: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:35.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:35 smithi116 ceph-mon[32207]: pgmap v261: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:35 smithi143 ceph-mon[35965]: pgmap v261: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:37 smithi117 ceph-mon[35788]: pgmap v262: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:37.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:37 smithi116 ceph-mon[32207]: pgmap v262: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:37 smithi143 ceph-mon[35965]: pgmap v262: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:30:39.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:39 smithi116 ceph-mon[32207]: pgmap v263: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:39 smithi143 ceph-mon[35965]: pgmap v263: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:39 smithi117 ceph-mon[35788]: pgmap v263: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:41.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:41 smithi143 ceph-mon[35965]: pgmap v264: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:41 smithi117 ceph-mon[35788]: pgmap v264: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:41 smithi116 ceph-mon[32207]: pgmap v264: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:43.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:43 smithi143 ceph-mon[35965]: pgmap v265: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:43.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:43 smithi117 ceph-mon[35788]: pgmap v265: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:43 smithi116 ceph-mon[32207]: pgmap v265: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:45.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:45 smithi116 ceph-mon[32207]: pgmap v266: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:45.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:45 smithi143 ceph-mon[35965]: pgmap v266: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:45.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:45 smithi117 ceph-mon[35788]: pgmap v266: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:47.327 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:47 smithi143 ceph-mon[35965]: pgmap v267: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:47.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:47 smithi117 ceph-mon[35788]: pgmap v267: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:47.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:47 smithi116 ceph-mon[32207]: pgmap v267: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:49.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:49 smithi143 ceph-mon[35965]: pgmap v268: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:49.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:49 smithi117 ceph-mon[35788]: pgmap v268: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:49.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:49 smithi116 ceph-mon[32207]: pgmap v268: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:51.368 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:51 smithi117 ceph-mon[35788]: pgmap v269: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:51 smithi143 ceph-mon[35965]: pgmap v269: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:51.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:51 smithi116 ceph-mon[32207]: pgmap v269: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:52.061 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:30:52.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:30:52.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 -- 172.21.15.116:0/3077622053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec41016a0 msgr2=0x7fcec4101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 --2- 172.21.15.116:0/3077622053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec41016a0 0x7fcec4101b20 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fceac009930 tx=0x7fceac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:52.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 -- 172.21.15.116:0/3077622053 shutdown_connections 2024-03-24T09:30:52.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 --2- 172.21.15.116:0/3077622053 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcec4102060 0x7fcec410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 --2- 172.21.15.116:0/3077622053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec41016a0 0x7fcec4101b20 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 --2- 172.21.15.116:0/3077622053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec41076b0 0x7fcec4107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 -- 172.21.15.116:0/3077622053 >> 172.21.15.116:0/3077622053 conn(0x7fcec40fd170 msgr2=0x7fcec40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 -- 172.21.15.116:0/3077622053 shutdown_connections 2024-03-24T09:30:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.179+0000 7fcec93c7640 1 -- 172.21.15.116:0/3077622053 wait complete. 2024-03-24T09:30:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 Processor -- start 2024-03-24T09:30:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 -- start start 2024-03-24T09:30:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcec41016a0 0x7fcec419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 0x7fcec419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec41076b0 0x7fcec4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4198100 con 0x7fcec4102060 2024-03-24T09:30:52.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4198270 con 0x7fcec41076b0 2024-03-24T09:30:52.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec93c7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4198430 con 0x7fcec41016a0 2024-03-24T09:30:52.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 0x7fcec419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 0x7fcec419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45120/0 (socket says 172.21.15.116:45120) 2024-03-24T09:30:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 -- 172.21.15.116:0/849440188 learned_addr learned my addr 172.21.15.116:0/849440188 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec37fe640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec41076b0 0x7fcec4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcec41016a0 0x7fcec419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 -- 172.21.15.116:0/849440188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcec41016a0 msgr2=0x7fcec419d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcec41016a0 0x7fcec419d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 -- 172.21.15.116:0/849440188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec41076b0 msgr2=0x7fcec4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec41076b0 0x7fcec4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 -- 172.21.15.116:0/849440188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb4009490 con 0x7fcec4102060 2024-03-24T09:30:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.180+0000 7fcebbfff640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 0x7fcec419de40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fceac009230 tx=0x7fceac005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.181+0000 7fcec0ff9640 1 -- 172.21.15.116:0/849440188 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac004350 con 0x7fcec4102060 2024-03-24T09:30:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.181+0000 7fcec0ff9640 1 -- 172.21.15.116:0/849440188 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fceac02de90 con 0x7fcec4102060 2024-03-24T09:30:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.181+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceac009560 con 0x7fcec4102060 2024-03-24T09:30:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.181+0000 7fcec0ff9640 1 -- 172.21.15.116:0/849440188 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceac005220 con 0x7fcec4102060 2024-03-24T09:30:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.181+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41a47c0 con 0x7fcec4102060 2024-03-24T09:30:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.183+0000 7fcec0ff9640 1 -- 172.21.15.116:0/849440188 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fceac003c30 con 0x7fcec4102060 2024-03-24T09:30:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.183+0000 7fcecbfa5740 1 Processor -- start 2024-03-24T09:30:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.183+0000 7fcecbfa5740 1 -- start start 2024-03-24T09:30:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec0ff9640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fceb00a2230 0x7fceb00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f7d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f86190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f88ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0ea1f30 con 0x55f0f0f866d0 2024-03-24T09:30:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0eeb810 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0ee8d40 con 0x55f0f0f7daf0 2024-03-24T09:30:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec37fe640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fceb00a2230 0x7fceb00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f7d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f7d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50370/0 (socket says 172.21.15.116:50370) 2024-03-24T09:30:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f88ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 -- 172.21.15.116:0/310890592 learned_addr learned my addr 172.21.15.116:0/310890592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcebbfff640 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f86190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec0ff9640 1 -- 172.21.15.116:0/849440188 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fceac0f02e0 con 0x7fcec4102060 2024-03-24T09:30:52.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 -- 172.21.15.116:0/310890592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7daf0 msgr2=0x55f0f0f86190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f86190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 -- 172.21.15.116:0/310890592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 msgr2=0x55f0f0f88ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f88ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 -- 172.21.15.116:0/310890592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f0ea5ea0 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec37fe640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fceb00a2230 0x7fceb00a46f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcebc005fd0 tx=0x7fcebc009f10 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.184+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f7d5b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009da0 tx=0x7fceb4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37bdfcc455445ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fceb97fa640 1 -- 172.21.15.116:0/310890592 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb40042f0 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fceb97fa640 1 -- 172.21.15.116:0/310890592 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fceb4004450 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fceb97fa640 1 -- 172.21.15.116:0/310890592 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4019710 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 -- 172.21.15.116:0/310890592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 msgr2=0x55f0f0f7d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f7d5b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009da0 tx=0x7fceb4011810 comp rx=0 tx=0).stop 2024-03-24T09:30:52.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 -- 172.21.15.116:0/310890592 shutdown_connections 2024-03-24T09:30:52.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f88ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f86190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/310890592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f7d5b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 -- 172.21.15.116:0/310890592 >> 172.21.15.116:0/310890592 conn(0x55f0f0f77900 msgr2=0x55f0f0f77ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 -- 172.21.15.116:0/310890592 shutdown_connections 2024-03-24T09:30:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.185+0000 7fcecbfa5740 1 -- 172.21.15.116:0/310890592 wait complete. 2024-03-24T09:30:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 -- start start 2024-03-24T09:30:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 Processor -- start 2024-03-24T09:30:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f836d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f84150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0ee4520 con 0x55f0f0f866d0 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0bb7430 con 0x55f0f0f7daf0 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0b73e00 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcecbfa5740 10 client.0 _ll_register_callbacks cb 0x55f0f0e82d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f84150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec2ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f836d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f84150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45138/0 (socket says 172.21.15.116:45138) 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50376/0 (socket says 172.21.15.116:50376) 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 -- 172.21.15.116:0/911293237 learned_addr learned my addr 172.21.15.116:0/911293237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 -- 172.21.15.116:0/911293237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 msgr2=0x55f0f0f836d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f836d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 -- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 msgr2=0x55f0f0f83c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.186+0000 7fcec37fe640 1 -- 172.21.15.116:0/911293237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb4009a00 con 0x55f0f0f866d0 2024-03-24T09:30:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcec37fe640 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f84150 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fcebc01e570 tx=0x7fcebc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b203586f2b3b20df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fceaa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fceaa7fc640 1 -- 172.21.15.116:0/911293237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc016040 con 0x55f0f0f866d0 2024-03-24T09:30:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fceaa7fc640 1 -- 172.21.15.116:0/911293237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fcebc003970 con 0x55f0f0f866d0 2024-03-24T09:30:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f0f0be37e0 con 0x55f0f0f866d0 2024-03-24T09:30:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fceaa7fc640 1 -- 172.21.15.116:0/911293237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcebc003b30 con 0x55f0f0f866d0 2024-03-24T09:30:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 10 client.18069 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f0f0ee2ba0 con 0x55f0f0f866d0 2024-03-24T09:30:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fceaa7fc640 1 -- 172.21.15.116:0/911293237 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f0f0be37e0 con 0x55f0f0f866d0 2024-03-24T09:30:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fceaa7fc640 1 -- 172.21.15.116:0/911293237 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcebc004020 con 0x55f0f0f866d0 2024-03-24T09:30:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 10 client.18069 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 10 client.18069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 20 client.18069 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0f96b80 0x55f0f0f98fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 4 client.18069 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:52.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.187+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0f0c20720 con 0x55f0f0f96b80 2024-03-24T09:30:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.188+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0f96b80 0x55f0f0f98fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.188+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0f96b80 0x55f0f0f98fa0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688334c2d9ee7b2b server_cookie=4755bc77fe3b3275 in_seq=0 out_seq=0 2024-03-24T09:30:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.188+0000 7fceaa7fc640 10 client.18069 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fceaa7fc640 1 -- 172.21.15.116:0/911293237 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f0f0c20720 con 0x55f0f0f96b80 2024-03-24T09:30:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fceaa7fc640 10 client.18069 handle_command_reply: tid=0 2024-03-24T09:30:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 client.18069 shutdown 2024-03-24T09:30:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 msgr2=0x55f0f0f84150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f84150 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fcebc01e570 tx=0x7fcebc01d000 comp rx=0 tx=0).stop 2024-03-24T09:30:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 shutdown_connections 2024-03-24T09:30:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f84150 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0f96b80 0x55f0f0f98fa0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/911293237 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f836d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fceaa7fc640 0 client.18069 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:52.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 >> 172.21.15.116:0/911293237 conn(0x55f0f0f77900 msgr2=0x55f0f0f76b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 shutdown_connections 2024-03-24T09:30:52.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 1 -- 172.21.15.116:0/911293237 wait complete. 2024-03-24T09:30:52.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.189+0000 7fcecbfa5740 20 client.18069 trim_cache size 0 max 16384 2024-03-24T09:30:52.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.206+0000 7fcecbfa5740 1 Processor -- start 2024-03-24T09:30:52.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.206+0000 7fcecbfa5740 1 -- start start 2024-03-24T09:30:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f91f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f9e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0e69430 con 0x55f0f0f866d0 2024-03-24T09:30:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0f95360 con 0x55f0f0f7daf0 2024-03-24T09:30:52.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0f73240 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f9e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f9e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45146/0 (socket says 172.21.15.116:45146) 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcebbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 -- 172.21.15.116:0/1338870041 learned_addr learned my addr 172.21.15.116:0/1338870041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f91f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 -- 172.21.15.116:0/1338870041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 msgr2=0x55f0f0f91f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f91f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 -- 172.21.15.116:0/1338870041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 msgr2=0x55f0f0f83f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 -- 172.21.15.116:0/1338870041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f0e7eb20 con 0x55f0f0f866d0 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcebbfff640 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fcec37fe640 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f9e8b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fcebc011fd0 tx=0x7fcebc01e480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99b4b979e956d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fceb8ff9640 1 -- 172.21.15.116:0/1338870041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc016040 con 0x55f0f0f866d0 2024-03-24T09:30:52.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fceb8ff9640 1 -- 172.21.15.116:0/1338870041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fcebc011950 con 0x55f0f0f866d0 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.207+0000 7fceb8ff9640 1 -- 172.21.15.116:0/1338870041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc0047d0 con 0x55f0f0f866d0 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 -- 172.21.15.116:0/1338870041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 msgr2=0x55f0f0f9e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f9e8b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fcebc011fd0 tx=0x7fcebc01e480 comp rx=0 tx=0).stop 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 -- 172.21.15.116:0/1338870041 shutdown_connections 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0f9e8b0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f83f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/1338870041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f91f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 -- 172.21.15.116:0/1338870041 >> 172.21.15.116:0/1338870041 conn(0x55f0f0f77900 msgr2=0x55f0f0f76b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 -- 172.21.15.116:0/1338870041 shutdown_connections 2024-03-24T09:30:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 -- 172.21.15.116:0/1338870041 wait complete. 2024-03-24T09:30:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.208+0000 7fcecbfa5740 1 -- start start 2024-03-24T09:30:52.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 Processor -- start 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f8a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f8ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0ff5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0ff6460 con 0x55f0f0f866d0 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0ff65d0 con 0x55f0f0f7daf0 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f0f0ff6790 con 0x55f0f0f7d1d0 2024-03-24T09:30:52.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 10 client.0 _ll_register_callbacks cb 0x7fcebc009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0ff5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0ff5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45148/0 (socket says 172.21.15.116:45148) 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 -- 172.21.15.116:0/849166159 learned_addr learned my addr 172.21.15.116:0/849166159 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f8a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcebbfff640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f8ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 -- 172.21.15.116:0/849166159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 msgr2=0x55f0f0f8a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f8a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 -- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 msgr2=0x55f0f0f8ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f8ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 -- 172.21.15.116:0/849166159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41a47c0 con 0x55f0f0f866d0 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcec37fe640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0ff5f20 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004a20 tx=0x7fcebc020110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e649f5e0ea36f465 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fceaa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fceaa7fc640 1 -- 172.21.15.116:0/849166159 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc016070 con 0x55f0f0f866d0 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fceaa7fc640 1 -- 172.21.15.116:0/849166159 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fcebc020210 con 0x55f0f0f866d0 2024-03-24T09:30:52.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.209+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fceb4004450 con 0x55f0f0f866d0 2024-03-24T09:30:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fceaa7fc640 1 -- 172.21.15.116:0/849166159 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fcebc004d30 con 0x55f0f0f866d0 2024-03-24T09:30:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 10 client.18075 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f0f0ee2ba0 con 0x55f0f0f866d0 2024-03-24T09:30:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fceaa7fc640 1 -- 172.21.15.116:0/849166159 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fceb4004450 con 0x55f0f0f866d0 2024-03-24T09:30:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fceaa7fc640 1 -- 172.21.15.116:0/849166159 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fcebc003a00 con 0x55f0f0f866d0 2024-03-24T09:30:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 10 client.18075 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 10 client.18075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 20 client.18075 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0ffdbc0 0x55f0f0ffffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 4 client.18075 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f0f0c6f890 con 0x55f0f0ffdbc0 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0ffdbc0 0x55f0f0ffffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fcec2ffd640 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0ffdbc0 0x55f0f0ffffe0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35e05be761e2c1fb server_cookie=36b65ad7213302a in_seq=0 out_seq=0 2024-03-24T09:30:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.210+0000 7fceaa7fc640 10 client.18075 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fceaa7fc640 1 -- 172.21.15.116:0/849166159 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f0f0c6f890 con 0x55f0f0ffdbc0 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fceaa7fc640 10 client.18075 handle_command_reply: tid=0 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 client.18075 shutdown 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 msgr2=0x55f0f0ff5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0ff5f20 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004a20 tx=0x7fcebc020110 comp rx=0 tx=0).stop 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 shutdown_connections 2024-03-24T09:30:52.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f0f0f866d0 0x55f0f0ff5f20 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f0f0ffdbc0 0x55f0f0ffffe0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f0f0f7daf0 0x55f0f0f8ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 --2- 172.21.15.116:0/849166159 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f0f0f7d1d0 0x55f0f0f8a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fceaa7fc640 0 client.18075 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 >> 172.21.15.116:0/849166159 conn(0x55f0f0f77900 msgr2=0x55f0f0f78cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 shutdown_connections 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 1 -- 172.21.15.116:0/849166159 wait complete. 2024-03-24T09:30:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcecbfa5740 20 client.18075 trim_cache size 0 max 16384 2024-03-24T09:30:52.230 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:30:52.230 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:30:52.230 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:30:52.230 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:30:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fceb00a2230 msgr2=0x7fceb00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fceb00a2230 0x7fceb00a46f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcebc005fd0 tx=0x7fcebc009f10 comp rx=0 tx=0).stop 2024-03-24T09:30:52.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 msgr2=0x7fcec419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 0x7fcec419de40 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fceac009230 tx=0x7fceac005dd0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 shutdown_connections 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcec41076b0 0x7fcec4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcec4102060 0x7fcec419de40 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fceb00a2230 0x7fceb00a46f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 --2- 172.21.15.116:0/849440188 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcec41016a0 0x7fcec419d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 >> 172.21.15.116:0/849440188 conn(0x7fcec40fd170 msgr2=0x7fcec40fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 shutdown_connections 2024-03-24T09:30:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.211+0000 7fcec93c7640 1 -- 172.21.15.116:0/849440188 wait complete. 2024-03-24T09:30:52.234 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:30:52.234 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:30:52.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2818799251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0100690 msgr2=0x7fb1e0100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2818799251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0100690 0x7fb1e0100b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009860 tx=0x7fb1d402bb70 comp rx=0 tx=0).stop 2024-03-24T09:30:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2818799251 shutdown_connections 2024-03-24T09:30:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2818799251 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1e0101050 0x7fb1e010d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2818799251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0100690 0x7fb1e0100b10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2818799251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0106650 0x7fb1e0106a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2818799251 >> 172.21.15.116:0/2818799251 conn(0x7fb1e0077770 msgr2=0x7fb1e0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2818799251 shutdown_connections 2024-03-24T09:30:52.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2818799251 wait complete. 2024-03-24T09:30:52.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.336+0000 7fb1e50eb640 1 Processor -- start 2024-03-24T09:30:52.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 -- start start 2024-03-24T09:30:52.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1e0100690 0x7fb1e019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 0x7fb1e019cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0106650 0x7fb1e0196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e0197060 con 0x7fb1e0101050 2024-03-24T09:30:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e01971d0 con 0x7fb1e0106650 2024-03-24T09:30:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1e50eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e0197390 con 0x7fb1e0100690 2024-03-24T09:30:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 0x7fb1e019cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 0x7fb1e019cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45174/0 (socket says 172.21.15.116:45174) 2024-03-24T09:30:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 -- 172.21.15.116:0/2550586673 learned_addr learned my addr 172.21.15.116:0/2550586673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1ded9d640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1e0100690 0x7fb1e019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1df59e640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0106650 0x7fb1e0196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 -- 172.21.15.116:0/2550586673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1e0100690 msgr2=0x7fb1e019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1e0100690 0x7fb1e019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 -- 172.21.15.116:0/2550586673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0106650 msgr2=0x7fb1e0196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0106650 0x7fb1e0196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 -- 172.21.15.116:0/2550586673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d0009560 con 0x7fb1e0101050 2024-03-24T09:30:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.337+0000 7fb1de59c640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 0x7fb1e019cd50 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4037040 tx=0x7fb1d4004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.338+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4034950 con 0x7fb1e0101050 2024-03-24T09:30:52.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.338+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb1d402ce90 con 0x7fb1e0101050 2024-03-24T09:30:52.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.338+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d4009490 con 0x7fb1e0101050 2024-03-24T09:30:52.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.338+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4003ea0 con 0x7fb1e0101050 2024-03-24T09:30:52.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.338+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e01a36d0 con 0x7fb1e0101050 2024-03-24T09:30:52.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.339+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e0068950 con 0x7fb1e0101050 2024-03-24T09:30:52.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.340+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fb1d403b040 con 0x7fb1e0101050 2024-03-24T09:30:52.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.340+0000 7fb1c7fff640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb1c80a2230 0x7fb1c80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.340+0000 7fb1ded9d640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb1c80a2230 0x7fb1c80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.340+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fb1d40f0090 con 0x7fb1e0101050 2024-03-24T09:30:52.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.341+0000 7fb1ded9d640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb1c80a2230 0x7fb1c80a46f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0104730 tx=0x7fb1d0009260 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.344+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb1d40b8ad0 con 0x7fb1e0101050 2024-03-24T09:30:52.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.554+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1e0108bd0 con 0x7fb1e0101050 2024-03-24T09:30:52.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.555+0000 7fb1c7fff640 1 -- 172.21.15.116:0/2550586673 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7fb1d4041090 con 0x7fb1e0101050 2024-03-24T09:30:52.556 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:30:52.556 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:30:52.556 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb1c80a2230 msgr2=0x7fb1c80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb1c80a2230 0x7fb1c80a46f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0104730 tx=0x7fb1d0009260 comp rx=0 tx=0).stop 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 msgr2=0x7fb1e019cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 0x7fb1e019cd50 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4037040 tx=0x7fb1d4004590 comp rx=0 tx=0).stop 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 shutdown_connections 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1e0106650 0x7fb1e0196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1e0101050 0x7fb1e019cd50 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb1c80a2230 0x7fb1c80a46f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 --2- 172.21.15.116:0/2550586673 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb1e0100690 0x7fb1e019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 >> 172.21.15.116:0/2550586673 conn(0x7fb1e0077770 msgr2=0x7fb1e0102ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 shutdown_connections 2024-03-24T09:30:52.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.557+0000 7fb1e50eb640 1 -- 172.21.15.116:0/2550586673 wait complete. 2024-03-24T09:30:52.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:30:52.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 -- 172.21.15.116:0/3043546393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 msgr2=0x7fe530100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 --2- 172.21.15.116:0/3043546393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530100a20 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fe524009900 tx=0x7fe52402bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:52.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 -- 172.21.15.116:0/3043546393 shutdown_connections 2024-03-24T09:30:52.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 --2- 172.21.15.116:0/3043546393 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530100f60 0x7fe53010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 --2- 172.21.15.116:0/3043546393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530100a20 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 --2- 172.21.15.116:0/3043546393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530106660 0x7fe530106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 -- 172.21.15.116:0/3043546393 >> 172.21.15.116:0/3043546393 conn(0x7fe530077710 msgr2=0x7fe530077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 -- 172.21.15.116:0/3043546393 shutdown_connections 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.671+0000 7fe5374da640 1 -- 172.21.15.116:0/3043546393 wait complete. 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 Processor -- start 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 -- start start 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530100f60 0x7fe530198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530106660 0x7fe5301a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5301997a0 con 0x7fe5301005a0 2024-03-24T09:30:52.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe530199910 con 0x7fe530100f60 2024-03-24T09:30:52.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe5374da640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe5301a4080 con 0x7fe530106660 2024-03-24T09:30:52.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe534a75640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530100f60 0x7fe530198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45200/0 (socket says 172.21.15.116:45200) 2024-03-24T09:30:52.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 -- 172.21.15.116:0/1436339739 learned_addr learned my addr 172.21.15.116:0/1436339739 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535a77640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530106660 0x7fe5301a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 -- 172.21.15.116:0/1436339739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530106660 msgr2=0x7fe5301a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530106660 0x7fe5301a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 -- 172.21.15.116:0/1436339739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530100f60 msgr2=0x7fe530198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530100f60 0x7fe530198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 -- 172.21.15.116:0/1436339739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe524009560 con 0x7fe5301005a0 2024-03-24T09:30:52.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535a77640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530106660 0x7fe5301a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe534a75640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530100f60 0x7fe530198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.672+0000 7fe535276640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530198a70 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ebc0 tx=0x7fe52800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.673+0000 7fe52e7fc640 1 -- 172.21.15.116:0/1436339739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5280099b0 con 0x7fe5301005a0 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.673+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5301a4360 con 0x7fe5301005a0 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.673+0000 7fe52e7fc640 1 -- 172.21.15.116:0/1436339739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe52800eec0 con 0x7fe5301005a0 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.673+0000 7fe52e7fc640 1 -- 172.21.15.116:0/1436339739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe528018a80 con 0x7fe5301005a0 2024-03-24T09:30:52.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.673+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5301a4840 con 0x7fe5301005a0 2024-03-24T09:30:52.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.675+0000 7fe52e7fc640 1 -- 172.21.15.116:0/1436339739 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe528018be0 con 0x7fe5301005a0 2024-03-24T09:30:52.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.675+0000 7fe53a18b740 1 Processor -- start 2024-03-24T09:30:52.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.675+0000 7fe53a18b740 1 -- start start 2024-03-24T09:30:52.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 0x55f944c747d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c74d10 0x55f944c7d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c7fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe53a18b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f944be2840 con 0x55f944c7d8f0 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe53a18b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f944bdfd70 con 0x55f944c743f0 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe53a18b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f944bdb550 con 0x55f944c74d10 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe52e7fc640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe5180a2230 0x7fe5180a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 0x55f944c747d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe534a75640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c7fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 0x55f944c747d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50442/0 (socket says 172.21.15.116:50442) 2024-03-24T09:30:52.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 -- 172.21.15.116:0/1077586162 learned_addr learned my addr 172.21.15.116:0/1077586162 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe5180a2230 0x7fe5180a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535276640 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c74d10 0x55f944c7d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe52e7fc640 1 -- 172.21.15.116:0/1436339739 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fe528014070 con 0x7fe5301005a0 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 -- 172.21.15.116:0/1077586162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c74d10 msgr2=0x55f944c7d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c74d10 0x55f944c7d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 -- 172.21.15.116:0/1077586162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 msgr2=0x55f944c7fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c7fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 -- 172.21.15.116:0/1077586162 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f944b70de0 con 0x55f944c743f0 2024-03-24T09:30:52.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe534a75640 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c7fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe5180a2230 0x7fe5180a46f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe52000fc90 tx=0x7fe52001e000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe535a77640 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 0x55f944c747d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe520011e90 tx=0x7fe52000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d68782c15e4fab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.676+0000 7fe5177fe640 1 -- 172.21.15.116:0/1077586162 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52001dbf0 con 0x55f944c743f0 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe5177fe640 1 -- 172.21.15.116:0/1077586162 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe52001dd50 con 0x55f944c743f0 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe5177fe640 1 -- 172.21.15.116:0/1077586162 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520026d50 con 0x55f944c743f0 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 -- 172.21.15.116:0/1077586162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 msgr2=0x55f944c747d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 0x55f944c747d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe520011e90 tx=0x7fe52000f790 comp rx=0 tx=0).stop 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 -- 172.21.15.116:0/1077586162 shutdown_connections 2024-03-24T09:30:52.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c7fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c74d10 0x55f944c7d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 --2- 172.21.15.116:0/1077586162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c743f0 0x55f944c747d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 -- 172.21.15.116:0/1077586162 >> 172.21.15.116:0/1077586162 conn(0x55f944c6eb20 msgr2=0x55f944c6ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 -- 172.21.15.116:0/1077586162 shutdown_connections 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.677+0000 7fe53a18b740 1 -- 172.21.15.116:0/1077586162 wait complete. 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 -- start start 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 Processor -- start 2024-03-24T09:30:52.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c7a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 0x55f944c7af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c7d8f0 0x55f944c7b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f9448ae430 con 0x55f944c74d10 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f94486ae00 con 0x55f944c7d8f0 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f944b60590 con 0x55f944c743f0 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe53a18b740 10 client.0 _ll_register_callbacks cb 0x55f944bcffc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 0x55f944c7af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 0x55f944c7af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45222/0 (socket says 172.21.15.116:45222) 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 -- 172.21.15.116:0/661532524 learned_addr learned my addr 172.21.15.116:0/661532524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe535276640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c7a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe535a77640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c7d8f0 0x55f944c7b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 -- 172.21.15.116:0/661532524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 msgr2=0x55f944c7a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c7a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 -- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c7d8f0 msgr2=0x55f944c7b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c7d8f0 0x55f944c7b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.678+0000 7fe534a75640 1 -- 172.21.15.116:0/661532524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe520019040 con 0x55f944c74d10 2024-03-24T09:30:52.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe534a75640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 0x55f944c7af00 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe524009900 tx=0x7fe524004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b81764592f41b3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe514ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe514ff9640 1 -- 172.21.15.116:0/661532524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe524004a70 con 0x55f944c74d10 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe514ff9640 1 -- 172.21.15.116:0/661532524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fe5240035f0 con 0x55f944c74d10 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f9448da7e0 con 0x55f944c74d10 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe514ff9640 1 -- 172.21.15.116:0/661532524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe5240037b0 con 0x55f944c74d10 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 10 client.18093 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f944bd9bd0 con 0x55f944c74d10 2024-03-24T09:30:52.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe514ff9640 1 -- 172.21.15.116:0/661532524 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f9448da7e0 con 0x55f944c74d10 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe514ff9640 1 -- 172.21.15.116:0/661532524 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe52403c040 con 0x55f944c74d10 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 10 client.18093 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 10 client.18093 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 20 client.18093 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944c8dad0 0x55f944c8fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 4 client.18093 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f944917720 con 0x55f944c8dad0 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.679+0000 7fe535276640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944c8dad0 0x55f944c8fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.680+0000 7fe514ff9640 10 client.18093 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:52.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.680+0000 7fe535276640 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944c8dad0 0x55f944c8fef0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a9ea40857cbdb292 server_cookie=cd4c8d8e849439c4 in_seq=0 out_seq=0 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe514ff9640 1 -- 172.21.15.116:0/661532524 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f944917720 con 0x55f944c8dad0 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe514ff9640 10 client.18093 handle_command_reply: tid=0 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 client.18093 shutdown 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 msgr2=0x55f944c7af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 0x55f944c7af00 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fe524009900 tx=0x7fe524004270 comp rx=0 tx=0).stop 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 shutdown_connections 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c7d8f0 0x55f944c7b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944c8dad0 0x55f944c8fef0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c74d10 0x55f944c7af00 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 --2- 172.21.15.116:0/661532524 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c7a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe514ff9640 0 client.18093 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 >> 172.21.15.116:0/661532524 conn(0x55f944c6eb20 msgr2=0x55f944c70870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 shutdown_connections 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 1 -- 172.21.15.116:0/661532524 wait complete. 2024-03-24T09:30:52.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.681+0000 7fe53a18b740 20 client.18093 trim_cache size 0 max 16384 2024-03-24T09:30:52.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.694+0000 7fe53a18b740 1 Processor -- start 2024-03-24T09:30:52.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.694+0000 7fe53a18b740 1 -- start start 2024-03-24T09:30:52.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c89120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944c7b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c957c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe53a18b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f944c6a210 con 0x55f944c7d8f0 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe53a18b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f944c74050 con 0x55f944c74d10 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe53a18b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f944c5fe20 con 0x55f944c743f0 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c957c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45236/0 (socket says 172.21.15.116:45236) 2024-03-24T09:30:52.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 -- 172.21.15.116:0/2970958445 learned_addr learned my addr 172.21.15.116:0/2970958445 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe534a75640 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944c7b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535276640 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c89120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 -- 172.21.15.116:0/2970958445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 msgr2=0x55f944c89120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c89120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 -- 172.21.15.116:0/2970958445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 msgr2=0x55f944c7b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944c7b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 -- 172.21.15.116:0/2970958445 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f944b75c80 con 0x55f944c7d8f0 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe535a77640 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c957c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe5200096f0 tx=0x7fe520000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb7c0b1e8bb37cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe516ffd640 1 -- 172.21.15.116:0/2970958445 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520011970 con 0x55f944c7d8f0 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe516ffd640 1 -- 172.21.15.116:0/2970958445 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe52002b040 con 0x55f944c7d8f0 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.695+0000 7fe516ffd640 1 -- 172.21.15.116:0/2970958445 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5200039f0 con 0x55f944c7d8f0 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 -- 172.21.15.116:0/2970958445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 msgr2=0x55f944c957c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c957c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fe5200096f0 tx=0x7fe520000bc0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 -- 172.21.15.116:0/2970958445 shutdown_connections 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c7d8f0 0x55f944c957c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944c7b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2970958445 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c743f0 0x55f944c89120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 -- 172.21.15.116:0/2970958445 >> 172.21.15.116:0/2970958445 conn(0x55f944c6eb20 msgr2=0x55f944c707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 -- 172.21.15.116:0/2970958445 shutdown_connections 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 -- 172.21.15.116:0/2970958445 wait complete. 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 -- start start 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.696+0000 7fe53a18b740 1 Processor -- start 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 0x55f944ced100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944ced640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c7d8f0 0x55f944c817f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f944c81e00 con 0x55f944c743f0 2024-03-24T09:30:52.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f944c81f70 con 0x55f944c74d10 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f944c82130 con 0x55f944c7d8f0 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 10 client.0 _ll_register_callbacks cb 0x7fe524009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 0x55f944ced100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe534a75640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944ced640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 0x55f944ced100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45252/0 (socket says 172.21.15.116:45252) 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe534a75640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944ced640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50476/0 (socket says 172.21.15.116:50476) 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 -- 172.21.15.116:0/2136780670 learned_addr learned my addr 172.21.15.116:0/2136780670 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535a77640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c7d8f0 0x55f944c817f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 -- 172.21.15.116:0/2136780670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c7d8f0 msgr2=0x55f944c817f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c7d8f0 0x55f944c817f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 -- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 msgr2=0x55f944ced640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944ced640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 -- 172.21.15.116:0/2136780670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f944b70de0 con 0x55f944c743f0 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe534a75640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944ced640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe535276640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 0x55f944ced100 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe528097a70 tx=0x7fe5280661a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e2a1275ffc3342 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe514ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe514ff9640 1 -- 172.21.15.116:0/2136780670 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52809aa00 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe514ff9640 1 -- 172.21.15.116:0/2136780670 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fe528010d90 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.697+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe52001dd50 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe514ff9640 1 -- 172.21.15.116:0/2136780670 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe52809b0a0 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 10 client.18099 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f944b9b920 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe514ff9640 1 -- 172.21.15.116:0/2136780670 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe5280b7080 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe514ff9640 1 -- 172.21.15.116:0/2136780670 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fe5280b3d20 con 0x55f944c743f0 2024-03-24T09:30:52.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 10 client.18099 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 10 client.18099 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 20 client.18099 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944cf4fd0 0x55f944cf73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 4 client.18099 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f944966890 con 0x55f944cf4fd0 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe534a75640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944cf4fd0 0x55f944cf73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe534a75640 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944cf4fd0 0x55f944cf73f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b8f7825c1f53f71 server_cookie=14939ee3fcfaf0aa in_seq=0 out_seq=0 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe514ff9640 10 client.18099 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:52.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe514ff9640 1 -- 172.21.15.116:0/2136780670 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f944966890 con 0x55f944cf4fd0 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.698+0000 7fe514ff9640 10 client.18099 handle_command_reply: tid=0 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 client.18099 shutdown 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 msgr2=0x55f944ced100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 0x55f944ced100 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe528097a70 tx=0x7fe5280661a0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 shutdown_connections 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f944c7d8f0 0x55f944c817f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f944c74d10 0x55f944ced640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f944cf4fd0 0x55f944cf73f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 --2- 172.21.15.116:0/2136780670 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f944c743f0 0x55f944ced100 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe514ff9640 0 client.18099 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 >> 172.21.15.116:0/2136780670 conn(0x55f944c6eb20 msgr2=0x55f944c6d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 shutdown_connections 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 1 -- 172.21.15.116:0/2136780670 wait complete. 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe53a18b740 20 client.18099 trim_cache size 0 max 16384 2024-03-24T09:30:52.711 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe5180a2230 msgr2=0x7fe5180a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe5180a2230 0x7fe5180a46f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe52000fc90 tx=0x7fe52001e000 comp rx=0 tx=0).stop 2024-03-24T09:30:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 msgr2=0x7fe530198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530198a70 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ebc0 tx=0x7fe52800c6a0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 shutdown_connections 2024-03-24T09:30:52.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe5180a2230 0x7fe5180a46f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe530106660 0x7fe5301a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe530100f60 0x7fe530198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 --2- 172.21.15.116:0/1436339739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5301005a0 0x7fe530198a70 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 >> 172.21.15.116:0/1436339739 conn(0x7fe530077710 msgr2=0x7fe5300ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.699+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 shutdown_connections 2024-03-24T09:30:52.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.700+0000 7fe5374da640 1 -- 172.21.15.116:0/1436339739 wait complete. 2024-03-24T09:30:52.714 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:30:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 -- 172.21.15.116:0/2558949160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404107910 msgr2=0x7fa404107cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2558949160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404107910 0x7fa404107cf0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009900 tx=0x7fa3f402bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 -- 172.21.15.116:0/2558949160 shutdown_connections 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2558949160 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa404100990 0x7fa40410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2558949160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4040fffd0 0x7fa404100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2558949160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404107910 0x7fa404107cf0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 -- 172.21.15.116:0/2558949160 >> 172.21.15.116:0/2558949160 conn(0x7fa404075950 msgr2=0x7fa404075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 -- 172.21.15.116:0/2558949160 shutdown_connections 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.817+0000 7fa40a69a640 1 -- 172.21.15.116:0/2558949160 wait complete. 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 Processor -- start 2024-03-24T09:30:52.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 -- start start 2024-03-24T09:30:52.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4040fffd0 0x7fa4041011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 0x7fa4041016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404107910 0x7fa404101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041a5fe0 con 0x7fa404100990 2024-03-24T09:30:52.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041a6150 con 0x7fa404107910 2024-03-24T09:30:52.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa40a69a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fa4041a6310 con 0x7fa4040fffd0 2024-03-24T09:30:52.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 0x7fa4041016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 0x7fa4041016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45270/0 (socket says 172.21.15.116:45270) 2024-03-24T09:30:52.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa4037fe640 1 -- 172.21.15.116:0/2123598695 learned_addr learned my addr 172.21.15.116:0/2123598695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa408c37640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404107910 0x7fa404101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.818+0000 7fa403fff640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4040fffd0 0x7fa4041011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4037fe640 1 -- 172.21.15.116:0/2123598695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4040fffd0 msgr2=0x7fa4041011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4037fe640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4040fffd0 0x7fa4041011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4037fe640 1 -- 172.21.15.116:0/2123598695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404107910 msgr2=0x7fa404101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4037fe640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404107910 0x7fa404101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4037fe640 1 -- 172.21.15.116:0/2123598695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f4009560 con 0x7fa404100990 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4037fe640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 0x7fa4041016f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800e9f0 tx=0x7fa3f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4017fa640 1 -- 172.21.15.116:0/2123598695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8009700 con 0x7fa404100990 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4017fa640 1 -- 172.21.15.116:0/2123598695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3f800cd80 con 0x7fa404100990 2024-03-24T09:30:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4041a65f0 con 0x7fa404100990 2024-03-24T09:30:52.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa4017fa640 1 -- 172.21.15.116:0/2123598695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8018840 con 0x7fa404100990 2024-03-24T09:30:52.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.819+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4041a6b00 con 0x7fa404100990 2024-03-24T09:30:52.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa4017fa640 1 -- 172.21.15.116:0/2123598695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fa3f80189a0 con 0x7fa404100990 2024-03-24T09:30:52.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 Processor -- start 2024-03-24T09:30:52.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 -- start start 2024-03-24T09:30:52.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 0x55e8956dc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dcd10 0x55e8956e53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e89564a840 con 0x55e8956dc3f0 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e895647d70 con 0x55e8956e58f0 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa40d277740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e895643550 con 0x55e8956dcd10 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 0x55e8956dc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 0x55e8956dc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45272/0 (socket says 172.21.15.116:45272) 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 -- 172.21.15.116:0/284264082 learned_addr learned my addr 172.21.15.116:0/284264082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa403fff640 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dcd10 0x55e8956e53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa4017fa640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3ec0a2260 0x7fa3ec0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 -- 172.21.15.116:0/284264082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dcd10 msgr2=0x55e8956e53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dcd10 0x55e8956e53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 -- 172.21.15.116:0/284264082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 msgr2=0x55e8956e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.822+0000 7fa408c37640 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa408c37640 1 -- 172.21.15.116:0/284264082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8955d8de0 con 0x55e8956dc3f0 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa4017fa640 1 -- 172.21.15.116:0/2123598695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fa3f8014070 con 0x7fa404100990 2024-03-24T09:30:52.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa408c37640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3ec0a2260 0x7fa3ec0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa408c37640 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 0x55e8956dc7d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc011590 tx=0x7fa3fc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab896ee1697a2ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa3ea7fc640 1 -- 172.21.15.116:0/284264082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc01bb90 con 0x55e8956dc3f0 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa3ea7fc640 1 -- 172.21.15.116:0/284264082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3fc01f070 con 0x55e8956dc3f0 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.823+0000 7fa3ea7fc640 1 -- 172.21.15.116:0/284264082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3fc025dc0 con 0x55e8956dc3f0 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- 172.21.15.116:0/284264082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 msgr2=0x55e8956dc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa408c37640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3ec0a2260 0x7fa3ec0a4720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc022000 tx=0x7fa3fc023000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 0x55e8956dc7d0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc011590 tx=0x7fa3fc011a50 comp rx=0 tx=0).stop 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- 172.21.15.116:0/284264082 shutdown_connections 2024-03-24T09:30:52.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dcd10 0x55e8956e53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- 172.21.15.116:0/284264082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dc3f0 0x55e8956dc7d0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- 172.21.15.116:0/284264082 >> 172.21.15.116:0/284264082 conn(0x55e8956d6b20 msgr2=0x55e8956d6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- 172.21.15.116:0/284264082 shutdown_connections 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- 172.21.15.116:0/284264082 wait complete. 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- start start 2024-03-24T09:30:52.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 Processor -- start 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956e2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e895316430 con 0x55e8956dcd10 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e8952d2e00 con 0x55e8956e58f0 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8955c8590 con 0x55e8956dc3f0 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.824+0000 7fa40d277740 10 client.0 _ll_register_callbacks cb 0x55e895637fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa403fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956e2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956e2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45278/0 (socket says 172.21.15.116:45278) 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 -- 172.21.15.116:0/1722725832 learned_addr learned my addr 172.21.15.116:0/1722725832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa408c37640 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 -- 172.21.15.116:0/1722725832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 msgr2=0x55e8956e2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956e2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 -- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 msgr2=0x55e8956e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 -- 172.21.15.116:0/1722725832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc009c50 con 0x55e8956dcd10 2024-03-24T09:30:52.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa4037fe640 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956e2eb0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80cbb20 tx=0x7fa3f80d1cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990992c922e3f345 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa3e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa3e37fe640 1 -- 172.21.15.116:0/1722725832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f809e020 con 0x55e8956dcd10 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa3e37fe640 1 -- 172.21.15.116:0/1722725832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fa3f809ae10 con 0x55e8956dcd10 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e8953427e0 con 0x55e8956dcd10 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa3e37fe640 1 -- 172.21.15.116:0/1722725832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa3f809a770 con 0x55e8956dcd10 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa40d277740 10 client.18111 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e895641bd0 con 0x55e8956dcd10 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.825+0000 7fa3e37fe640 1 -- 172.21.15.116:0/1722725832 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3f80b7020 con 0x55e8956dcd10 2024-03-24T09:30:52.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa3e37fe640 1 -- 172.21.15.116:0/1722725832 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa3f8096b60 con 0x55e8956dcd10 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa40d277740 10 client.18111 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa40d277740 10 client.18111 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa40d277740 20 client.18111 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa40d277740 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e8956f5b00 0x55e8956f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa40d277740 4 client.18111 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e89537f720 con 0x55e8956f5b00 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa403fff640 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e8956f5b00 0x55e8956f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa3e37fe640 10 client.18111 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:52.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.826+0000 7fa403fff640 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e8956f5b00 0x55e8956f7f20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=305f71b6d2c838d1 server_cookie=73c6e23bbd43d859 in_seq=0 out_seq=0 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa3e37fe640 1 -- 172.21.15.116:0/1722725832 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e89537f720 con 0x55e8956f5b00 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa3e37fe640 10 client.18111 handle_command_reply: tid=0 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 client.18111 shutdown 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 msgr2=0x55e8956e2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956e2eb0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80cbb20 tx=0x7fa3f80d1cc0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 shutdown_connections 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e8956f5b00 0x55e8956f7f20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956e2eb0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 --2- 172.21.15.116:0/1722725832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956e2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa3e37fe640 0 client.18111 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 >> 172.21.15.116:0/1722725832 conn(0x55e8956d6b20 msgr2=0x55e8956d8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 shutdown_connections 2024-03-24T09:30:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 1 -- 172.21.15.116:0/1722725832 wait complete. 2024-03-24T09:30:52.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.827+0000 7fa40d277740 20 client.18111 trim_cache size 0 max 16384 2024-03-24T09:30:52.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 Processor -- start 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 -- start start 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956dc3f0 0x55e8956e5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956d5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956e58f0 0x55e8956e3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8956db770 con 0x55e8956dcd10 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e8956e0ec0 con 0x55e8956dc3f0 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa40d277740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e8956e18b0 con 0x55e8956e58f0 2024-03-24T09:30:52.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956d5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956d5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45292/0 (socket says 172.21.15.116:45292) 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa4037fe640 1 -- 172.21.15.116:0/1578226624 learned_addr learned my addr 172.21.15.116:0/1578226624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa403fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956dc3f0 0x55e8956e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.840+0000 7fa408c37640 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956e58f0 0x55e8956e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa4037fe640 1 -- 172.21.15.116:0/1578226624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956e58f0 msgr2=0x55e8956e3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa4037fe640 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956e58f0 0x55e8956e3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa4037fe640 1 -- 172.21.15.116:0/1578226624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956dc3f0 msgr2=0x55e8956e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa4037fe640 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956dc3f0 0x55e8956e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa4037fe640 1 -- 172.21.15.116:0/1578226624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8956d63c0 con 0x55e8956dcd10 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa408c37640 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956e58f0 0x55e8956e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa4037fe640 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956d5cc0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c6a0 tx=0x7fa3f800c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cd8c43f23524e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa3e9ffb640 1 -- 172.21.15.116:0/1578226624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f809e070 con 0x55e8956dcd10 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa3e9ffb640 1 -- 172.21.15.116:0/1578226624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3f80b7030 con 0x55e8956dcd10 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa3e9ffb640 1 -- 172.21.15.116:0/1578226624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80a3500 con 0x55e8956dcd10 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 -- 172.21.15.116:0/1578226624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 msgr2=0x55e8956d5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956d5cc0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800c6a0 tx=0x7fa3f800c7d0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 -- 172.21.15.116:0/1578226624 shutdown_connections 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956e58f0 0x55e8956e3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956d5cc0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 --2- 172.21.15.116:0/1578226624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956dc3f0 0x55e8956e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.841+0000 7fa40d277740 1 -- 172.21.15.116:0/1578226624 >> 172.21.15.116:0/1578226624 conn(0x55e8956d6b20 msgr2=0x55e8956d7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 -- 172.21.15.116:0/1578226624 shutdown_connections 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 -- 172.21.15.116:0/1578226624 wait complete. 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 -- start start 2024-03-24T09:30:52.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 Processor -- start 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956ea780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956eb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e8956edfd0 con 0x55e8956dcd10 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e89575be50 con 0x55e8956e58f0 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e89575bfc0 con 0x55e8956dc3f0 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa40d277740 10 client.0 _ll_register_callbacks cb 0x7fa3f80d1dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa403fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956ea780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa408c37640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956eb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa4037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45302/0 (socket says 172.21.15.116:45302) 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.842+0000 7fa4037fe640 1 -- 172.21.15.116:0/2505983440 learned_addr learned my addr 172.21.15.116:0/2505983440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa4037fe640 1 -- 172.21.15.116:0/2505983440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 msgr2=0x55e8956ea780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa4037fe640 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956ea780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa4037fe640 1 -- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 msgr2=0x55e8956eb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa4037fe640 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956eb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa4037fe640 1 -- 172.21.15.116:0/2505983440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e895641bd0 con 0x55e8956dcd10 2024-03-24T09:30:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa403fff640 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956ea780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa4037fe640 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956eacc0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80cc0c0 tx=0x7fa3f809a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935f4326d8e8c0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa3e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa3e37fe640 1 -- 172.21.15.116:0/2505983440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f809e070 con 0x55e8956dcd10 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa3e37fe640 1 -- 172.21.15.116:0/2505983440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fa3f80b8080 con 0x55e8956dcd10 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3fc01f070 con 0x55e8956dcd10 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa3e37fe640 1 -- 172.21.15.116:0/2505983440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fa3f809eeb0 con 0x55e8956dcd10 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 10 client.18117 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e8955ddc80 con 0x55e8956dcd10 2024-03-24T09:30:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa3e37fe640 1 -- 172.21.15.116:0/2505983440 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fa3fc01f070 con 0x55e8956dcd10 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa3e37fe640 1 -- 172.21.15.116:0/2505983440 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fa3f80b3a90 con 0x55e8956dcd10 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 10 client.18117 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 10 client.18117 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 20 client.18117 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e89575d270 0x55e89575f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 4 client.18117 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.843+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e8953ce890 con 0x55e89575d270 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa403fff640 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e89575d270 0x55e89575f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa403fff640 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e89575d270 0x55e89575f660 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=748dd8d746f8b498 server_cookie=85c09154b40a94aa in_seq=0 out_seq=0 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa3e37fe640 10 client.18117 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa3e37fe640 1 -- 172.21.15.116:0/2505983440 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e8953ce890 con 0x55e89575d270 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa3e37fe640 10 client.18117 handle_command_reply: tid=0 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 client.18117 shutdown 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 msgr2=0x55e8956eacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956eacc0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80cc0c0 tx=0x7fa3f809a860 comp rx=0 tx=0).stop 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 shutdown_connections 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55e89575d270 0x55e89575f660 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e8956e58f0 0x55e8956eb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e8956dcd10 0x55e8956eacc0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 --2- 172.21.15.116:0/2505983440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e8956dc3f0 0x55e8956ea780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa3e37fe640 0 client.18117 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 >> 172.21.15.116:0/2505983440 conn(0x55e8956d6b20 msgr2=0x55e8956d5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 shutdown_connections 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 1 -- 172.21.15.116:0/2505983440 wait complete. 2024-03-24T09:30:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.844+0000 7fa40d277740 20 client.18117 trim_cache size 0 max 16384 2024-03-24T09:30:52.857 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3ec0a2260 msgr2=0x7fa3ec0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3ec0a2260 0x7fa3ec0a4720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc022000 tx=0x7fa3fc023000 comp rx=0 tx=0).stop 2024-03-24T09:30:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 msgr2=0x7fa4041016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 0x7fa4041016f0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fa3f800e9f0 tx=0x7fa3f800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 shutdown_connections 2024-03-24T09:30:52.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa404107910 0x7fa404101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa404100990 0x7fa4041016f0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fa3ec0a2260 0x7fa3ec0a4720 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 --2- 172.21.15.116:0/2123598695 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa4040fffd0 0x7fa4041011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 >> 172.21.15.116:0/2123598695 conn(0x7fa404075950 msgr2=0x7fa4040fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 shutdown_connections 2024-03-24T09:30:52.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.845+0000 7fa40a69a640 1 -- 172.21.15.116:0/2123598695 wait complete. 2024-03-24T09:30:52.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:30:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 -- 172.21.15.116:0/3841207617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 msgr2=0x7f9b3c107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3841207617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c107ab0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009890 tx=0x7f9b3802bb70 comp rx=0 tx=0).stop 2024-03-24T09:30:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 -- 172.21.15.116:0/3841207617 shutdown_connections 2024-03-24T09:30:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3841207617 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c1005b0 0x7f9b3c10e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3841207617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3841207617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c107ab0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 -- 172.21.15.116:0/3841207617 >> 172.21.15.116:0/3841207617 conn(0x7f9b3c076bd0 msgr2=0x7f9b3c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 -- 172.21.15.116:0/3841207617 shutdown_connections 2024-03-24T09:30:52.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.963+0000 7f9b436fa640 1 -- 172.21.15.116:0/3841207617 wait complete. 2024-03-24T09:30:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 Processor -- start 2024-03-24T09:30:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 -- start start 2024-03-24T09:30:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b3c1005b0 0x7f9b3c19de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c198110 con 0x7f9b3c1076d0 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c198280 con 0x7f9b3c0ffc10 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b436fa640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c198440 con 0x7f9b3c1005b0 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41496640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45310/0 (socket says 172.21.15.116:45310) 2024-03-24T09:30:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41496640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c19d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50550/0 (socket says 172.21.15.116:50550) 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 -- 172.21.15.116:0/3945722174 learned_addr learned my addr 172.21.15.116:0/3945722174 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b40c95640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b3c1005b0 0x7f9b3c19de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 -- 172.21.15.116:0/3945722174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b3c1005b0 msgr2=0x7f9b3c19de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b3c1005b0 0x7f9b3c19de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 -- 172.21.15.116:0/3945722174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 msgr2=0x7f9b3c19d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c19d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41c97640 1 -- 172.21.15.116:0/3945722174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b34009900 con 0x7f9b3c1076d0 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.964+0000 7f9b41496640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.965+0000 7f9b41c97640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c197ac0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400efc0 tx=0x7f9b3400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.965+0000 7f9b327fc640 1 -- 172.21.15.116:0/3945722174 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34003970 con 0x7f9b3c1076d0 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.965+0000 7f9b327fc640 1 -- 172.21.15.116:0/3945722174 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b34004020 con 0x7f9b3c1076d0 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.965+0000 7f9b327fc640 1 -- 172.21.15.116:0/3945722174 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3400b830 con 0x7f9b3c1076d0 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.965+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b38009490 con 0x7f9b3c1076d0 2024-03-24T09:30:52.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.965+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c1a47d0 con 0x7f9b3c1076d0 2024-03-24T09:30:52.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.967+0000 7f9b327fc640 1 -- 172.21.15.116:0/3945722174 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9b3400a040 con 0x7f9b3c1076d0 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.967+0000 7f9b463b0740 1 Processor -- start 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.967+0000 7f9b463b0740 1 -- start start 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b327fc640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b280a2230 0x7f9b280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427a3f0 0x56285427a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x5628542833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854285ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b463b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5628541e8840 con 0x5628542838f0 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b463b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628541e5d70 con 0x56285427ad10 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b463b0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5628541e1550 con 0x56285427a3f0 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b40c95640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b280a2230 0x7f9b280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854285ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854285ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45318/0 (socket says 172.21.15.116:45318) 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427a3f0 0x56285427a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41c97640 1 -- 172.21.15.116:0/160529364 learned_addr learned my addr 172.21.15.116:0/160529364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b327fc640 1 -- 172.21.15.116:0/3945722174 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f9b340ce5f0 con 0x7f9b3c1076d0 2024-03-24T09:30:52.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b40c95640 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x5628542833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 -- 172.21.15.116:0/160529364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 msgr2=0x5628542833b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x5628542833b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 -- 172.21.15.116:0/160529364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 msgr2=0x562854285ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854285ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 -- 172.21.15.116:0/160529364 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562854176de0 con 0x56285427a3f0 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41c97640 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854285ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b40c95640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b280a2230 0x7f9b280a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009710 tx=0x7f9b2c0098b0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.968+0000 7f9b41496640 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427a3f0 0x56285427a7d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c198c00 tx=0x7f9b38005890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f56a3e114e7cb36e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b277fe640 1 -- 172.21.15.116:0/160529364 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38034440 con 0x56285427a3f0 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b277fe640 1 -- 172.21.15.116:0/160529364 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b3802ce90 con 0x56285427a3f0 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b277fe640 1 -- 172.21.15.116:0/160529364 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3802c7c0 con 0x56285427a3f0 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 -- 172.21.15.116:0/160529364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427a3f0 msgr2=0x56285427a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427a3f0 0x56285427a7d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c198c00 tx=0x7f9b38005890 comp rx=0 tx=0).stop 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 -- 172.21.15.116:0/160529364 shutdown_connections 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854285ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x5628542833b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 --2- 172.21.15.116:0/160529364 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427a3f0 0x56285427a7d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 -- 172.21.15.116:0/160529364 >> 172.21.15.116:0/160529364 conn(0x562854274b20 msgr2=0x562854274f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 -- 172.21.15.116:0/160529364 shutdown_connections 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.969+0000 7f9b463b0740 1 -- 172.21.15.116:0/160529364 wait complete. 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 -- start start 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 Processor -- start 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 0x562854280930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x562854280e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628542838f0 0x5628542813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562853eb4430 con 0x56285427a3f0 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562853e70e00 con 0x56285427ad10 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562854166590 con 0x5628542838f0 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b463b0740 10 client.0 _ll_register_callbacks cb 0x5628541d5fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 0x562854280930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 0x562854280930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45334/0 (socket says 172.21.15.116:45334) 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 -- 172.21.15.116:0/110334374 learned_addr learned my addr 172.21.15.116:0/110334374 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b40c95640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x562854280e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41c97640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628542838f0 0x5628542813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 -- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628542838f0 msgr2=0x5628542813b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628542838f0 0x5628542813b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 -- 172.21.15.116:0/110334374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 msgr2=0x562854280e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.970+0000 7f9b41496640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x562854280e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b41496640 1 -- 172.21.15.116:0/110334374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c10ee90 con 0x56285427a3f0 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b40c95640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x562854280e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b41496640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 0x562854280930 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f9b38035fd0 tx=0x7f9b380348b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622679fd8f8da8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b24ff9640 1 -- 172.21.15.116:0/110334374 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b380044c0 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b24ff9640 1 -- 172.21.15.116:0/110334374 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f9b3802c580 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562853ee07e0 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b24ff9640 1 -- 172.21.15.116:0/110334374 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9b3802c740 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 10 client.18129 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628541dfbd0 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b24ff9640 1 -- 172.21.15.116:0/110334374 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562853ee07e0 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b24ff9640 1 -- 172.21.15.116:0/110334374 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9b38032600 con 0x56285427a3f0 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 10 client.18129 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 10 client.18129 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 20 client.18129 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x562854293610 0x562854295a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 4 client.18129 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.971+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562853f1d720 con 0x562854293610 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.972+0000 7f9b40c95640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x562854293610 0x562854295a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.972+0000 7f9b24ff9640 10 client.18129 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.972+0000 7f9b40c95640 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x562854293610 0x562854295a50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a8cd7dc5f2df0c7 server_cookie=b6a3f53031df7a0e in_seq=0 out_seq=0 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b24ff9640 1 -- 172.21.15.116:0/110334374 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562853f1d720 con 0x562854293610 2024-03-24T09:30:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b24ff9640 10 client.18129 handle_command_reply: tid=0 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 client.18129 shutdown 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 msgr2=0x562854280930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 0x562854280930 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f9b38035fd0 tx=0x7f9b380348b0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 shutdown_connections 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5628542838f0 0x5628542813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x562854293610 0x562854295a50 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427ad10 0x562854280e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 --2- 172.21.15.116:0/110334374 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56285427a3f0 0x562854280930 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b24ff9640 0 client.18129 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 >> 172.21.15.116:0/110334374 conn(0x562854274b20 msgr2=0x562854276870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 shutdown_connections 2024-03-24T09:30:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 1 -- 172.21.15.116:0/110334374 wait complete. 2024-03-24T09:30:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.973+0000 7f9b463b0740 20 client.18129 trim_cache size 0 max 16384 2024-03-24T09:30:52.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.986+0000 7f9b463b0740 1 Processor -- start 2024-03-24T09:30:52.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.986+0000 7f9b463b0740 1 -- start start 2024-03-24T09:30:52.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x56285428f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x562854273b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x56285429b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b463b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562853ee0810 con 0x5628542838f0 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b463b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562854281930 con 0x56285427a3f0 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b463b0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562854280720 con 0x56285427ad10 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x56285429b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x56285429b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45336/0 (socket says 172.21.15.116:45336) 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 -- 172.21.15.116:0/3659841833 learned_addr learned my addr 172.21.15.116:0/3659841833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41496640 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x56285428f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b40c95640 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x562854273b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 -- 172.21.15.116:0/3659841833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 msgr2=0x562854273b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x562854273b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 -- 172.21.15.116:0/3659841833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 msgr2=0x56285428f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x56285428f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 -- 172.21.15.116:0/3659841833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56285417bc80 con 0x5628542838f0 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b41c97640 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x56285429b360 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409a8d0 tx=0x7f9b34012860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be299fa644bd29e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b26ffd640 1 -- 172.21.15.116:0/3659841833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34060650 con 0x5628542838f0 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b26ffd640 1 -- 172.21.15.116:0/3659841833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b34004190 con 0x5628542838f0 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.987+0000 7f9b26ffd640 1 -- 172.21.15.116:0/3659841833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3409ed20 con 0x5628542838f0 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 -- 172.21.15.116:0/3659841833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 msgr2=0x56285429b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x56285429b360 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409a8d0 tx=0x7f9b34012860 comp rx=0 tx=0).stop 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 -- 172.21.15.116:0/3659841833 shutdown_connections 2024-03-24T09:30:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x56285429b360 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x562854273b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- 172.21.15.116:0/3659841833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x56285428f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 -- 172.21.15.116:0/3659841833 >> 172.21.15.116:0/3659841833 conn(0x562854274b20 msgr2=0x562854273430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 -- 172.21.15.116:0/3659841833 shutdown_connections 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 -- 172.21.15.116:0/3659841833 wait complete. 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 -- start start 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 Processor -- start 2024-03-24T09:30:52.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x5628542f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x5628542f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.988+0000 7f9b463b0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854287940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562854287f00 con 0x5628542838f0 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562854288070 con 0x56285427a3f0 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562854288230 con 0x56285427ad10 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 10 client.0 _ll_register_callbacks cb 0x7f9b2c013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:52.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854287940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854287940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45340/0 (socket says 172.21.15.116:45340) 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 -- 172.21.15.116:0/681054761 learned_addr learned my addr 172.21.15.116:0/681054761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41496640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x5628542f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b40c95640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x5628542f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 -- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 msgr2=0x5628542f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x5628542f21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 -- 172.21.15.116:0/681054761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 msgr2=0x5628542f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:52.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x5628542f1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 -- 172.21.15.116:0/681054761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b38009490 con 0x5628542838f0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b41c97640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854287940 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409a850 tx=0x7f9b340990e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501e381a9f731ff0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b24ff9640 1 -- 172.21.15.116:0/681054761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b34064070 con 0x5628542838f0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b24ff9640 1 -- 172.21.15.116:0/681054761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f9b34065620 con 0x5628542838f0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b3802ce90 con 0x5628542838f0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b24ff9640 1 -- 172.21.15.116:0/681054761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f9b3409b2d0 con 0x5628542838f0 2024-03-24T09:30:53.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 10 client.18135 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5628541a1920 con 0x5628542838f0 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.989+0000 7f9b24ff9640 1 -- 172.21.15.116:0/681054761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b3407d080 con 0x5628542838f0 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b24ff9640 1 -- 172.21.15.116:0/681054761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f9b34092590 con 0x5628542838f0 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 10 client.18135 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 10 client.18135 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 20 client.18135 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5628542fa7b0 0x5628542fcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 4 client.18135 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562853f6c890 con 0x5628542fa7b0 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b41496640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5628542fa7b0 0x5628542fcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b24ff9640 10 client.18135 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b41496640 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5628542fa7b0 0x5628542fcba0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9124413d01df730e server_cookie=fc42ac6c0984f01c in_seq=0 out_seq=0 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b24ff9640 1 -- 172.21.15.116:0/681054761 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562853f6c890 con 0x5628542fa7b0 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b24ff9640 10 client.18135 handle_command_reply: tid=0 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 client.18135 shutdown 2024-03-24T09:30:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 msgr2=0x562854287940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854287940 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f9b3409a850 tx=0x7f9b340990e0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 shutdown_connections 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5628542838f0 0x562854287940 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56285427ad10 0x5628542f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x5628542fa7b0 0x5628542fcba0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 --2- 172.21.15.116:0/681054761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56285427a3f0 0x5628542f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b24ff9640 0 client.18135 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:30:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 >> 172.21.15.116:0/681054761 conn(0x562854274b20 msgr2=0x562854273cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.990+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 shutdown_connections 2024-03-24T09:30:53.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b463b0740 1 -- 172.21.15.116:0/681054761 wait complete. 2024-03-24T09:30:53.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b463b0740 20 client.18135 trim_cache size 0 max 16384 2024-03-24T09:30:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b280a2230 msgr2=0x7f9b280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b280a2230 0x7f9b280a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009710 tx=0x7f9b2c0098b0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 msgr2=0x7f9b3c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c197ac0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f9b3400efc0 tx=0x7f9b3400c800 comp rx=0 tx=0).stop 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 shutdown_connections 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b3c1076d0 0x7f9b3c197ac0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b280a2230 0x7f9b280a46f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b3c1005b0 0x7f9b3c19de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 --2- 172.21.15.116:0/3945722174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3c0ffc10 0x7f9b3c19d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 >> 172.21.15.116:0/3945722174 conn(0x7f9b3c076bd0 msgr2=0x7f9b3c0fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 shutdown_connections 2024-03-24T09:30:53.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:52.991+0000 7f9b436fa640 1 -- 172.21.15.116:0/3945722174 wait complete. 2024-03-24T09:30:53.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:30:53.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 -- 172.21.15.116:0/2940477391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 msgr2=0x7f260c06a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 --2- 172.21.15.116:0/2940477391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c06a1c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f25fc009930 tx=0x7f25fc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 -- 172.21.15.116:0/2940477391 shutdown_connections 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 --2- 172.21.15.116:0/2940477391 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c06a700 0x7f260c10e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 --2- 172.21.15.116:0/2940477391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c06a1c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 --2- 172.21.15.116:0/2940477391 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1076b0 0x7f260c107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 -- 172.21.15.116:0/2940477391 >> 172.21.15.116:0/2940477391 conn(0x7f260c0776e0 msgr2=0x7f260c077af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 -- 172.21.15.116:0/2940477391 shutdown_connections 2024-03-24T09:30:53.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 -- 172.21.15.116:0/2940477391 wait complete. 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 Processor -- start 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.111+0000 7f2613c81640 1 -- start start 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c06a700 0x7f260c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1076b0 0x7f260c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f260c1982a0 con 0x7f260c069d40 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f260c198410 con 0x7f260c06a700 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f260c1985d0 con 0x7f260c1076b0 2024-03-24T09:30:53.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c19da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45360/0 (socket says 172.21.15.116:45360) 2024-03-24T09:30:53.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 -- 172.21.15.116:0/3039920585 learned_addr learned my addr 172.21.15.116:0/3039920585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f261121c640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c06a700 0x7f260c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f261221e640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1076b0 0x7f260c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 -- 172.21.15.116:0/3039920585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1076b0 msgr2=0x7f260c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1076b0 0x7f260c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 -- 172.21.15.116:0/3039920585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c06a700 msgr2=0x7f260c19df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c06a700 0x7f260c19df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 -- 172.21.15.116:0/3039920585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc009560 con 0x7f260c069d40 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2611a1d640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c19da50 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f260400eb90 tx=0x7f260400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2602ffd640 1 -- 172.21.15.116:0/3039920585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26040099b0 con 0x7f260c069d40 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2602ffd640 1 -- 172.21.15.116:0/3039920585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f260400ee90 con 0x7f260c069d40 2024-03-24T09:30:53.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260c1988b0 con 0x7f260c069d40 2024-03-24T09:30:53.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2602ffd640 1 -- 172.21.15.116:0/3039920585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604010690 con 0x7f260c069d40 2024-03-24T09:30:53.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.112+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260c1a49d0 con 0x7f260c069d40 2024-03-24T09:30:53.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.115+0000 7f2602ffd640 1 -- 172.21.15.116:0/3039920585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2604005020 con 0x7f260c069d40 2024-03-24T09:30:53.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.115+0000 7f261685e740 1 Processor -- start 2024-03-24T09:30:53.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 -- start start 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626ac7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626acd10 0x55e7626b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 0x55e7626b7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e76261a840 con 0x55e7626ac3f0 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e762617d70 con 0x55e7626b58f0 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261685e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e762613550 con 0x55e7626acd10 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261221e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626ac7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 0x55e7626b7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f2611a1d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626acd10 0x55e7626b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 0x55e7626b7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50612/0 (socket says 172.21.15.116:50612) 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261221e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626ac7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45368/0 (socket says 172.21.15.116:45368) 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 -- 172.21.15.116:0/1427306379 learned_addr learned my addr 172.21.15.116:0/1427306379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f2602ffd640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f25f80a2230 0x7f25f80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 -- 172.21.15.116:0/1427306379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626acd10 msgr2=0x55e7626b53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626acd10 0x55e7626b53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 -- 172.21.15.116:0/1427306379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 msgr2=0x55e7626ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626ac7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261121c640 1 -- 172.21.15.116:0/1427306379 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7625a8de0 con 0x55e7626b58f0 2024-03-24T09:30:53.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261221e640 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f261221e640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f25f80a2230 0x7f25f80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.116+0000 7f2602ffd640 1 -- 172.21.15.116:0/3039920585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f2604014070 con 0x7f260c069d40 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261121c640 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 0x55e7626b7ce0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25fc036040 tx=0x7f25fc005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec72ff5331e8ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261221e640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f25f80a2230 0x7f25f80a46f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2608009f50 tx=0x7f26080046c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f25f7fff640 1 -- 172.21.15.116:0/1427306379 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc004800 con 0x55e7626b58f0 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f25f7fff640 1 -- 172.21.15.116:0/1427306379 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc02de90 con 0x55e7626b58f0 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f25f7fff640 1 -- 172.21.15.116:0/1427306379 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0035f0 con 0x55e7626b58f0 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 -- 172.21.15.116:0/1427306379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 msgr2=0x55e7626b7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 0x55e7626b7ce0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25fc036040 tx=0x7f25fc005de0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 -- 172.21.15.116:0/1427306379 shutdown_connections 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626b58f0 0x55e7626b7ce0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626acd10 0x55e7626b53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 --2- 172.21.15.116:0/1427306379 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626ac7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.117+0000 7f261685e740 1 -- 172.21.15.116:0/1427306379 >> 172.21.15.116:0/1427306379 conn(0x55e7626a6b20 msgr2=0x55e7626a6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 -- 172.21.15.116:0/1427306379 shutdown_connections 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 -- 172.21.15.116:0/1427306379 wait complete. 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 -- start start 2024-03-24T09:30:53.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 Processor -- start 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7622e6430 con 0x55e7626ac3f0 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7622a2e00 con 0x55e7626acd10 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e762598590 con 0x55e7626b58f0 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261685e740 10 client.0 _ll_register_callbacks cb 0x55e762607fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f2611a1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f2611a1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626b2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45376/0 (socket says 172.21.15.116:45376) 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f2611a1d640 1 -- 172.21.15.116:0/3479260612 learned_addr learned my addr 172.21.15.116:0/3479260612 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261221e640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.118+0000 7f261121c640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f2611a1d640 1 -- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 msgr2=0x55e7626b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f2611a1d640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f2611a1d640 1 -- 172.21.15.116:0/3479260612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 msgr2=0x55e7626b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f2611a1d640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f2611a1d640 1 -- 172.21.15.116:0/3479260612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260c10ee70 con 0x55e7626ac3f0 2024-03-24T09:30:53.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f2611a1d640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626b2900 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f260400cba0 tx=0x7f2604004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a3ade6c14e893c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f25f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f25f57fa640 1 -- 172.21.15.116:0/3479260612 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260409f120 con 0x55e7626ac3f0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f25f57fa640 1 -- 172.21.15.116:0/3479260612 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f260409b890 con 0x55e7626ac3f0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e7623127e0 con 0x55e7626ac3f0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f25f57fa640 1 -- 172.21.15.116:0/3479260612 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f260409aaf0 con 0x55e7626ac3f0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 10 client.18147 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e762611bd0 con 0x55e7626ac3f0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f25f57fa640 1 -- 172.21.15.116:0/3479260612 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55e7623127e0 con 0x55e7626ac3f0 2024-03-24T09:30:53.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f25f57fa640 1 -- 172.21.15.116:0/3479260612 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f260401f060 con 0x55e7626ac3f0 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 10 client.18147 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 10 client.18147 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 20 client.18147 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e7626c5930 0x55e7626c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 4 client.18147 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.119+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e76234f720 con 0x55e7626c5930 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.120+0000 7f261121c640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e7626c5930 0x55e7626c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.120+0000 7f25f57fa640 10 client.18147 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:53.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.120+0000 7f261121c640 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e7626c5930 0x55e7626c7d60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93f4fccca2301bec server_cookie=8b8a5d80a3da2352 in_seq=0 out_seq=0 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f25f57fa640 1 -- 172.21.15.116:0/3479260612 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e76234f720 con 0x55e7626c5930 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f25f57fa640 10 client.18147 handle_command_reply: tid=0 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 client.18147 shutdown 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 msgr2=0x55e7626b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626b2900 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f260400cba0 tx=0x7f2604004a40 comp rx=0 tx=0).stop 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 shutdown_connections 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e7626c5930 0x55e7626c7d60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 --2- 172.21.15.116:0/3479260612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626ac3f0 0x55e7626b2900 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f25f57fa640 0 client.18147 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 >> 172.21.15.116:0/3479260612 conn(0x55e7626a6b20 msgr2=0x55e7626a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 shutdown_connections 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 1 -- 172.21.15.116:0/3479260612 wait complete. 2024-03-24T09:30:53.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.121+0000 7f261685e740 20 client.18147 trim_cache size 0 max 16384 2024-03-24T09:30:53.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 Processor -- start 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 -- start start 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626ac3f0 0x55e7626b4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 0x55e7626b3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7626c23f0 con 0x55e7626acd10 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7626a7b20 con 0x55e7626ac3f0 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261685e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e7626b7ce0 con 0x55e7626b58f0 2024-03-24T09:30:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261121c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 0x55e7626b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261121c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 0x55e7626b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45382/0 (socket says 172.21.15.116:45382) 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261121c640 1 -- 172.21.15.116:0/851839468 learned_addr learned my addr 172.21.15.116:0/851839468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f2611a1d640 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626ac3f0 0x55e7626b4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.134+0000 7f261221e640 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f261121c640 1 -- 172.21.15.116:0/851839468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 msgr2=0x55e7626cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f261121c640 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f261121c640 1 -- 172.21.15.116:0/851839468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626ac3f0 msgr2=0x55e7626b4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f261121c640 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626ac3f0 0x55e7626b4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f261121c640 1 -- 172.21.15.116:0/851839468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7625adc80 con 0x55e7626acd10 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f261121c640 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 0x55e7626b3170 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f25fc036040 tx=0x7f25fc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5477801311161bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f25f77fe640 1 -- 172.21.15.116:0/851839468 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc004520 con 0x55e7626acd10 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f25f77fe640 1 -- 172.21.15.116:0/851839468 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc02c770 con 0x55e7626acd10 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.135+0000 7f25f77fe640 1 -- 172.21.15.116:0/851839468 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25fc052780 con 0x55e7626acd10 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- 172.21.15.116:0/851839468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 msgr2=0x55e7626b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 0x55e7626b3170 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f25fc036040 tx=0x7f25fc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- 172.21.15.116:0/851839468 shutdown_connections 2024-03-24T09:30:53.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626b58f0 0x55e7626cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626acd10 0x55e7626b3170 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- 172.21.15.116:0/851839468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626ac3f0 0x55e7626b4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- 172.21.15.116:0/851839468 >> 172.21.15.116:0/851839468 conn(0x55e7626a6b20 msgr2=0x55e7626a7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- 172.21.15.116:0/851839468 shutdown_connections 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- 172.21.15.116:0/851839468 wait complete. 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- start start 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 Processor -- start 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626ac3f0 0x55e7626bdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 0x55e7626b9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e7626b9b60 con 0x55e7626b58f0 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7626b9cd0 con 0x55e7626acd10 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55e76272ac60 con 0x55e7626ac3f0 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261685e740 10 client.0 _ll_register_callbacks cb 0x7f26040bc040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261221e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 0x55e7626b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261221e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 0x55e7626b9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45398/0 (socket says 172.21.15.116:45398) 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f261221e640 1 -- 172.21.15.116:0/2883009392 learned_addr learned my addr 172.21.15.116:0/2883009392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261121c640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.136+0000 7f2611a1d640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626ac3f0 0x55e7626bdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261221e640 1 -- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626ac3f0 msgr2=0x55e7626bdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261221e640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626ac3f0 0x55e7626bdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261221e640 1 -- 172.21.15.116:0/2883009392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 msgr2=0x55e7626b8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261221e640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261221e640 1 -- 172.21.15.116:0/2883009392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7625a8de0 con 0x55e7626b58f0 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261221e640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 0x55e7626b9460 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f260801fcb0 tx=0x7f2608004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe78f5c774d91ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f25f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:53.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f25f57fa640 1 -- 172.21.15.116:0/2883009392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2608024040 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f25f57fa640 1 -- 172.21.15.116:0/2883009392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f260801f2d0 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25fc02de90 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f25f57fa640 1 -- 172.21.15.116:0/2883009392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2608004030 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261685e740 10 client.18153 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e7625d3920 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f25f57fa640 1 -- 172.21.15.116:0/2883009392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f25fc02de90 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.137+0000 7f25f57fa640 1 -- 172.21.15.116:0/2883009392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2608032060 con 0x55e7626b58f0 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 10 client.18153 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:53.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 10 client.18153 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 20 client.18153 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e76272c5b0 0x55e76272e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 4 client.18153 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e76239e890 con 0x55e76272c5b0 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f2611a1d640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e76272c5b0 0x55e76272e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f25f57fa640 10 client.18153 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f2611a1d640 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e76272c5b0 0x55e76272e9d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efcdadb9115daa97 server_cookie=c5b88db597e03244 in_seq=0 out_seq=0 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f25f57fa640 1 -- 172.21.15.116:0/2883009392 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e76239e890 con 0x55e76272c5b0 2024-03-24T09:30:53.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f25f57fa640 10 client.18153 handle_command_reply: tid=0 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 client.18153 shutdown 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 msgr2=0x55e7626b9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 0x55e7626b9460 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f260801fcb0 tx=0x7f2608004730 comp rx=0 tx=0).stop 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 shutdown_connections 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55e76272c5b0 0x55e76272e9d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e7626b58f0 0x55e7626b9460 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7626acd10 0x55e7626b8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f25f57fa640 0 client.18153 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:30:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 --2- 172.21.15.116:0/2883009392 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55e7626ac3f0 0x55e7626bdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.138+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 >> 172.21.15.116:0/2883009392 conn(0x55e7626a6b20 msgr2=0x55e7626a8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 shutdown_connections 2024-03-24T09:30:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f261685e740 1 -- 172.21.15.116:0/2883009392 wait complete. 2024-03-24T09:30:53.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f261685e740 20 client.18153 trim_cache size 0 max 16384 2024-03-24T09:30:53.150 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:53.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f25f80a2230 msgr2=0x7f25f80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f25f80a2230 0x7f25f80a46f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2608009f50 tx=0x7f26080046c0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 msgr2=0x7f260c19da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c19da50 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f260400eb90 tx=0x7f260400c6a0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 shutdown_connections 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f25f80a2230 0x7f25f80a46f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f260c1076b0 0x7f260c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f260c06a700 0x7f260c19df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 --2- 172.21.15.116:0/3039920585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f260c069d40 0x7f260c19da50 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 >> 172.21.15.116:0/3039920585 conn(0x7f260c0776e0 msgr2=0x7f260c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 shutdown_connections 2024-03-24T09:30:53.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.139+0000 7f2613c81640 1 -- 172.21.15.116:0/3039920585 wait complete. 2024-03-24T09:30:53.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:53 smithi116 ceph-mon[32207]: pgmap v270: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:53.153 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:53 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2550586673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:30:53.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:30:53.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.257+0000 7f1b59e07640 1 -- 172.21.15.116:0/1609857014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b54069210 msgr2=0x7f1b5410db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.257+0000 7f1b59e07640 1 --2- 172.21.15.116:0/1609857014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b54069210 0x7f1b5410db70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009860 tx=0x7f1b3c02bb70 comp rx=0 tx=0).stop 2024-03-24T09:30:53.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.257+0000 7f1b59e07640 1 -- 172.21.15.116:0/1609857014 shutdown_connections 2024-03-24T09:30:53.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.257+0000 7f1b59e07640 1 --2- 172.21.15.116:0/1609857014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b541104a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 --2- 172.21.15.116:0/1609857014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b54069210 0x7f1b5410db70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 --2- 172.21.15.116:0/1609857014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b540688f0 0x7f1b54068cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- 172.21.15.116:0/1609857014 >> 172.21.15.116:0/1609857014 conn(0x7f1b54077710 msgr2=0x7f1b54077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- 172.21.15.116:0/1609857014 shutdown_connections 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- 172.21.15.116:0/1609857014 wait complete. 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 Processor -- start 2024-03-24T09:30:53.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- start start 2024-03-24T09:30:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 0x7f1b54199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b54069210 0x7f1b5419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b5419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5419ed00 con 0x7f1b5410e0b0 2024-03-24T09:30:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5419ee70 con 0x7f1b540688f0 2024-03-24T09:30:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.258+0000 7f1b59e07640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5419f030 con 0x7f1b54069210 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b5419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 0x7f1b54199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b5419e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45406/0 (socket says 172.21.15.116:45406) 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 0x7f1b54199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:50666/0 (socket says 172.21.15.116:50666) 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 -- 172.21.15.116:0/3043680734 learned_addr learned my addr 172.21.15.116:0/3043680734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b4affd640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b54069210 0x7f1b5419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 -- 172.21.15.116:0/3043680734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b54069210 msgr2=0x7f1b5419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b54069210 0x7f1b5419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 -- 172.21.15.116:0/3043680734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 msgr2=0x7f1b54199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 0x7f1b54199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 -- 172.21.15.116:0/3043680734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c009490 con 0x7f1b5410e0b0 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b4affd640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b54069210 0x7f1b5419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b537fe640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 0x7f1b54199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b53fff640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b5419e700 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009fa0 tx=0x7f1b4c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b517fa640 1 -- 172.21.15.116:0/3043680734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c009b50 con 0x7f1b5410e0b0 2024-03-24T09:30:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5419f310 con 0x7f1b5410e0b0 2024-03-24T09:30:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b517fa640 1 -- 172.21.15.116:0/3043680734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b4c003970 con 0x7f1b5410e0b0 2024-03-24T09:30:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b517fa640 1 -- 172.21.15.116:0/3043680734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4c0047a0 con 0x7f1b5410e0b0 2024-03-24T09:30:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.259+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b541a6b90 con 0x7f1b5410e0b0 2024-03-24T09:30:53.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b517fa640 1 -- 172.21.15.116:0/3043680734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f1b4c00a040 con 0x7f1b5410e0b0 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 Processor -- start 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 -- start start 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 0x55eed63dd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63ddd10 0x55eed63e63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eed634b840 con 0x55eed63dd3f0 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eed6348d70 con 0x55eed63e68f0 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eed6344550 con 0x55eed63ddd10 2024-03-24T09:30:53.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b517fa640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b400a2230 0x7f1b400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 0x55eed63dd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 0x55eed63dd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45414/0 (socket says 172.21.15.116:45414) 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b4affd640 1 -- 172.21.15.116:0/2860308172 learned_addr learned my addr 172.21.15.116:0/2860308172 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63ddd10 0x55eed63e63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b53fff640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b400a2230 0x7f1b400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b4affd640 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.262+0000 7f1b517fa640 1 -- 172.21.15.116:0/3043680734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f1b4c0ce380 con 0x7f1b5410e0b0 2024-03-24T09:30:53.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b537fe640 1 -- 172.21.15.116:0/2860308172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 msgr2=0x55eed63e8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b537fe640 1 -- 172.21.15.116:0/2860308172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 msgr2=0x55eed63dd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 0x55eed63dd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b537fe640 1 -- 172.21.15.116:0/2860308172 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eed62d9de0 con 0x55eed63ddd10 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b4affd640 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 0x55eed63dd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b53fff640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b400a2230 0x7f1b400a46f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c09a660 tx=0x7f1b4c06c920 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63ddd10 0x55eed63e63b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009fd0 tx=0x7f1b440118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce6216dcda111712 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b49ffb640 1 -- 172.21.15.116:0/2860308172 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44004030 con 0x55eed63ddd10 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b49ffb640 1 -- 172.21.15.116:0/2860308172 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b44005020 con 0x55eed63ddd10 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.263+0000 7f1b49ffb640 1 -- 172.21.15.116:0/2860308172 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44004ba0 con 0x55eed63ddd10 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2860308172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63ddd10 msgr2=0x55eed63e63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63ddd10 0x55eed63e63b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009fd0 tx=0x7f1b440118d0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2860308172 shutdown_connections 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63ddd10 0x55eed63e63b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2860308172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63dd3f0 0x55eed63dd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2860308172 >> 172.21.15.116:0/2860308172 conn(0x55eed63d7b20 msgr2=0x55eed63d7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2860308172 shutdown_connections 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2860308172 wait complete. 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 -- start start 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.264+0000 7f1b5c9e5740 1 Processor -- start 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63e3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63ddd10 0x55eed63e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63e68f0 0x55eed63e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eed6017430 con 0x55eed63e68f0 2024-03-24T09:30:53.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eed5fd3e00 con 0x55eed63ddd10 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eed62c9590 con 0x55eed63dd3f0 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 10 client.0 _ll_register_callbacks cb 0x55eed6338fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63e68f0 0x55eed63e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63e3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63ddd10 0x55eed63e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63e3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:44290/0 (socket says 172.21.15.116:44290) 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b4affd640 1 -- 172.21.15.116:0/4192710991 learned_addr learned my addr 172.21.15.116:0/4192710991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 -- 172.21.15.116:0/4192710991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 msgr2=0x55eed63e3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63e3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 -- 172.21.15.116:0/4192710991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63e68f0 msgr2=0x55eed63e43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63e68f0 0x55eed63e43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 -- 172.21.15.116:0/4192710991 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b44009c00 con 0x55eed63ddd10 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b4affd640 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b537fe640 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63ddd10 0x55eed63e3eb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009fa0 tx=0x7f1b44004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=329ad6eadcbd1823 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b32ffd640 1 -- 172.21.15.116:0/4192710991 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44004460 con 0x55eed63ddd10 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b32ffd640 1 -- 172.21.15.116:0/4192710991 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f1b440045c0 con 0x55eed63ddd10 2024-03-24T09:30:53.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eed60437e0 con 0x55eed63ddd10 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b32ffd640 1 -- 172.21.15.116:0/4192710991 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1b44004ab0 con 0x55eed63ddd10 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.265+0000 7f1b5c9e5740 10 client.27980 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55eed6342bd0 con 0x55eed63ddd10 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b32ffd640 1 -- 172.21.15.116:0/4192710991 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55eed60437e0 con 0x55eed63ddd10 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b32ffd640 1 -- 172.21.15.116:0/4192710991 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1b44021070 con 0x55eed63ddd10 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 10 client.27980 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 10 client.27980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 20 client.27980 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:53.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed63f6a90 0x55eed63f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 4 client.27980 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eed6080720 con 0x55eed63f6a90 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b4affd640 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed63f6a90 0x55eed63f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b4affd640 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed63f6a90 0x55eed63f8eb0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1963d846ee893538 server_cookie=8ff5b6785968c78b in_seq=0 out_seq=0 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.266+0000 7f1b32ffd640 10 client.27980 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.267+0000 7f1b32ffd640 1 -- 172.21.15.116:0/4192710991 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eed6080720 con 0x55eed63f6a90 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.267+0000 7f1b32ffd640 10 client.27980 handle_command_reply: tid=0 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.267+0000 7f1b5c9e5740 1 client.27980 shutdown 2024-03-24T09:30:53.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.267+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63ddd10 msgr2=0x55eed63e3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.267+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63ddd10 0x55eed63e3eb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009fa0 tx=0x7f1b44004270 comp rx=0 tx=0).stop 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 shutdown_connections 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63e68f0 0x55eed63e43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed63f6a90 0x55eed63f8eb0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63ddd10 0x55eed63e3eb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/4192710991 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63e3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b32ffd640 0 client.27980 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:53.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 >> 172.21.15.116:0/4192710991 conn(0x55eed63d7b20 msgr2=0x55eed63d9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 shutdown_connections 2024-03-24T09:30:53.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/4192710991 wait complete. 2024-03-24T09:30:53.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.268+0000 7f1b5c9e5740 20 client.27980 trim_cache size 0 max 16384 2024-03-24T09:30:53.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 Processor -- start 2024-03-24T09:30:53.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 -- start start 2024-03-24T09:30:53.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63f2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63fe780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eed63d8b20 con 0x55eed63ddd10 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eed63d7650 con 0x55eed63e68f0 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eed63d33e0 con 0x55eed63dd3f0 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63d6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45436/0 (socket says 172.21.15.116:45436) 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 -- 172.21.15.116:0/2486674807 learned_addr learned my addr 172.21.15.116:0/2486674807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b4affd640 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b4affd640 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63fe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 -- 172.21.15.116:0/2486674807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 msgr2=0x55eed63f2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63f2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 -- 172.21.15.116:0/2486674807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 msgr2=0x55eed63fe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63fe780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 -- 172.21.15.116:0/2486674807 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eed62dec80 con 0x55eed63ddd10 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b537fe640 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63d6b40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f1b44000c00 tx=0x7f1b44001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48997522034d92e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b497fa640 1 -- 172.21.15.116:0/2486674807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44005680 con 0x55eed63ddd10 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b497fa640 1 -- 172.21.15.116:0/2486674807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b44005d30 con 0x55eed63ddd10 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.281+0000 7f1b497fa640 1 -- 172.21.15.116:0/2486674807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b440042e0 con 0x55eed63ddd10 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2486674807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 msgr2=0x55eed63d6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63d6b40 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f1b44000c00 tx=0x7f1b44001e50 comp rx=0 tx=0).stop 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2486674807 shutdown_connections 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63fe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63d6b40 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/2486674807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2486674807 >> 172.21.15.116:0/2486674807 conn(0x55eed63d7b20 msgr2=0x55eed63d6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2486674807 shutdown_connections 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/2486674807 wait complete. 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.282+0000 7f1b5c9e5740 1 -- start start 2024-03-24T09:30:53.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 Processor -- start 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55eed63eef10 con 0x55eed63ddd10 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eed63e0880 con 0x55eed63e68f0 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55eed63e0a40 con 0x55eed63dd3f0 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b5c9e5740 10 client.0 _ll_register_callbacks cb 0x7f1b4c009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:53.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63dfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45450/0 (socket says 172.21.15.116:45450) 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 -- 172.21.15.116:0/1010801443 learned_addr learned my addr 172.21.15.116:0/1010801443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b4affd640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b4affd640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 -- 172.21.15.116:0/1010801443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 msgr2=0x55eed63ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63ee7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 -- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 msgr2=0x55eed63e0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 -- 172.21.15.116:0/1010801443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eed6342bd0 con 0x55eed63ddd10 2024-03-24T09:30:53.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b4affd640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b537fe640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63dfe00 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1b44013040 tx=0x7f1b4400f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fdd80a14eda97fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.283+0000 7f1b32ffd640 1 -- 172.21.15.116:0/1010801443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44005680 con 0x55eed63ddd10 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b32ffd640 1 -- 172.21.15.116:0/1010801443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7f1b4400fb40 con 0x55eed63ddd10 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b44005020 con 0x55eed63ddd10 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b32ffd640 1 -- 172.21.15.116:0/1010801443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1b4400fd00 con 0x55eed63ddd10 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 10 client.18171 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55eed6304920 con 0x55eed63ddd10 2024-03-24T09:30:53.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b32ffd640 1 -- 172.21.15.116:0/1010801443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1b44005020 con 0x55eed63ddd10 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b32ffd640 1 -- 172.21.15.116:0/1010801443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f1b44021040 con 0x55eed63ddd10 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 10 client.18171 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 10 client.18171 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 20 client.18171 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed645cd00 0x55eed645f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 4 client.18171 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eed60cf890 con 0x55eed645cd00 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b4affd640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed645cd00 0x55eed645f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:53.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b32ffd640 10 client.18171 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.284+0000 7f1b4affd640 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed645cd00 0x55eed645f0f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=335cd067d77a2e0c server_cookie=7d4ddba70fe9fdb2 in_seq=0 out_seq=0 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b32ffd640 1 -- 172.21.15.116:0/1010801443 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eed60cf890 con 0x55eed645cd00 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b32ffd640 10 client.18171 handle_command_reply: tid=0 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 client.18171 shutdown 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 msgr2=0x55eed63dfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63dfe00 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f1b44013040 tx=0x7f1b4400f9c0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 shutdown_connections 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55eed645cd00 0x55eed645f0f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eed63e68f0 0x55eed63e0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55eed63ddd10 0x55eed63dfe00 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b32ffd640 0 client.18171 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 --2- 172.21.15.116:0/1010801443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55eed63dd3f0 0x55eed63ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 >> 172.21.15.116:0/1010801443 conn(0x55eed63d7b20 msgr2=0x55eed63d8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 shutdown_connections 2024-03-24T09:30:53.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 1 -- 172.21.15.116:0/1010801443 wait complete. 2024-03-24T09:30:53.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b5c9e5740 20 client.18171 trim_cache size 0 max 16384 2024-03-24T09:30:53.298 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:30:53.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b400a2230 msgr2=0x7f1b400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b59e07640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b400a2230 0x7f1b400a46f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c09a660 tx=0x7f1b4c06c920 comp rx=0 tx=0).stop 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 msgr2=0x7f1b5419e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.285+0000 7f1b59e07640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b5419e700 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009fa0 tx=0x7f1b4c00c800 comp rx=0 tx=0).stop 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 shutdown_connections 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b5410e0b0 0x7f1b5419e700 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1b400a2230 0x7f1b400a46f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1b54069210 0x7f1b5419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 --2- 172.21.15.116:0/3043680734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b540688f0 0x7f1b54199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:53.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 >> 172.21.15.116:0/3043680734 conn(0x7f1b54077710 msgr2=0x7f1b5410f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:53.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 shutdown_connections 2024-03-24T09:30:53.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:53.286+0000 7f1b59e07640 1 -- 172.21.15.116:0/3043680734 wait complete. 2024-03-24T09:30:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:53 smithi143 ceph-mon[35965]: pgmap v270: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:53 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2550586673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:30:53.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:53 smithi117 ceph-mon[35788]: pgmap v270: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:30:53.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:53 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2550586673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:30:54.301 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:30:54.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:30:54.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:30:54.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 -- 172.21.15.116:0/1596844033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41016b0 msgr2=0x7fd2c4101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/1596844033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41016b0 0x7fd2c4101b30 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009930 tx=0x7fd2c002bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 -- 172.21.15.116:0/1596844033 shutdown_connections 2024-03-24T09:30:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/1596844033 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c4102070 0x7fd2c410e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/1596844033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41016b0 0x7fd2c4101b30 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/1596844033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2c41076c0 0x7fd2c4107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 -- 172.21.15.116:0/1596844033 >> 172.21.15.116:0/1596844033 conn(0x7fd2c40fd1a0 msgr2=0x7fd2c40ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 -- 172.21.15.116:0/1596844033 shutdown_connections 2024-03-24T09:30:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.419+0000 7fd2cc56c640 1 -- 172.21.15.116:0/1596844033 wait complete. 2024-03-24T09:30:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 Processor -- start 2024-03-24T09:30:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 -- start start 2024-03-24T09:30:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c41016b0 0x7fd2c419d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2c4102070 0x7fd2c419de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 0x7fd2c4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4198110 con 0x7fd2c41076c0 2024-03-24T09:30:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4198280 con 0x7fd2c4102070 2024-03-24T09:30:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cc56c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c4198440 con 0x7fd2c41016b0 2024-03-24T09:30:54.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 0x7fd2c4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 0x7fd2c4197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45478/0 (socket says 172.21.15.116:45478) 2024-03-24T09:30:54.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 -- 172.21.15.116:0/3160466211 learned_addr learned my addr 172.21.15.116:0/3160466211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2c4102070 0x7fd2c419de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2ca308640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c41016b0 0x7fd2c419d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 -- 172.21.15.116:0/3160466211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c41016b0 msgr2=0x7fd2c419d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c41016b0 0x7fd2c419d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 -- 172.21.15.116:0/3160466211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2c4102070 msgr2=0x7fd2c419de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2c4102070 0x7fd2c419de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.420+0000 7fd2cab09640 1 -- 172.21.15.116:0/3160466211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c0009560 con 0x7fd2c41076c0 2024-03-24T09:30:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.421+0000 7fd2cab09640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 0x7fd2c4197ac0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00eb10 tx=0x7fd2bc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.421+0000 7fd2b77fe640 1 -- 172.21.15.116:0/3160466211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc004b20 con 0x7fd2c41076c0 2024-03-24T09:30:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.421+0000 7fd2b77fe640 1 -- 172.21.15.116:0/3160466211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2bc004c80 con 0x7fd2c41076c0 2024-03-24T09:30:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.421+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c4198720 con 0x7fd2c41076c0 2024-03-24T09:30:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.421+0000 7fd2b77fe640 1 -- 172.21.15.116:0/3160466211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc010690 con 0x7fd2c41076c0 2024-03-24T09:30:54.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.421+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c41a49b0 con 0x7fd2c41076c0 2024-03-24T09:30:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.423+0000 7fd2b77fe640 1 -- 172.21.15.116:0/3160466211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd2bc005020 con 0x7fd2c41076c0 2024-03-24T09:30:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.423+0000 7fd2cf229740 1 Processor -- start 2024-03-24T09:30:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 -- start start 2024-03-24T09:30:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a20bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a214720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a217050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559c0a179530 con 0x559c0a20c080 2024-03-24T09:30:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559c0a176a60 con 0x559c0a214c60 2024-03-24T09:30:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cf229740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559c0a172240 con 0x559c0a20b760 2024-03-24T09:30:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2b77fe640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2b00a2230 0x7fd2b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2ca308640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a214720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2ca308640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a214720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45482/0 (socket says 172.21.15.116:45482) 2024-03-24T09:30:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2ca308640 1 -- 172.21.15.116:0/1844886873 learned_addr learned my addr 172.21.15.116:0/1844886873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a20bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cab09640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2b00a2230 0x7fd2b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a217050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2b77fe640 1 -- 172.21.15.116:0/3160466211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fd2bc014070 con 0x7fd2c41076c0 2024-03-24T09:30:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1844886873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 msgr2=0x559c0a217050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a217050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1844886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 msgr2=0x559c0a214720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a214720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1844886873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c0a1708c0 con 0x559c0a20b760 2024-03-24T09:30:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2ca308640 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a214720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.424+0000 7fd2cab09640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2b00a2230 0x7fd2b00a46f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0cc3b0 tx=0x7fd2bc01bb30 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a20bb40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009230 tx=0x7fd2c0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=675413f5e48617c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2b4bfe640 1 -- 172.21.15.116:0/1844886873 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0005d50 con 0x559c0a20b760 2024-03-24T09:30:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2b4bfe640 1 -- 172.21.15.116:0/1844886873 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c002de90 con 0x559c0a20b760 2024-03-24T09:30:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2b4bfe640 1 -- 172.21.15.116:0/1844886873 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0003860 con 0x559c0a20b760 2024-03-24T09:30:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 -- 172.21.15.116:0/1844886873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 msgr2=0x559c0a20bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a20bb40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009230 tx=0x7fd2c0009260 comp rx=0 tx=0).stop 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 -- 172.21.15.116:0/1844886873 shutdown_connections 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a217050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a214720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1844886873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a20bb40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.425+0000 7fd2cf229740 1 -- 172.21.15.116:0/1844886873 >> 172.21.15.116:0/1844886873 conn(0x559c0a205e90 msgr2=0x559c0a206270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 -- 172.21.15.116:0/1844886873 shutdown_connections 2024-03-24T09:30:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 -- 172.21.15.116:0/1844886873 wait complete. 2024-03-24T09:30:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 -- start start 2024-03-24T09:30:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 Processor -- start 2024-03-24T09:30:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 0x559c0a211c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a2121d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a214c60 0x559c0a212710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559c0a116f80 con 0x559c0a20b760 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559c09e454b0 con 0x559c0a20c080 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559c0a0f7ad0 con 0x559c0a214c60 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2cf229740 10 client.0 _ll_register_callbacks cb 0x559c0a12ab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2c9b07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 0x559c0a211c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2c9b07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 0x559c0a211c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45484/0 (socket says 172.21.15.116:45484) 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1187387557 learned_addr learned my addr 172.21.15.116:0/1187387557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a214c60 0x559c0a212710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.426+0000 7fd2ca308640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a2121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1187387557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a214c60 msgr2=0x559c0a212710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a214c60 0x559c0a212710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 msgr2=0x559c0a2121d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a2121d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2c9b07640 1 -- 172.21.15.116:0/1187387557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c410ee80 con 0x559c0a20b760 2024-03-24T09:30:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 0x559c0a211c90 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0004650 tx=0x7fd2c00048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cb7882d37eaaa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2adffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2adffb640 1 -- 172.21.15.116:0/1187387557 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0003860 con 0x559c0a20b760 2024-03-24T09:30:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2adffb640 1 -- 172.21.15.116:0/1187387557 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fd2c00039c0 con 0x559c0a20b760 2024-03-24T09:30:54.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559c0a1967a0 con 0x559c0a20b760 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2adffb640 1 -- 172.21.15.116:0/1187387557 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd2c0003b80 con 0x559c0a20b760 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2cf229740 10 client.18183 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559c0a1286d0 con 0x559c0a20b760 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2adffb640 1 -- 172.21.15.116:0/1187387557 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559c0a1967a0 con 0x559c0a20b760 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2adffb640 1 -- 172.21.15.116:0/1187387557 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd2c0059020 con 0x559c0a20b760 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2cf229740 10 client.18183 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2cf229740 10 client.18183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.427+0000 7fd2cf229740 20 client.18183 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.428+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a224a20 0x559c0a226e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.428+0000 7fd2cf229740 4 client.18183 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.428+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c0a1681f0 con 0x559c0a224a20 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.428+0000 7fd2ca308640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a224a20 0x559c0a226e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.428+0000 7fd2ca308640 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a224a20 0x559c0a226e40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ea745b2f5051fd6 server_cookie=ebc99e57d452916f in_seq=0 out_seq=0 2024-03-24T09:30:54.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.428+0000 7fd2adffb640 10 client.18183 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2adffb640 1 -- 172.21.15.116:0/1187387557 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c0a1681f0 con 0x559c0a224a20 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2adffb640 10 client.18183 handle_command_reply: tid=0 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 client.18183 shutdown 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 msgr2=0x559c0a211c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 0x559c0a211c90 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0004650 tx=0x7fd2c00048a0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 shutdown_connections 2024-03-24T09:30:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a214c60 0x559c0a212710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a224a20 0x559c0a226e40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a2121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 --2- 172.21.15.116:0/1187387557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20b760 0x559c0a211c90 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2adffb640 0 client.18183 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 >> 172.21.15.116:0/1187387557 conn(0x559c0a205e90 msgr2=0x559c0a207be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 shutdown_connections 2024-03-24T09:30:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 1 -- 172.21.15.116:0/1187387557 wait complete. 2024-03-24T09:30:54.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.429+0000 7fd2cf229740 20 client.18183 trim_cache size 0 max 16384 2024-03-24T09:30:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.446+0000 7fd2cf229740 1 Processor -- start 2024-03-24T09:30:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.446+0000 7fd2cf229740 1 -- start start 2024-03-24T09:30:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a2143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a220490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a22c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2cf229740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559c0a205670 con 0x559c0a20c080 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2cf229740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559c0a221780 con 0x559c0a214c60 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2cf229740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559c0a201650 con 0x559c0a20b760 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a220490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a220490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45486/0 (socket says 172.21.15.116:45486) 2024-03-24T09:30:54.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 -- 172.21.15.116:0/3478998684 learned_addr learned my addr 172.21.15.116:0/3478998684 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a2143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a22c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 -- 172.21.15.116:0/3478998684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 msgr2=0x559c0a2143c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a2143c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 -- 172.21.15.116:0/3478998684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 msgr2=0x559c0a22c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a22c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 -- 172.21.15.116:0/3478998684 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c0a10cc60 con 0x559c0a20c080 2024-03-24T09:30:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2ca308640 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a220490 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009530 tx=0x7fd2b8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b8f232c96948f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2affff640 1 -- 172.21.15.116:0/3478998684 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8017040 con 0x559c0a20c080 2024-03-24T09:30:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2affff640 1 -- 172.21.15.116:0/3478998684 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2b8018040 con 0x559c0a20c080 2024-03-24T09:30:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.447+0000 7fd2affff640 1 -- 172.21.15.116:0/3478998684 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b801ed70 con 0x559c0a20c080 2024-03-24T09:30:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- 172.21.15.116:0/3478998684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 msgr2=0x559c0a220490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a220490 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009530 tx=0x7fd2b8005f90 comp rx=0 tx=0).stop 2024-03-24T09:30:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- 172.21.15.116:0/3478998684 shutdown_connections 2024-03-24T09:30:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a214c60 0x559c0a22c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a20c080 0x559c0a220490 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- 172.21.15.116:0/3478998684 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a2143c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- 172.21.15.116:0/3478998684 >> 172.21.15.116:0/3478998684 conn(0x559c0a205e90 msgr2=0x559c0a211c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- 172.21.15.116:0/3478998684 shutdown_connections 2024-03-24T09:30:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- 172.21.15.116:0/3478998684 wait complete. 2024-03-24T09:30:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- start start 2024-03-24T09:30:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 Processor -- start 2024-03-24T09:30:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a21cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a219a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 0x559c0a219fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559c0a21d000 con 0x559c0a214c60 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559c0a21d140 con 0x559c0a20c080 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.448+0000 7fd2cf229740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559c0a21d300 con 0x559c0a20b760 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2cf229740 10 client.0 _ll_register_callbacks cb 0x7fd2c002cbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2c9b07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 0x559c0a219fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a219a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2c9b07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 0x559c0a219fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45494/0 (socket says 172.21.15.116:45494) 2024-03-24T09:30:54.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2c9b07640 1 -- 172.21.15.116:0/2637664068 learned_addr learned my addr 172.21.15.116:0/2637664068 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a21cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 -- 172.21.15.116:0/2637664068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 msgr2=0x559c0a21cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a21cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 -- 172.21.15.116:0/2637664068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 msgr2=0x559c0a219fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 0x559c0a219fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 -- 172.21.15.116:0/2637664068 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b8009b60 con 0x559c0a20c080 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 0x559c0a219fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2ca308640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a219a90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x559c0a2247a0 tx=0x7fd2b801fc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ebc9d4ee7064d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2adffb640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2adffb640 1 -- 172.21.15.116:0/2637664068 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b801e620 con 0x559c0a20c080 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2c002de90 con 0x559c0a20c080 2024-03-24T09:30:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2adffb640 1 -- 172.21.15.116:0/2637664068 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fd2b801e780 con 0x559c0a20c080 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2adffb640 1 -- 172.21.15.116:0/2637664068 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd2b8005040 con 0x559c0a20c080 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2cf229740 10 client.28004 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.449+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559c0a132970 con 0x559c0a20c080 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2adffb640 1 -- 172.21.15.116:0/2637664068 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2c002de90 con 0x559c0a20c080 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2adffb640 1 -- 172.21.15.116:0/2637664068 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd2b80206b0 con 0x559c0a20c080 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2cf229740 10 client.28004 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:54.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2cf229740 10 client.28004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2cf229740 20 client.28004 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2cf229740 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a28be00 0x559c0a28e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2cf229740 4 client.28004 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559c09e71860 con 0x559c0a28be00 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a28be00 0x559c0a28e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2adffb640 10 client.28004 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.450+0000 7fd2c9b07640 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a28be00 0x559c0a28e220 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5664e48a1a1e0d1b server_cookie=1be0ac51871fe24e in_seq=0 out_seq=0 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.451+0000 7fd2adffb640 1 -- 172.21.15.116:0/2637664068 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559c09e71860 con 0x559c0a28be00 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.451+0000 7fd2adffb640 10 client.28004 handle_command_reply: tid=0 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.451+0000 7fd2cf229740 1 client.28004 shutdown 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 msgr2=0x559c0a219a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a219a90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x559c0a2247a0 tx=0x7fd2b801fc10 comp rx=0 tx=0).stop 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 shutdown_connections 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559c0a214c60 0x559c0a219fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559c0a28be00 0x559c0a28e220 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559c0a20c080 0x559c0a219a90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 --2- 172.21.15.116:0/2637664068 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559c0a20b760 0x559c0a21cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2adffb640 0 client.28004 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 >> 172.21.15.116:0/2637664068 conn(0x559c0a205e90 msgr2=0x559c0a204ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 shutdown_connections 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 1 -- 172.21.15.116:0/2637664068 wait complete. 2024-03-24T09:30:54.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cf229740 20 client.28004 trim_cache size 0 max 16384 2024-03-24T09:30:54.470 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:30:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:30:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "b3f7385a-85e0-4b83-8c42-2b8afa0bc4ef", 2024-03-24T09:30:54.470 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:30:54.470 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:30:54.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2b00a2230 msgr2=0x7fd2b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2b00a2230 0x7fd2b00a46f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0cc3b0 tx=0x7fd2bc01bb30 comp rx=0 tx=0).stop 2024-03-24T09:30:54.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 msgr2=0x7fd2c4197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 0x7fd2c4197ac0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc00eb10 tx=0x7fd2bc00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 shutdown_connections 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2c41076c0 0x7fd2c4197ac0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2c4102070 0x7fd2c419de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2b00a2230 0x7fd2b00a46f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 --2- 172.21.15.116:0/3160466211 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2c41016b0 0x7fd2c419d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 >> 172.21.15.116:0/3160466211 conn(0x7fd2c40fd1a0 msgr2=0x7fd2c40fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 shutdown_connections 2024-03-24T09:30:54.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.452+0000 7fd2cc56c640 1 -- 172.21.15.116:0/3160466211 wait complete. 2024-03-24T09:30:54.474 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:30:54.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.594+0000 7fefa63b6640 1 -- 172.21.15.116:0/4274866377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 msgr2=0x7fefa0100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.594+0000 7fefa63b6640 1 --2- 172.21.15.116:0/4274866377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 0x7fefa0100070 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef94009930 tx=0x7fef9402bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 -- 172.21.15.116:0/4274866377 shutdown_connections 2024-03-24T09:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 --2- 172.21.15.116:0/4274866377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01005b0 0x7fefa010e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 --2- 172.21.15.116:0/4274866377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 0x7fefa0100070 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 --2- 172.21.15.116:0/4274866377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fefa01076d0 0x7fefa0107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 -- 172.21.15.116:0/4274866377 >> 172.21.15.116:0/4274866377 conn(0x7fefa0076bd0 msgr2=0x7fefa00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 -- 172.21.15.116:0/4274866377 shutdown_connections 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 -- 172.21.15.116:0/4274866377 wait complete. 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 Processor -- start 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.595+0000 7fefa63b6640 1 -- start start 2024-03-24T09:30:54.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa63b6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 0x7fefa019b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa63b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fefa01005b0 0x7fefa019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa63b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 0x7fefa0195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa63b6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fefa0195ee0 con 0x7fefa01076d0 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa63b6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fefa0196050 con 0x7fefa01005b0 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa63b6640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fefa0196210 con 0x7fefa00ffc10 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 0x7fefa0195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 0x7fefa0195890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45514/0 (socket says 172.21.15.116:45514) 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 -- 172.21.15.116:0/257739014 learned_addr learned my addr 172.21.15.116:0/257739014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fef9ffff640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 0x7fefa019b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fefa01005b0 0x7fefa019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 -- 172.21.15.116:0/257739014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 msgr2=0x7fefa019b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 0x7fefa019b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 -- 172.21.15.116:0/257739014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fefa01005b0 msgr2=0x7fefa019bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fefa01005b0 0x7fefa019bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.596+0000 7fefa4953640 1 -- 172.21.15.116:0/257739014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9800dc50 con 0x7fefa01076d0 2024-03-24T09:30:54.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.597+0000 7fefa4953640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 0x7fefa0195890 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fef98010130 tx=0x7fef980105f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.597+0000 7fef9d7fa640 1 -- 172.21.15.116:0/257739014 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9801b070 con 0x7fefa01076d0 2024-03-24T09:30:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.597+0000 7fef9d7fa640 1 -- 172.21.15.116:0/257739014 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef98013040 con 0x7fefa01076d0 2024-03-24T09:30:54.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.597+0000 7fef9d7fa640 1 -- 172.21.15.116:0/257739014 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef98012050 con 0x7fefa01076d0 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.597+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef94009560 con 0x7fefa01076d0 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.597+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa01a25a0 con 0x7fefa01076d0 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fef9d7fa640 1 -- 172.21.15.116:0/257739014 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fef9800d2e0 con 0x7fefa01076d0 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fefa8f91740 1 Processor -- start 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fefa8f91740 1 -- start start 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac755b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fef9d7fa640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef880a22b0 0x7fef880a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.600+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac80ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa8f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577eab99f30 con 0x5577eac751d0 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa8f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5577eabe3810 con 0x5577eac7e6d0 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa8f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5577eabe0d40 con 0x5577eac75af0 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac755b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9d7fa640 1 -- 172.21.15.116:0/257739014 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fef980d1f80 con 0x7fefa01076d0 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac755b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45524/0 (socket says 172.21.15.116:45524) 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac80ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9f7fe640 1 -- 172.21.15.116:0/673372752 learned_addr learned my addr 172.21.15.116:0/673372752 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef880a22b0 0x7fef880a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9ffff640 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 -- 172.21.15.116:0/673372752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 msgr2=0x5577eac7e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 -- 172.21.15.116:0/673372752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 msgr2=0x5577eac755b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac755b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 -- 172.21.15.116:0/673372752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577eab9dea0 con 0x5577eac7e6d0 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac755b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef880a22b0 0x7fef880a4770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fef94009900 tx=0x7fef94005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.601+0000 7fefa4953640 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac80ac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fef980cf2c0 tx=0x7fef9809e3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=908afc45e7471e0b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.602+0000 7fef867fc640 1 -- 172.21.15.116:0/673372752 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980b0bd0 con 0x5577eac7e6d0 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.602+0000 7fef867fc640 1 -- 172.21.15.116:0/673372752 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9809ec60 con 0x5577eac7e6d0 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.602+0000 7fef867fc640 1 -- 172.21.15.116:0/673372752 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980a6d90 con 0x5577eac7e6d0 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 -- 172.21.15.116:0/673372752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 msgr2=0x5577eac80ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac80ac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fef980cf2c0 tx=0x7fef9809e3f0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 -- 172.21.15.116:0/673372752 shutdown_connections 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac80ac0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 --2- 172.21.15.116:0/673372752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac755b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 -- 172.21.15.116:0/673372752 >> 172.21.15.116:0/673372752 conn(0x5577eac6f900 msgr2=0x5577eac6fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 -- 172.21.15.116:0/673372752 shutdown_connections 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 -- 172.21.15.116:0/673372752 wait complete. 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.603+0000 7fefa8f91740 1 -- start start 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 Processor -- start 2024-03-24T09:30:54.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac7b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac7c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577eabdc520 con 0x5577eac751d0 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac7b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5577ea8af430 con 0x5577eac7e6d0 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5577ea86be00 con 0x5577eac75af0 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac7b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45534/0 (socket says 172.21.15.116:45534) 2024-03-24T09:30:54.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 -- 172.21.15.116:0/2073395983 learned_addr learned my addr 172.21.15.116:0/2073395983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa8f91740 10 client.0 _ll_register_callbacks cb 0x5577eab7ad90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fefa4953640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac7c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 -- 172.21.15.116:0/2073395983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 msgr2=0x5577eac7bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 -- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 msgr2=0x5577eac7c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac7c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.604+0000 7fef9ffff640 1 -- 172.21.15.116:0/2073395983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa01a25a0 con 0x5577eac751d0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fef9ffff640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac7b730 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fef90012700 tx=0x7fef90012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241adc5bae872d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fef7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fef7f7fe640 1 -- 172.21.15.116:0/2073395983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90017070 con 0x5577eac751d0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fef7f7fe640 1 -- 172.21.15.116:0/2073395983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fef90004050 con 0x5577eac751d0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5577ea8db7e0 con 0x5577eac751d0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fef7f7fe640 1 -- 172.21.15.116:0/2073395983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fef90004210 con 0x5577eac751d0 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fefa8f91740 10 client.18201 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:54.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5577eabdaba0 con 0x5577eac751d0 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.605+0000 7fef7f7fe640 1 -- 172.21.15.116:0/2073395983 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5577ea8db7e0 con 0x5577eac751d0 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fef7f7fe640 1 -- 172.21.15.116:0/2073395983 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fef90003950 con 0x5577eac751d0 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fefa8f91740 10 client.18201 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fefa8f91740 10 client.18201 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fefa8f91740 20 client.18201 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fefa8f91740 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eac8eba0 0x5577eac90fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fefa8f91740 4 client.18201 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577ea918720 con 0x5577eac8eba0 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.606+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eac8eba0 0x5577eac90fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.607+0000 7fef7f7fe640 10 client.18201 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.607+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eac8eba0 0x5577eac90fe0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94b8fcb2a6cac731 server_cookie=58e7ab983fc80560 in_seq=0 out_seq=0 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fef7f7fe640 1 -- 172.21.15.116:0/2073395983 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577ea918720 con 0x5577eac8eba0 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fef7f7fe640 10 client.18201 handle_command_reply: tid=0 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 client.18201 shutdown 2024-03-24T09:30:54.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 msgr2=0x5577eac7b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac7b730 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fef90012700 tx=0x7fef90012bc0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 shutdown_connections 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eac8eba0 0x5577eac90fe0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eac7c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac75af0 0x5577eac7bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fef7f7fe640 0 client.18201 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 --2- 172.21.15.116:0/2073395983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac751d0 0x5577eac7b730 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 >> 172.21.15.116:0/2073395983 conn(0x5577eac6f900 msgr2=0x5577eac6eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 shutdown_connections 2024-03-24T09:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 1 -- 172.21.15.116:0/2073395983 wait complete. 2024-03-24T09:30:54.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.608+0000 7fefa8f91740 20 client.18201 trim_cache size 0 max 16384 2024-03-24T09:30:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 Processor -- start 2024-03-24T09:30:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 -- start start 2024-03-24T09:30:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac7de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac75af0 0x5577eac7bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 0x5577eac713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577eab61430 con 0x5577eac7e6d0 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5577eac704d0 con 0x5577eac75af0 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa8f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5577eac77300 con 0x5577eac751d0 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa4953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 0x5577eac713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac7de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa4953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 0x5577eac713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45548/0 (socket says 172.21.15.116:45548) 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.631+0000 7fefa4953640 1 -- 172.21.15.116:0/465036339 learned_addr learned my addr 172.21.15.116:0/465036339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac75af0 0x5577eac7bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fefa4953640 1 -- 172.21.15.116:0/465036339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 msgr2=0x5577eac7de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fefa4953640 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac7de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fefa4953640 1 -- 172.21.15.116:0/465036339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac75af0 msgr2=0x5577eac7bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fefa4953640 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac75af0 0x5577eac7bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fefa4953640 1 -- 172.21.15.116:0/465036339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577eab76b20 con 0x5577eac7e6d0 2024-03-24T09:30:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fefa4953640 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 0x5577eac713f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fef980cf8c0 tx=0x7fef9809edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a604755646d954 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fef85ffb640 1 -- 172.21.15.116:0/465036339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980a26b0 con 0x5577eac7e6d0 2024-03-24T09:30:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fef85ffb640 1 -- 172.21.15.116:0/465036339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9809e560 con 0x5577eac7e6d0 2024-03-24T09:30:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.632+0000 7fef85ffb640 1 -- 172.21.15.116:0/465036339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef980b20e0 con 0x5577eac7e6d0 2024-03-24T09:30:54.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 -- 172.21.15.116:0/465036339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 msgr2=0x5577eac713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 0x5577eac713f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fef980cf8c0 tx=0x7fef9809edd0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 -- 172.21.15.116:0/465036339 shutdown_connections 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac7e6d0 0x5577eac713f0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac75af0 0x5577eac7bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 --2- 172.21.15.116:0/465036339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac7de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 -- 172.21.15.116:0/465036339 >> 172.21.15.116:0/465036339 conn(0x5577eac6f900 msgr2=0x5577eac70ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 -- 172.21.15.116:0/465036339 shutdown_connections 2024-03-24T09:30:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 -- 172.21.15.116:0/465036339 wait complete. 2024-03-24T09:30:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.633+0000 7fefa8f91740 1 -- start start 2024-03-24T09:30:54.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 Processor -- start 2024-03-24T09:30:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac82270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 0x5577eac827b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eacf3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5577eac830b0 con 0x5577eac75af0 2024-03-24T09:30:54.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 0x5577eac827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5577eacf4460 con 0x5577eac7e6d0 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5577eacf45a0 con 0x5577eac751d0 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 0x5577eac827b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45554/0 (socket says 172.21.15.116:45554) 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fef9f7fe640 1 -- 172.21.15.116:0/357679917 learned_addr learned my addr 172.21.15.116:0/357679917 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fef9ffff640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac82270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa4953640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eacf3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.634+0000 7fefa8f91740 10 client.0 _ll_register_callbacks cb 0x7fef900097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:30:54.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef9f7fe640 1 -- 172.21.15.116:0/357679917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 msgr2=0x5577eac82270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac82270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef9f7fe640 1 -- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 msgr2=0x5577eacf3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eacf3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef9f7fe640 1 -- 172.21.15.116:0/357679917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577eab9dea0 con 0x5577eac75af0 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fefa4953640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eacf3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef9f7fe640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 0x5577eac827b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fef94036040 tx=0x7fef9402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9865802001395a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:30:54.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef7f7fe640 1 -- 172.21.15.116:0/357679917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94048040 con 0x5577eac75af0 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fef7f7fe640 1 -- 172.21.15.116:0/357679917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9554+0+0 (secure 0 0 0) 0x7fef94056500 con 0x5577eac75af0 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.635+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef9809ec60 con 0x5577eac75af0 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fef7f7fe640 1 -- 172.21.15.116:0/357679917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fef94004510 con 0x5577eac75af0 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 10 client.18207 fetch_fsmap learned FSMap version 35 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5577eab9c700 con 0x5577eac75af0 2024-03-24T09:30:54.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fef7f7fe640 1 -- 172.21.15.116:0/357679917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef9809ec60 con 0x5577eac75af0 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fef7f7fe640 1 -- 172.21.15.116:0/357679917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fef94004dc0 con 0x5577eac75af0 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 10 client.18207 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 20 client.18207 populate_metadata read hostname 'smithi116' 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eacf5db0 0x5577eacf81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.636+0000 7fefa8f91740 4 client.18207 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.637+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5577ea967890 con 0x5577eacf5db0 2024-03-24T09:30:54.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.637+0000 7fef9ffff640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eacf5db0 0x5577eacf81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:30:54.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.637+0000 7fef9ffff640 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eacf5db0 0x5577eacf81d0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78d0cc1344cabc4f server_cookie=624b17ed56f4de93 in_seq=0 out_seq=0 2024-03-24T09:30:54.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.637+0000 7fef7f7fe640 10 client.18207 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fef7f7fe640 1 -- 172.21.15.116:0/357679917 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5577ea967890 con 0x5577eacf5db0 2024-03-24T09:30:54.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fef7f7fe640 10 client.18207 handle_command_reply: tid=0 2024-03-24T09:30:54.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 client.18207 shutdown 2024-03-24T09:30:54.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 msgr2=0x5577eac827b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 0x5577eac827b0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fef94036040 tx=0x7fef9402bc30 comp rx=0 tx=0).stop 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 shutdown_connections 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5577eacf5db0 0x5577eacf81d0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577eac7e6d0 0x5577eacf3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5577eac75af0 0x5577eac827b0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 --2- 172.21.15.116:0/357679917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5577eac751d0 0x5577eac82270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 >> 172.21.15.116:0/357679917 conn(0x5577eac6f900 msgr2=0x5577eac70ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fef7f7fe640 0 client.18207 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:30:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 shutdown_connections 2024-03-24T09:30:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 1 -- 172.21.15.116:0/357679917 wait complete. 2024-03-24T09:30:54.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.638+0000 7fefa8f91740 20 client.18207 trim_cache size 0 max 16384 2024-03-24T09:30:54.651 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "b3f7385a-85e0-4b83-8c42-2b8afa0bc4ef": { 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "b3f7385a-85e0-4b83-8c42-2b8afa0bc4ef", 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:30:54.652 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:30:54.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:30:54.653 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:30:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef880a22b0 msgr2=0x7fef880a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef880a22b0 0x7fef880a4770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fef94009900 tx=0x7fef94005fd0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 msgr2=0x7fefa0195890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 0x7fefa0195890 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fef98010130 tx=0x7fef980105f0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 shutdown_connections 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fefa01076d0 0x7fefa0195890 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fefa01005b0 0x7fefa019bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef880a22b0 0x7fef880a4770 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 --2- 172.21.15.116:0/257739014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fefa00ffc10 0x7fefa019b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:30:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 >> 172.21.15.116:0/257739014 conn(0x7fefa0076bd0 msgr2=0x7fefa00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:30:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 shutdown_connections 2024-03-24T09:30:54.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:30:54.639+0000 7fefa63b6640 1 -- 172.21.15.116:0/257739014 wait complete. 2024-03-24T09:30:54.656 INFO:tasks.cephfs.filesystem:scrub status for tag:b3f7385a-85e0-4b83-8c42-2b8afa0bc4ef - {'path': '/', 'tag': 'b3f7385a-85e0-4b83-8c42-2b8afa0bc4ef', 'options': 'recursive,force'} 2024-03-24T09:30:55.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: pgmap v271: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:55.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:30:55.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:55.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:55.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:55.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:55.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:55.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:55 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: pgmap v271: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:55.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:30:55.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:55.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:55.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:55.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:55.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:55.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:55 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: pgmap v271: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:55.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:30:55.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:55.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:55.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:55.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:55.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:55.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:55.571 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:55 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:56 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:30:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:56 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:56.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:56 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:56.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:56 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:30:56.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:56 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:56.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:56 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:56.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:56 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:30:56.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:56 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:56.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:56 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:30:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:57 smithi143 ceph-mon[35965]: pgmap v272: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:57.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:57 smithi117 ceph-mon[35788]: pgmap v272: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:57.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:57 smithi116 ceph-mon[32207]: pgmap v272: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:30:59.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:30:59 smithi143 ceph-mon[35965]: pgmap v273: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:59.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:30:59 smithi117 ceph-mon[35788]: pgmap v273: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:30:59.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:30:59 smithi116 ceph-mon[32207]: pgmap v273: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:01.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:01 smithi143 ceph-mon[35965]: pgmap v274: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:01 smithi117 ceph-mon[35788]: pgmap v274: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:01 smithi116 ceph-mon[32207]: pgmap v274: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:03.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:03 smithi116 ceph-mon[32207]: pgmap v275: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:31:03.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:03 smithi143 ceph-mon[35965]: pgmap v275: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:31:03.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:03 smithi117 ceph-mon[35788]: pgmap v275: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:31:05.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:05 smithi143 ceph-mon[35965]: pgmap v276: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:31:05.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:05 smithi117 ceph-mon[35788]: pgmap v276: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:31:05.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:05 smithi116 ceph-mon[32207]: pgmap v276: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:31:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:07 smithi143 ceph-mon[35965]: pgmap v277: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:31:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:07 smithi117 ceph-mon[35788]: pgmap v277: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:31:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:07 smithi116 ceph-mon[32207]: pgmap v277: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:31:08.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:08 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:08.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:08 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:08.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:08 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:08.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:08 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:08.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:08 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:08.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:08 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:09 smithi143 ceph-mon[35965]: pgmap v278: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:31:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:09 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:09 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:09 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:09.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:09 smithi143 ceph-mon[35965]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T09:31:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:09 smithi117 ceph-mon[35788]: pgmap v278: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:31:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:09 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:09 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:09 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:09.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:09 smithi117 ceph-mon[35788]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T09:31:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:09 smithi116 ceph-mon[32207]: pgmap v278: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:31:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:09 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:09 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:09 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:09.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:09 smithi116 ceph-mon[32207]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T09:31:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:11 smithi143 ceph-mon[35965]: pgmap v280: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:31:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:11 smithi117 ceph-mon[35788]: pgmap v280: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:31:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:11 smithi116 ceph-mon[32207]: pgmap v280: 129 pgs: 129 active+clean; 150 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:31:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:13 smithi143 ceph-mon[35965]: pgmap v281: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:13 smithi117 ceph-mon[35788]: pgmap v281: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:13 smithi116 ceph-mon[32207]: pgmap v281: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:15 smithi117 ceph-mon[35788]: pgmap v282: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:15.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:15 smithi117 ceph-mon[35788]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T09:31:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:15 smithi116 ceph-mon[32207]: pgmap v282: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:15 smithi116 ceph-mon[32207]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T09:31:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:15 smithi143 ceph-mon[35965]: pgmap v282: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:15 smithi143 ceph-mon[35965]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T09:31:17.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:17 smithi117 ceph-mon[35788]: pgmap v284: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:31:17.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:17 smithi116 ceph-mon[32207]: pgmap v284: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:31:17.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:17 smithi143 ceph-mon[35965]: pgmap v284: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:31:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:19 smithi117 ceph-mon[35788]: pgmap v285: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 219 B/s wr, 11 op/s 2024-03-24T09:31:19.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:19 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:31:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:19 smithi116 ceph-mon[32207]: pgmap v285: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 219 B/s wr, 11 op/s 2024-03-24T09:31:19.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:19 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:31:19.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:19 smithi143 ceph-mon[35965]: pgmap v285: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 219 B/s wr, 11 op/s 2024-03-24T09:31:19.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:19 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:31:20.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:20 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:31:20.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:20 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:31:20.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:20 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:31:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:20 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:31:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:20 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:31:20.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:20 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:31:20.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:20 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:31:20.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:20 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:31:20.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:20 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:31:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:21 smithi143 ceph-mon[35965]: pgmap v286: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:21.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:21 smithi117 ceph-mon[35788]: pgmap v286: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:21.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:21 smithi116 ceph-mon[32207]: pgmap v286: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:31:23.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:23 smithi143 ceph-mon[35965]: pgmap v287: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:31:23.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:23 smithi117 ceph-mon[35788]: pgmap v287: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:31:23.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:23 smithi116 ceph-mon[32207]: pgmap v287: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:31:24.657 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:31:24.658 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:31:24.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 -- 172.21.15.116:0/1069314008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c101680 msgr2=0x7fec1c101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 --2- 172.21.15.116:0/1069314008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c101680 0x7fec1c101b00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009930 tx=0x7fec0c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 -- 172.21.15.116:0/1069314008 shutdown_connections 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 --2- 172.21.15.116:0/1069314008 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c102040 0x7fec1c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 --2- 172.21.15.116:0/1069314008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c101680 0x7fec1c101b00 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 --2- 172.21.15.116:0/1069314008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec1c107690 0x7fec1c107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 -- 172.21.15.116:0/1069314008 >> 172.21.15.116:0/1069314008 conn(0x7fec1c0fd170 msgr2=0x7fec1c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 -- 172.21.15.116:0/1069314008 shutdown_connections 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.772+0000 7fec243b6640 1 -- 172.21.15.116:0/1069314008 wait complete. 2024-03-24T09:31:24.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 Processor -- start 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 -- start start 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 0x7fec1c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec1c102040 0x7fec1c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 0x7fec1c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c198120 con 0x7fec1c107690 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c198290 con 0x7fec1c102040 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec243b6640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c198450 con 0x7fec1c101680 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 0x7fec1c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec22152640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 0x7fec1c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec22152640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 0x7fec1c19d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:59922/0 (socket says 172.21.15.116:59922) 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec22152640 1 -- 172.21.15.116:0/19049051 learned_addr learned my addr 172.21.15.116:0/19049051 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 0x7fec1c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54526/0 (socket says 172.21.15.116:54526) 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.773+0000 7fec21951640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec1c102040 0x7fec1c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22152640 1 -- 172.21.15.116:0/19049051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec1c102040 msgr2=0x7fec1c19de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22152640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec1c102040 0x7fec1c19de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22152640 1 -- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 msgr2=0x7fec1c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22152640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 0x7fec1c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22152640 1 -- 172.21.15.116:0/19049051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0c009560 con 0x7fec1c101680 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22953640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 0x7fec1c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec22152640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 0x7fec1c19d920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec1800e9f0 tx=0x7fec1800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec137fe640 1 -- 172.21.15.116:0/19049051 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18009700 con 0x7fec1c101680 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec137fe640 1 -- 172.21.15.116:0/19049051 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec1800cd80 con 0x7fec1c101680 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1c198730 con 0x7fec1c101680 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec137fe640 1 -- 172.21.15.116:0/19049051 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18018840 con 0x7fec1c101680 2024-03-24T09:31:24.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.774+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c1a49c0 con 0x7fec1c101680 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.777+0000 7fec137fe640 1 -- 172.21.15.116:0/19049051 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fec180189a0 con 0x7fec1c101680 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.777+0000 7fec26f91740 1 Processor -- start 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.777+0000 7fec26f91740 1 -- start start 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec26f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec26f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbeaa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec26f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeacac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec26f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbdc5f30 con 0x55ccfbea11d0 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec26f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbe0f810 con 0x55ccfbea1af0 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec26f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbe0cd40 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec137fe640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec080a2230 0x7fec080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbeaa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec21951640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeacac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54530/0 (socket says 172.21.15.116:54530) 2024-03-24T09:31:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22953640 1 -- 172.21.15.116:0/936875263 learned_addr learned my addr 172.21.15.116:0/936875263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22953640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec080a2230 0x7fec080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec137fe640 1 -- 172.21.15.116:0/19049051 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fec18014070 con 0x7fec1c101680 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 -- 172.21.15.116:0/936875263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 msgr2=0x55ccfbeacac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeacac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 -- 172.21.15.116:0/936875263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 msgr2=0x55ccfbea15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 -- 172.21.15.116:0/936875263 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccfbdc9ea0 con 0x55ccfbea1af0 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22152640 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbeaa190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec180cba80 tx=0x7fec1806df00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=659c374d4f7613ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.778+0000 7fec22953640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec080a2230 0x7fec080a46f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fec1400f4f0 tx=0x7fec14011480 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec10bfe640 1 -- 172.21.15.116:0/936875263 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18010b50 con 0x55ccfbea1af0 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec10bfe640 1 -- 172.21.15.116:0/936875263 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec180cbd00 con 0x55ccfbea1af0 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec10bfe640 1 -- 172.21.15.116:0/936875263 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180bc0d0 con 0x55ccfbea1af0 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 -- 172.21.15.116:0/936875263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 msgr2=0x55ccfbeaa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbeaa190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec180cba80 tx=0x7fec1806df00 comp rx=0 tx=0).stop 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 -- 172.21.15.116:0/936875263 shutdown_connections 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeacac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbeaa190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 --2- 172.21.15.116:0/936875263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 -- 172.21.15.116:0/936875263 >> 172.21.15.116:0/936875263 conn(0x55ccfbe9b900 msgr2=0x55ccfbe9bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 -- 172.21.15.116:0/936875263 shutdown_connections 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.779+0000 7fec26f91740 1 -- 172.21.15.116:0/936875263 wait complete. 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 -- start start 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 Processor -- start 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbea8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbe08520 con 0x55ccfbea11d0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbadb430 con 0x55ccfbea1af0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ccfba97e00 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec26f91740 10 client.0 _ll_register_callbacks cb 0x55ccfbda6d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54542/0 (socket says 172.21.15.116:54542) 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 -- 172.21.15.116:0/858173710 learned_addr learned my addr 172.21.15.116:0/858173710 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec21951640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22953640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbea8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 -- 172.21.15.116:0/858173710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 msgr2=0x55ccfbea8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbea8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 -- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 msgr2=0x55ccfbea7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec22152640 1 -- 172.21.15.116:0/858173710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c1a49c0 con 0x55ccfbea11d0 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.780+0000 7fec21951640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec22152640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea7790 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fec1809aad0 tx=0x7fec180971a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd41862d3c9de791 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec05ffb640 1 -- 172.21.15.116:0/858173710 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec180a32b0 con 0x55ccfbea11d0 2024-03-24T09:31:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec05ffb640 1 -- 172.21.15.116:0/858173710 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fec180cbe70 con 0x55ccfbea11d0 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ccfbb077e0 con 0x55ccfbea11d0 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec05ffb640 1 -- 172.21.15.116:0/858173710 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec180990d0 con 0x55ccfbea11d0 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 10 client.18219 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ccfbe06ba0 con 0x55ccfbea11d0 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec05ffb640 1 -- 172.21.15.116:0/858173710 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ccfbb077e0 con 0x55ccfbea11d0 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec05ffb640 1 -- 172.21.15.116:0/858173710 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec18099c90 con 0x55ccfbea11d0 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 10 client.18219 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 10 client.18219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 20 client.18219 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbeba910 0x55ccfbebcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 4 client.18219 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.781+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccfbb44720 con 0x55ccfbeba910 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.782+0000 7fec21951640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbeba910 0x55ccfbebcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.782+0000 7fec05ffb640 10 client.18219 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.782+0000 7fec21951640 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbeba910 0x55ccfbebcd60 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d47e6e0a9ede98fd server_cookie=7c36120b92df2320 in_seq=0 out_seq=0 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec05ffb640 1 -- 172.21.15.116:0/858173710 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ccfbb44720 con 0x55ccfbeba910 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec05ffb640 10 client.18219 handle_command_reply: tid=0 2024-03-24T09:31:24.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 client.18219 shutdown 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 msgr2=0x55ccfbea7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea7790 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fec1809aad0 tx=0x7fec180971a0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 shutdown_connections 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbea8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbeba910 0x55ccfbebcd60 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 --2- 172.21.15.116:0/858173710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbea11d0 0x55ccfbea7790 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec05ffb640 0 client.18219 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:24.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 >> 172.21.15.116:0/858173710 conn(0x55ccfbe9b900 msgr2=0x55ccfbe9ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 shutdown_connections 2024-03-24T09:31:24.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 1 -- 172.21.15.116:0/858173710 wait complete. 2024-03-24T09:31:24.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.783+0000 7fec26f91740 20 client.18219 trim_cache size 0 max 16384 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 Processor -- start 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 -- start start 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbe9a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeb5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbd8d430 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbea75e0 con 0x55ccfbea11d0 2024-03-24T09:31:24.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec26f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbe9c980 con 0x55ccfbea1af0 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeb5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeb5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54544/0 (socket says 172.21.15.116:54544) 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec22953640 1 -- 172.21.15.116:0/2939431997 learned_addr learned my addr 172.21.15.116:0/2939431997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec21951640 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.800+0000 7fec22152640 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbe9a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec22953640 1 -- 172.21.15.116:0/2939431997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 msgr2=0x55ccfbea7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec22953640 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec22953640 1 -- 172.21.15.116:0/2939431997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 msgr2=0x55ccfbe9a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec22953640 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbe9a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec22953640 1 -- 172.21.15.116:0/2939431997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccfbda2b20 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec21951640 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec22953640 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeb5f60 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fec14009320 tx=0x7fec14013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74341ce64207a5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec07fff640 1 -- 172.21.15.116:0/2939431997 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14026040 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec07fff640 1 -- 172.21.15.116:0/2939431997 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec14011630 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec07fff640 1 -- 172.21.15.116:0/2939431997 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec140055a0 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec26f91740 1 -- 172.21.15.116:0/2939431997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 msgr2=0x55ccfbeb5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.801+0000 7fec26f91740 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeb5f60 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fec14009320 tx=0x7fec14013000 comp rx=0 tx=0).stop 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- 172.21.15.116:0/2939431997 shutdown_connections 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbeb5f60 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 --2- 172.21.15.116:0/2939431997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbe9a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- 172.21.15.116:0/2939431997 >> 172.21.15.116:0/2939431997 conn(0x55ccfbe9b900 msgr2=0x55ccfbea7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- 172.21.15.116:0/2939431997 shutdown_connections 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- 172.21.15.116:0/2939431997 wait complete. 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- start start 2024-03-24T09:31:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 Processor -- start 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbea4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbf1fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbea5520 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbf20140 con 0x55ccfbea11d0 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ccfbf20280 con 0x55ccfbea1af0 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec26f91740 10 client.0 _ll_register_callbacks cb 0x7fec14011810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbf1fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec22953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbf1fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54558/0 (socket says 172.21.15.116:54558) 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec22953640 1 -- 172.21.15.116:0/4025109536 learned_addr learned my addr 172.21.15.116:0/4025109536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec21951640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.802+0000 7fec22152640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbea4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec22953640 1 -- 172.21.15.116:0/4025109536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 msgr2=0x55ccfbea4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec22953640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec22953640 1 -- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 msgr2=0x55ccfbea4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec22953640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbea4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec22953640 1 -- 172.21.15.116:0/4025109536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1c10ee50 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec22953640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbf1fc00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fec140051e0 tx=0x7fec14011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ba8343d315b00b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec05ffb640 1 -- 172.21.15.116:0/4025109536 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14004a40 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec05ffb640 1 -- 172.21.15.116:0/4025109536 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fec14004ba0 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ccfbe9a930 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec05ffb640 1 -- 172.21.15.116:0/4025109536 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fec14004d60 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 10 client.18225 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7fec14019040 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec05ffb640 1 -- 172.21.15.116:0/4025109536 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ccfbe9a930 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec05ffb640 1 -- 172.21.15.116:0/4025109536 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fec14033070 con 0x55ccfbeaa6d0 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 10 client.18225 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 10 client.18225 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 20 client.18225 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbf21a30 0x55ccfbf23e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 4 client.18225 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.803+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fec180cbd00 con 0x55ccfbf21a30 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec22152640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbf21a30 0x55ccfbf23e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec05ffb640 10 client.18225 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec22152640 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbf21a30 0x55ccfbf23e50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7317c0d26c8b32a6 server_cookie=cb510135390bd95c in_seq=0 out_seq=0 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec05ffb640 1 -- 172.21.15.116:0/4025109536 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fec180cbd00 con 0x55ccfbf21a30 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec05ffb640 10 client.18225 handle_command_reply: tid=0 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 client.18225 shutdown 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 msgr2=0x55ccfbf1fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbf1fc00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fec140051e0 tx=0x7fec14011e80 comp rx=0 tx=0).stop 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 shutdown_connections 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ccfbeaa6d0 0x55ccfbf1fc00 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ccfbea1af0 0x55ccfbea4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ccfbf21a30 0x55ccfbf23e50 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 --2- 172.21.15.116:0/4025109536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccfbea11d0 0x55ccfbea4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec05ffb640 0 client.18225 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 >> 172.21.15.116:0/4025109536 conn(0x55ccfbe9b900 msgr2=0x55ccfbe9cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 shutdown_connections 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 1 -- 172.21.15.116:0/4025109536 wait complete. 2024-03-24T09:31:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.804+0000 7fec26f91740 20 client.18225 trim_cache size 0 max 16384 2024-03-24T09:31:24.816 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:31:24.816 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:31:24.816 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:31:24.816 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:31:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec080a2230 msgr2=0x7fec080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec080a2230 0x7fec080a46f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fec1400f4f0 tx=0x7fec14011480 comp rx=0 tx=0).stop 2024-03-24T09:31:24.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 msgr2=0x7fec1c19d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 0x7fec1c19d920 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fec1800e9f0 tx=0x7fec1800eeb0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 shutdown_connections 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec1c107690 0x7fec1c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec1c102040 0x7fec1c19de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec080a2230 0x7fec080a46f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 --2- 172.21.15.116:0/19049051 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec1c101680 0x7fec1c19d920 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 >> 172.21.15.116:0/19049051 conn(0x7fec1c0fd170 msgr2=0x7fec1c0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 shutdown_connections 2024-03-24T09:31:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.805+0000 7fec243b6640 1 -- 172.21.15.116:0/19049051 wait complete. 2024-03-24T09:31:24.819 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:31:24.819 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:31:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3428190257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8100d40 msgr2=0x7f99e810d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3428190257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8100d40 0x7f99e810d4e0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f99e0009ca0 tx=0x7f99e002c180 comp rx=0 tx=0).stop 2024-03-24T09:31:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3428190257 shutdown_connections 2024-03-24T09:31:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3428190257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8100d40 0x7f99e810d4e0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3428190257 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e8106fd0 0x7f99e8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3428190257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99e81066b0 0x7f99e8106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3428190257 >> 172.21.15.116:0/3428190257 conn(0x7f99e8077710 msgr2=0x7f99e8077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:24.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3428190257 shutdown_connections 2024-03-24T09:31:24.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.920+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3428190257 wait complete. 2024-03-24T09:31:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 Processor -- start 2024-03-24T09:31:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 -- start start 2024-03-24T09:31:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e8100d40 0x7f99e819d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99e81066b0 0x7f99e819dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 0x7f99e8197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81980a0 con 0x7f99e8106fd0 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f99e8198210 con 0x7f99e81066b0 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99ec8eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f99e81983d0 con 0x7f99e8100d40 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 0x7f99e8197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 0x7f99e8197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54586/0 (socket says 172.21.15.116:54586) 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 -- 172.21.15.116:0/3177567004 learned_addr learned my addr 172.21.15.116:0/3177567004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:24.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e659c640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e8100d40 0x7f99e819d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99dffff640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99e81066b0 0x7f99e819dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 -- 172.21.15.116:0/3177567004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e8100d40 msgr2=0x7f99e819d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e8100d40 0x7f99e819d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 -- 172.21.15.116:0/3177567004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99e81066b0 msgr2=0x7f99e819dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99e81066b0 0x7f99e819dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.921+0000 7f99e6d9d640 1 -- 172.21.15.116:0/3177567004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d8009490 con 0x7f99e8106fd0 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.922+0000 7f99e6d9d640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 0x7f99e8197a50 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f99e002c790 tx=0x7f99e00346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.922+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0003d80 con 0x7f99e8106fd0 2024-03-24T09:31:24.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.922+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e0009900 con 0x7f99e8106fd0 2024-03-24T09:31:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.922+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f99e00348e0 con 0x7f99e8106fd0 2024-03-24T09:31:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.922+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e81a4760 con 0x7f99e8106fd0 2024-03-24T09:31:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.922+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e00045a0 con 0x7f99e8106fd0 2024-03-24T09:31:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.923+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e8068500 con 0x7f99e8106fd0 2024-03-24T09:31:24.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.924+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f99e003b040 con 0x7f99e8106fd0 2024-03-24T09:31:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.924+0000 7f99df7fe640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99d40a2230 0x7f99d40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.924+0000 7f99e659c640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99d40a2230 0x7f99d40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.924+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f99e00f0cd0 con 0x7f99e8106fd0 2024-03-24T09:31:24.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.925+0000 7f99e659c640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99d40a2230 0x7f99d40a46f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f99e8198b90 tx=0x7f99d8005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:24.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:24.929+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f99e00b9780 con 0x7f99e8106fd0 2024-03-24T09:31:25.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.140+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99e81106c0 con 0x7f99e8106fd0 2024-03-24T09:31:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.141+0000 7f99df7fe640 1 -- 172.21.15.116:0/3177567004 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f99e0039090 con 0x7f99e8106fd0 2024-03-24T09:31:25.142 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:31:25.142 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:31:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:31:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99d40a2230 msgr2=0x7f99d40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99d40a2230 0x7f99d40a46f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f99e8198b90 tx=0x7f99d8005f90 comp rx=0 tx=0).stop 2024-03-24T09:31:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 msgr2=0x7f99e8197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 0x7f99e8197a50 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f99e002c790 tx=0x7f99e00346f0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 shutdown_connections 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e8106fd0 0x7f99e8197a50 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99e81066b0 0x7f99e819dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99d40a2230 0x7f99d40a46f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 --2- 172.21.15.116:0/3177567004 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99e8100d40 0x7f99e819d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 >> 172.21.15.116:0/3177567004 conn(0x7f99e8077710 msgr2=0x7f99e80ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 shutdown_connections 2024-03-24T09:31:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.143+0000 7f99ec8eb640 1 -- 172.21.15.116:0/3177567004 wait complete. 2024-03-24T09:31:25.155 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:31:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.258+0000 7f353b14f640 1 -- 172.21.15.116:0/2973051328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 msgr2=0x7f3534068d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.258+0000 7f353b14f640 1 --2- 172.21.15.116:0/2973051328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f3534068d70 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f3528009930 tx=0x7f352802bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- 172.21.15.116:0/2973051328 shutdown_connections 2024-03-24T09:31:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 --2- 172.21.15.116:0/2973051328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35340692b0 0x7f353410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 --2- 172.21.15.116:0/2973051328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f3534068d70 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 --2- 172.21.15.116:0/2973051328 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534107690 0x7f3534107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- 172.21.15.116:0/2973051328 >> 172.21.15.116:0/2973051328 conn(0x7f3534077710 msgr2=0x7f3534077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- 172.21.15.116:0/2973051328 shutdown_connections 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- 172.21.15.116:0/2973051328 wait complete. 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 Processor -- start 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- start start 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f353419d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35340692b0 0x7f353419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534107690 0x7f3534197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3534198100 con 0x7f35340688f0 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3534198270 con 0x7f35340692b0 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f353b14f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3534198430 con 0x7f3534107690 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f3538eeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f353419d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f3538eeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f353419d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54592/0 (socket says 172.21.15.116:54592) 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.259+0000 7f3538eeb640 1 -- 172.21.15.116:0/1135047218 learned_addr learned my addr 172.21.15.116:0/1135047218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f35396ec640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534107690 0x7f3534197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3533fff640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35340692b0 0x7f353419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3538eeb640 1 -- 172.21.15.116:0/1135047218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534107690 msgr2=0x7f3534197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534107690 0x7f3534197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3538eeb640 1 -- 172.21.15.116:0/1135047218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35340692b0 msgr2=0x7f353419de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35340692b0 0x7f353419de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3538eeb640 1 -- 172.21.15.116:0/1135047218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3528009560 con 0x7f35340688f0 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f353419d900 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f352400e9f0 tx=0x7f352400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3531ffb640 1 -- 172.21.15.116:0/1135047218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524009700 con 0x7f35340688f0 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3534198710 con 0x7f35340688f0 2024-03-24T09:31:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3531ffb640 1 -- 172.21.15.116:0/1135047218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f352400cd80 con 0x7f35340688f0 2024-03-24T09:31:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f3531ffb640 1 -- 172.21.15.116:0/1135047218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3524018840 con 0x7f35340688f0 2024-03-24T09:31:25.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.260+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a49a0 con 0x7f35340688f0 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.262+0000 7f3531ffb640 1 -- 172.21.15.116:0/1135047218 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f35240189a0 con 0x7f35340688f0 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3531ffb640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f351c0a2230 0x7f351c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 Processor -- start 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 -- start start 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3533fff640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f351c0a2230 0x7f351c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3531ffb640 1 -- 172.21.15.116:0/1135047218 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3524014070 con 0x7f35340688f0 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d0f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d183b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d1ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3c7d840 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3c7ad70 con 0x55f3f3d0fd10 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f353de09740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3c76550 con 0x55f3f3d188f0 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d0f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d0f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54594/0 (socket says 172.21.15.116:54594) 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 -- 172.21.15.116:0/588215399 learned_addr learned my addr 172.21.15.116:0/588215399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f35396ec640 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d1ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3533fff640 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 -- 172.21.15.116:0/588215399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 msgr2=0x55f3f3d1ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d1ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 -- 172.21.15.116:0/588215399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0fd10 msgr2=0x55f3f3d183b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d183b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3538eeb640 1 -- 172.21.15.116:0/588215399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f3c0bde0 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f35396ec640 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d1ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.263+0000 7f3533fff640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f351c0a2230 0x7f351c0a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3528005ab0 tx=0x7f35280047a0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f3538eeb640 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d0f7d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f35240977a0 tx=0x7f352409acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dc212df49edbb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f351affd640 1 -- 172.21.15.116:0/588215399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35240a34c0 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f351affd640 1 -- 172.21.15.116:0/588215399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35240990d0 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f351affd640 1 -- 172.21.15.116:0/588215399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35240ce2f0 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 -- 172.21.15.116:0/588215399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 msgr2=0x55f3f3d0f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d0f7d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f35240977a0 tx=0x7f352409acd0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 -- 172.21.15.116:0/588215399 shutdown_connections 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d1ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d183b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 --2- 172.21.15.116:0/588215399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d0f7d0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 -- 172.21.15.116:0/588215399 >> 172.21.15.116:0/588215399 conn(0x55f3f3d09b20 msgr2=0x55f3f3d09f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 -- 172.21.15.116:0/588215399 shutdown_connections 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.264+0000 7f353de09740 1 -- 172.21.15.116:0/588215399 wait complete. 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 -- start start 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 Processor -- start 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d15f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d16440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3949430 con 0x55f3f3d0fd10 2024-03-24T09:31:25.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3905e00 con 0x55f3f3d188f0 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3bfb590 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f353de09740 10 client.0 _ll_register_callbacks cb 0x55f3f3c6afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f3533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d15f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f3533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d15f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54606/0 (socket says 172.21.15.116:54606) 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f3533fff640 1 -- 172.21.15.116:0/1605018956 learned_addr learned my addr 172.21.15.116:0/1605018956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.265+0000 7f35396ec640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d16440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3533fff640 1 -- 172.21.15.116:0/1605018956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 msgr2=0x55f3f3d159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3533fff640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d159c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3533fff640 1 -- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 msgr2=0x55f3f3d16440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3533fff640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d16440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3533fff640 1 -- 172.21.15.116:0/1605018956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35341a49a0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3533fff640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d15f00 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f3528009930 tx=0x7f352802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b21dbd48b4ec43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3513fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3513fff640 1 -- 172.21.15.116:0/1605018956 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35280038c0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3513fff640 1 -- 172.21.15.116:0/1605018956 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3528003a20 con 0x55f3f3d0fd10 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3f39757e0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3513fff640 1 -- 172.21.15.116:0/1605018956 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3528003be0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 10 client.18243 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f3f3c74bd0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3513fff640 1 -- 172.21.15.116:0/1605018956 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f3f39757e0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f3513fff640 1 -- 172.21.15.116:0/1605018956 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3528041460 con 0x55f3f3d0fd10 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 10 client.18243 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 10 client.18243 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 20 client.18243 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d28a30 0x55f3f3d2ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 4 client.18243 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:25.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.266+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3f39b2720 con 0x55f3f3d28a30 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.267+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d28a30 0x55f3f3d2ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.267+0000 7f3513fff640 10 client.18243 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.267+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d28a30 0x55f3f3d2ae70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4463b5414a93f489 server_cookie=f4c6f3ab1036bda5 in_seq=0 out_seq=0 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.267+0000 7f3513fff640 1 -- 172.21.15.116:0/1605018956 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f3f39b2720 con 0x55f3f3d28a30 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f3513fff640 10 client.18243 handle_command_reply: tid=0 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 client.18243 shutdown 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 msgr2=0x55f3f3d15f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d15f00 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f3528009930 tx=0x7f352802bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:25.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 shutdown_connections 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d16440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d28a30 0x55f3f3d2ae70 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d15f00 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f3513fff640 0 client.18243 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 --2- 172.21.15.116:0/1605018956 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d159c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 >> 172.21.15.116:0/1605018956 conn(0x55f3f3d09b20 msgr2=0x55f3f3d0b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 shutdown_connections 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 1 -- 172.21.15.116:0/1605018956 wait complete. 2024-03-24T09:31:25.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.268+0000 7f353de09740 20 client.18243 trim_cache size 0 max 16384 2024-03-24T09:31:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 Processor -- start 2024-03-24T09:31:25.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 -- start start 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d16260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d30730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3d09680 con 0x55f3f3d0fd10 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3d0edc0 con 0x55f3f3d188f0 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f353de09740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3d13f30 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d16260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d16260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54622/0 (socket says 172.21.15.116:54622) 2024-03-24T09:31:25.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 -- 172.21.15.116:0/1860376288 learned_addr learned my addr 172.21.15.116:0/1860376288 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3538eeb640 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f35396ec640 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d30730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 -- 172.21.15.116:0/1860376288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 msgr2=0x55f3f3d180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d180f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 -- 172.21.15.116:0/1860376288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 msgr2=0x55f3f3d30730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d30730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.281+0000 7f3533fff640 1 -- 172.21.15.116:0/1860376288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3f3c10c80 con 0x55f3f3d0fd10 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f3533fff640 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d16260 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3528006010 tx=0x7f3528005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7857517d77f28a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f351a7fc640 1 -- 172.21.15.116:0/1860376288 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528041660 con 0x55f3f3d0fd10 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f351a7fc640 1 -- 172.21.15.116:0/1860376288 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f352802de90 con 0x55f3f3d0fd10 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f351a7fc640 1 -- 172.21.15.116:0/1860376288 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528004dc0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 -- 172.21.15.116:0/1860376288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 msgr2=0x55f3f3d16260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d16260 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f3528006010 tx=0x7f3528005a20 comp rx=0 tx=0).stop 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 -- 172.21.15.116:0/1860376288 shutdown_connections 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d188f0 0x55f3f3d30730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d16260 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 --2- 172.21.15.116:0/1860376288 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 -- 172.21.15.116:0/1860376288 >> 172.21.15.116:0/1860376288 conn(0x55f3f3d09b20 msgr2=0x55f3f3d0ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 -- 172.21.15.116:0/1860376288 shutdown_connections 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.282+0000 7f353de09740 1 -- 172.21.15.116:0/1860376288 wait complete. 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 -- start start 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 Processor -- start 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d12d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d13280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d8dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3d8e2a0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3d8e3e0 con 0x55f3f3d0f3f0 2024-03-24T09:31:25.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f3f3d8e520 con 0x55f3f3d188f0 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f353de09740 10 client.0 _ll_register_callbacks cb 0x7f3524005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d13280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3538eeb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d12d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d13280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54638/0 (socket says 172.21.15.116:54638) 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 -- 172.21.15.116:0/448829826 learned_addr learned my addr 172.21.15.116:0/448829826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3538eeb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d12d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:41008/0 (socket says 172.21.15.116:41008) 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f35396ec640 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d8dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 -- 172.21.15.116:0/448829826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 msgr2=0x55f3f3d8dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d8dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 -- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0f3f0 msgr2=0x55f3f3d12d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d12d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.283+0000 7f3533fff640 1 -- 172.21.15.116:0/448829826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3f3c74bd0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3533fff640 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d13280 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3528009930 tx=0x7f352802c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f83797a0354ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3513fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3513fff640 1 -- 172.21.15.116:0/448829826 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528003ac0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3513fff640 1 -- 172.21.15.116:0/448829826 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3528003c20 con 0x55f3f3d0fd10 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35240990d0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3513fff640 1 -- 172.21.15.116:0/448829826 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3528003de0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 10 client.18249 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f3f3c36920 con 0x55f3f3d0fd10 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3513fff640 1 -- 172.21.15.116:0/448829826 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f35240990d0 con 0x55f3f3d0fd10 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3513fff640 1 -- 172.21.15.116:0/448829826 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3528068460 con 0x55f3f3d0fd10 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 10 client.18249 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 10 client.18249 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 20 client.18249 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d8f950 0x55f3f3d91d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 4 client.18249 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3f3a01890 con 0x55f3f3d8f950 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.284+0000 7f3538eeb640 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d8f950 0x55f3f3d91d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f3513fff640 10 client.18249 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f3538eeb640 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d8f950 0x55f3f3d91d70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8fcfa5d1d3e2c3c3 server_cookie=847f1c58d841ce88 in_seq=0 out_seq=0 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f3513fff640 1 -- 172.21.15.116:0/448829826 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3f3a01890 con 0x55f3f3d8f950 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f3513fff640 10 client.18249 handle_command_reply: tid=0 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 client.18249 shutdown 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 msgr2=0x55f3f3d13280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d13280 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f3528009930 tx=0x7f352802c3a0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 shutdown_connections 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f3f3d188f0 0x55f3f3d8dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x55f3f3d8f950 0x55f3f3d91d70 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f3f3d0fd10 0x55f3f3d13280 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f3513fff640 0 client.18249 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 --2- 172.21.15.116:0/448829826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f3f3d0f3f0 0x55f3f3d12d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 >> 172.21.15.116:0/448829826 conn(0x55f3f3d09b20 msgr2=0x55f3f3d0b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 shutdown_connections 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 1 -- 172.21.15.116:0/448829826 wait complete. 2024-03-24T09:31:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.285+0000 7f353de09740 20 client.18249 trim_cache size 0 max 16384 2024-03-24T09:31:25.296 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f351c0a2230 msgr2=0x7f351c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f351c0a2230 0x7f351c0a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3528005ab0 tx=0x7f35280047a0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 msgr2=0x7f353419d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f353419d900 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f352400e9f0 tx=0x7f352400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 shutdown_connections 2024-03-24T09:31:25.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f351c0a2230 0x7f351c0a46f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3534107690 0x7f3534197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35340692b0 0x7f353419de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 --2- 172.21.15.116:0/1135047218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35340688f0 0x7f353419d900 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 >> 172.21.15.116:0/1135047218 conn(0x7f3534077710 msgr2=0x7f35340fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 shutdown_connections 2024-03-24T09:31:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.286+0000 7f353b14f640 1 -- 172.21.15.116:0/1135047218 wait complete. 2024-03-24T09:31:25.298 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:31:25.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 -- 172.21.15.116:0/3679783680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37241016a0 msgr2=0x7f3724101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 --2- 172.21.15.116:0/3679783680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37241016a0 0x7f3724101b20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f370c009900 tx=0x7f370c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 -- 172.21.15.116:0/3679783680 shutdown_connections 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 --2- 172.21.15.116:0/3679783680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3724102060 0x7f372410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 --2- 172.21.15.116:0/3679783680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37241016a0 0x7f3724101b20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 --2- 172.21.15.116:0/3679783680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37241076b0 0x7f3724107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 -- 172.21.15.116:0/3679783680 >> 172.21.15.116:0/3679783680 conn(0x7f3724078b10 msgr2=0x7f3724078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 -- 172.21.15.116:0/3679783680 shutdown_connections 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.401+0000 7f3729eaf640 1 -- 172.21.15.116:0/3679783680 wait complete. 2024-03-24T09:31:25.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 Processor -- start 2024-03-24T09:31:25.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 -- start start 2024-03-24T09:31:25.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37241016a0 0x7f372419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3724102060 0x7f372419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 0x7f3724197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3724198080 con 0x7f37241076b0 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f37241981f0 con 0x7f3724102060 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3729eaf640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f37241983b0 con 0x7f37241016a0 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 0x7f3724197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 0x7f3724197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54668/0 (socket says 172.21.15.116:54668) 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3723fff640 1 -- 172.21.15.116:0/1718728272 learned_addr learned my addr 172.21.15.116:0/1718728272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f37237fe640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37241016a0 0x7f372419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.402+0000 7f3722ffd640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3724102060 0x7f372419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3723fff640 1 -- 172.21.15.116:0/1718728272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37241016a0 msgr2=0x7f372419d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3723fff640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37241016a0 0x7f372419d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3723fff640 1 -- 172.21.15.116:0/1718728272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3724102060 msgr2=0x7f372419ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3723fff640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3724102060 0x7f372419ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3723fff640 1 -- 172.21.15.116:0/1718728272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370c009560 con 0x7f37241076b0 2024-03-24T09:31:25.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3723fff640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 0x7f3724197a30 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f371c00e970 tx=0x7f371c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3720ff9640 1 -- 172.21.15.116:0/1718728272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c009720 con 0x7f37241076b0 2024-03-24T09:31:25.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3720ff9640 1 -- 172.21.15.116:0/1718728272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f371c0054d0 con 0x7f37241076b0 2024-03-24T09:31:25.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3720ff9640 1 -- 172.21.15.116:0/1718728272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c010430 con 0x7f37241076b0 2024-03-24T09:31:25.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3724198690 con 0x7f37241076b0 2024-03-24T09:31:25.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.403+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37241a4920 con 0x7f37241076b0 2024-03-24T09:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.405+0000 7f3720ff9640 1 -- 172.21.15.116:0/1718728272 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f371c004f30 con 0x7f37241076b0 2024-03-24T09:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.405+0000 7f372cb6a740 1 Processor -- start 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 -- start start 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3720ff9640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f37100a2230 0x7f37100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e9507d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e9593b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e95bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d85e8be840 con 0x55d85e9503f0 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d85e8bbd70 con 0x55d85e950d10 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f372cb6a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d85e8b7550 con 0x55d85e9598f0 2024-03-24T09:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3722ffd640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f37100a2230 0x7f37100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e9507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3720ff9640 1 -- 172.21.15.116:0/1718728272 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f371c014070 con 0x7f37241076b0 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e9507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54680/0 (socket says 172.21.15.116:54680) 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 -- 172.21.15.116:0/3060452549 learned_addr learned my addr 172.21.15.116:0/3060452549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3722ffd640 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e9593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3723fff640 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e95bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 -- 172.21.15.116:0/3060452549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 msgr2=0x55d85e95bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e95bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 -- 172.21.15.116:0/3060452549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 msgr2=0x55d85e9593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e9593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 -- 172.21.15.116:0/3060452549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d85e84cde0 con 0x55d85e9503f0 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3722ffd640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f37100a2230 0x7f37100a46f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f370c006010 tx=0x7f370c005e20 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f37237fe640 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e9507d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3714009fd0 tx=0x7f37140118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b323a4fc05bd69c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3719ffb640 1 -- 172.21.15.116:0/3060452549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714003c30 con 0x55d85e9503f0 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.406+0000 7f3719ffb640 1 -- 172.21.15.116:0/3060452549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3714003d90 con 0x55d85e9503f0 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f3719ffb640 1 -- 172.21.15.116:0/3060452549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371400f940 con 0x55d85e9503f0 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 -- 172.21.15.116:0/3060452549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 msgr2=0x55d85e9507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e9507d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3714009fd0 tx=0x7f37140118d0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 -- 172.21.15.116:0/3060452549 shutdown_connections 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e95bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e9593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3060452549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e9507d0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 -- 172.21.15.116:0/3060452549 >> 172.21.15.116:0/3060452549 conn(0x55d85e94ab20 msgr2=0x55d85e94af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 -- 172.21.15.116:0/3060452549 shutdown_connections 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 -- 172.21.15.116:0/3060452549 wait complete. 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.407+0000 7f372cb6a740 1 -- start start 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 Processor -- start 2024-03-24T09:31:25.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e9503f0 0x55d85e956910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e950d10 0x55d85e956e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e957390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d85e58a430 con 0x55d85e9598f0 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d85e546e00 con 0x55d85e9503f0 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d85e83c590 con 0x55d85e950d10 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f372cb6a740 10 client.0 _ll_register_callbacks cb 0x55d85e8abfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e957390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e957390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54682/0 (socket says 172.21.15.116:54682) 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f37237fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e9503f0 0x55d85e956910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 -- 172.21.15.116:0/2181147362 learned_addr learned my addr 172.21.15.116:0/2181147362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3722ffd640 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e950d10 0x55d85e956e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 -- 172.21.15.116:0/2181147362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e950d10 msgr2=0x55d85e956e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e950d10 0x55d85e956e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 -- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e9503f0 msgr2=0x55d85e956910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e9503f0 0x55d85e956910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.408+0000 7f3723fff640 1 -- 172.21.15.116:0/2181147362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3714009c00 con 0x55d85e9598f0 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3723fff640 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e957390 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f371c00c6a0 tx=0x7f371c09aa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0308bd6059a53e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3702ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3702ffd640 1 -- 172.21.15.116:0/2181147362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0aeea0 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3702ffd640 1 -- 172.21.15.116:0/2181147362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f371c09b550 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d85e5b67e0 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3702ffd640 1 -- 172.21.15.116:0/2181147362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f371c0120f0 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 10 client.18261 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d85e8b5bd0 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3702ffd640 1 -- 172.21.15.116:0/2181147362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f371c07d020 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f3702ffd640 1 -- 172.21.15.116:0/2181147362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f371c01f060 con 0x55d85e9598f0 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 10 client.18261 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 10 client.18261 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 20 client.18261 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e969990 0x55d85e96bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 4 client.18261 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d85e5f3720 con 0x55d85e969990 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.409+0000 7f37237fe640 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e969990 0x55d85e96bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.410+0000 7f37237fe640 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e969990 0x55d85e96bd80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d0c73497498220f0 server_cookie=8cdaae839e861889 in_seq=0 out_seq=0 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.410+0000 7f3702ffd640 10 client.18261 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.410+0000 7f3702ffd640 1 -- 172.21.15.116:0/2181147362 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d85e5f3720 con 0x55d85e969990 2024-03-24T09:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.410+0000 7f3702ffd640 10 client.18261 handle_command_reply: tid=0 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.410+0000 7f372cb6a740 1 client.18261 shutdown 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.410+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 msgr2=0x55d85e957390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e957390 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f371c00c6a0 tx=0x7f371c09aa50 comp rx=0 tx=0).stop 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 shutdown_connections 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e969990 0x55d85e96bd80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e957390 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e950d10 0x55d85e956e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f3702ffd640 0 client.18261 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:25.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2181147362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e9503f0 0x55d85e956910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 >> 172.21.15.116:0/2181147362 conn(0x55d85e94ab20 msgr2=0x55d85e94c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 shutdown_connections 2024-03-24T09:31:25.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 1 -- 172.21.15.116:0/2181147362 wait complete. 2024-03-24T09:31:25.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.411+0000 7f372cb6a740 20 client.18261 trim_cache size 0 max 16384 2024-03-24T09:31:25.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 Processor -- start 2024-03-24T09:31:25.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 -- start start 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9503f0 0x55d85e958740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e971640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e9617a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d85e966450 con 0x55d85e9598f0 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d85e957da0 con 0x55d85e950d10 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f372cb6a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d85e9593a0 con 0x55d85e9503f0 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e9617a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e9617a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54692/0 (socket says 172.21.15.116:54692) 2024-03-24T09:31:25.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 -- 172.21.15.116:0/2110631289 learned_addr learned my addr 172.21.15.116:0/2110631289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3722ffd640 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e971640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f37237fe640 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9503f0 0x55d85e958740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 -- 172.21.15.116:0/2110631289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9503f0 msgr2=0x55d85e958740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9503f0 0x55d85e958740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 -- 172.21.15.116:0/2110631289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 msgr2=0x55d85e971640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e971640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 -- 172.21.15.116:0/2110631289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d85e851c80 con 0x55d85e9598f0 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f37237fe640 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9503f0 0x55d85e958740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:25.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.424+0000 7f3723fff640 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e9617a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f371c0cbc60 tx=0x7f371c09aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bd85cde4417fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f37197fa640 1 -- 172.21.15.116:0/2110631289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0aeea0 con 0x55d85e9598f0 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f37197fa640 1 -- 172.21.15.116:0/2110631289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f371c07d070 con 0x55d85e9598f0 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f37197fa640 1 -- 172.21.15.116:0/2110631289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371c0b9380 con 0x55d85e9598f0 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 -- 172.21.15.116:0/2110631289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 msgr2=0x55d85e9617a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e9617a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f371c0cbc60 tx=0x7f371c09aac0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 -- 172.21.15.116:0/2110631289 shutdown_connections 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9598f0 0x55d85e9617a0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e971640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 --2- 172.21.15.116:0/2110631289 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9503f0 0x55d85e958740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 -- 172.21.15.116:0/2110631289 >> 172.21.15.116:0/2110631289 conn(0x55d85e94ab20 msgr2=0x55d85e958030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 -- 172.21.15.116:0/2110631289 shutdown_connections 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 -- 172.21.15.116:0/2110631289 wait complete. 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.425+0000 7f372cb6a740 1 -- start start 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 Processor -- start 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e95d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e95d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e9cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d85e95e060 con 0x55d85e9503f0 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d85e95e1d0 con 0x55d85e950d10 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55d85e9cf1b0 con 0x55d85e9598f0 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f372cb6a740 10 client.0 _ll_register_callbacks cb 0x7f371c09ac20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e95d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e95d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54702/0 (socket says 172.21.15.116:54702) 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 -- 172.21.15.116:0/3968774731 learned_addr learned my addr 172.21.15.116:0/3968774731 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f3722ffd640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e95d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f3723fff640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e9cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 -- 172.21.15.116:0/3968774731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 msgr2=0x55d85e9cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e9cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 -- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 msgr2=0x55d85e95d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e95d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 -- 172.21.15.116:0/3968774731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d85e8b5bd0 con 0x55d85e9503f0 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f3723fff640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e9cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f37237fe640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e95d3c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f371401c040 tx=0x7f3714009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97399fccb3707eec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.426+0000 7f3702ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f3702ffd640 1 -- 172.21.15.116:0/3968774731 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371402a070 con 0x55d85e9503f0 2024-03-24T09:31:25.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f3702ffd640 1 -- 172.21.15.116:0/3968774731 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3714004cd0 con 0x55d85e9503f0 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3714003d90 con 0x55d85e9503f0 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f3702ffd640 1 -- 172.21.15.116:0/3968774731 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3714004e90 con 0x55d85e9503f0 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 10 client.18267 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d85e877920 con 0x55d85e9503f0 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f3702ffd640 1 -- 172.21.15.116:0/3968774731 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3714003d90 con 0x55d85e9503f0 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f3702ffd640 1 -- 172.21.15.116:0/3968774731 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3714005040 con 0x55d85e9503f0 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 10 client.18267 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 10 client.18267 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:31:25.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 20 client.18267 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e9d09f0 0x55d85e9d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 4 client.18267 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d85e642890 con 0x55d85e9d09f0 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.427+0000 7f3722ffd640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e9d09f0 0x55d85e9d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3702ffd640 10 client.18267 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3722ffd640 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e9d09f0 0x55d85e9d2e10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9db87b5afe766ba server_cookie=1f8fc62e31f976f8 in_seq=0 out_seq=0 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3702ffd640 1 -- 172.21.15.116:0/3968774731 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d85e642890 con 0x55d85e9d09f0 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3702ffd640 10 client.18267 handle_command_reply: tid=0 2024-03-24T09:31:25.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 client.18267 shutdown 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 msgr2=0x55d85e95d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e95d3c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f371401c040 tx=0x7f3714009f60 comp rx=0 tx=0).stop 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 shutdown_connections 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55d85e9d09f0 0x55d85e9d2e10 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55d85e9598f0 0x55d85e9cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d85e950d10 0x55d85e95d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 --2- 172.21.15.116:0/3968774731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d85e9503f0 0x55d85e95d3c0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3702ffd640 0 client.18267 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:25.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 >> 172.21.15.116:0/3968774731 conn(0x55d85e94ab20 msgr2=0x55d85e949cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 shutdown_connections 2024-03-24T09:31:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 1 -- 172.21.15.116:0/3968774731 wait complete. 2024-03-24T09:31:25.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f372cb6a740 20 client.18267 trim_cache size 0 max 16384 2024-03-24T09:31:25.441 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:25.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f37100a2230 msgr2=0x7f37100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.428+0000 7f3729eaf640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f37100a2230 0x7f37100a46f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f370c006010 tx=0x7f370c005e20 comp rx=0 tx=0).stop 2024-03-24T09:31:25.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 msgr2=0x7f3724197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 0x7f3724197a30 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f371c00e970 tx=0x7f371c00ed70 comp rx=0 tx=0).stop 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 shutdown_connections 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37241076b0 0x7f3724197a30 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3724102060 0x7f372419ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f37100a2230 0x7f37100a46f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 --2- 172.21.15.116:0/1718728272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f37241016a0 0x7f372419d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 >> 172.21.15.116:0/1718728272 conn(0x7f3724078b10 msgr2=0x7f37241040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 shutdown_connections 2024-03-24T09:31:25.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.429+0000 7f3729eaf640 1 -- 172.21.15.116:0/1718728272 wait complete. 2024-03-24T09:31:25.444 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:25 smithi116 ceph-mon[32207]: pgmap v288: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:31:25.444 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:31:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:25 smithi143 ceph-mon[35965]: pgmap v288: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:31:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 -- 172.21.15.116:0/420723658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7810bd50 msgr2=0x7f6b7810e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/420723658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7810bd50 0x7f6b7810e1c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009ca0 tx=0x7f6b7002c180 comp rx=0 tx=0).stop 2024-03-24T09:31:25.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 -- 172.21.15.116:0/420723658 shutdown_connections 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/420723658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7810bd50 0x7f6b7810e1c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/420723658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b781032a0 0x7f6b7810b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/420723658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b78102980 0x7f6b78102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 -- 172.21.15.116:0/420723658 >> 172.21.15.116:0/420723658 conn(0x7f6b78077770 msgr2=0x7f6b78077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 -- 172.21.15.116:0/420723658 shutdown_connections 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.547+0000 7f6b7e33a640 1 -- 172.21.15.116:0/420723658 wait complete. 2024-03-24T09:31:25.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 Processor -- start 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 -- start start 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b78102980 0x7f6b7819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 0x7f6b7819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b7810bd50 0x7f6b78197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b78198100 con 0x7f6b781032a0 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6b78198270 con 0x7f6b78102980 2024-03-24T09:31:25.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7e33a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6b78198430 con 0x7f6b7810bd50 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 0x7f6b7819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b78102980 0x7f6b7819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 0x7f6b7819ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54724/0 (socket says 172.21.15.116:54724) 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 -- 172.21.15.116:0/4061636773 learned_addr learned my addr 172.21.15.116:0/4061636773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b7810bd50 0x7f6b78197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 -- 172.21.15.116:0/4061636773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b7810bd50 msgr2=0x7f6b78197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b7810bd50 0x7f6b78197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 -- 172.21.15.116:0/4061636773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b78102980 msgr2=0x7f6b7819d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b78102980 0x7f6b7819d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.548+0000 7f6b777fe640 1 -- 172.21.15.116:0/4061636773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b70009900 con 0x7f6b781032a0 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.549+0000 7f6b777fe640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 0x7f6b7819ddf0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00e9f0 tx=0x7f6b6c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.549+0000 7f6b757fa640 1 -- 172.21.15.116:0/4061636773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c009720 con 0x7f6b781032a0 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.549+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b78198710 con 0x7f6b781032a0 2024-03-24T09:31:25.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.549+0000 7f6b757fa640 1 -- 172.21.15.116:0/4061636773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b6c00cd80 con 0x7f6b781032a0 2024-03-24T09:31:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.549+0000 7f6b757fa640 1 -- 172.21.15.116:0/4061636773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c010430 con 0x7f6b781032a0 2024-03-24T09:31:25.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.549+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b781a4950 con 0x7f6b781032a0 2024-03-24T09:31:25.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.551+0000 7f6b757fa640 1 -- 172.21.15.116:0/4061636773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6b6c005020 con 0x7f6b781032a0 2024-03-24T09:31:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.551+0000 7f6b80fec740 1 Processor -- start 2024-03-24T09:31:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.551+0000 7f6b757fa640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b600a2230 0x7f6b600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.551+0000 7f6b80fec740 1 -- start start 2024-03-24T09:31:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.551+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b600a2230 0x7f6b600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b757fa640 1 -- 172.21.15.116:0/4061636773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f6b6c014070 con 0x7f6b781032a0 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd501f87d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd502013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50203ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b80fec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd50166840 con 0x55fd501f8d10 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b80fec740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fd50163d70 con 0x55fd502018f0 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b80fec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fd5015f550 con 0x55fd501f83f0 2024-03-24T09:31:25.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd502013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd502013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54740/0 (socket says 172.21.15.116:54740) 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd501f87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 -- 172.21.15.116:0/3125528767 learned_addr learned my addr 172.21.15.116:0/3125528767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50203ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 -- 172.21.15.116:0/3125528767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 msgr2=0x55fd501f87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd501f87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 -- 172.21.15.116:0/3125528767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 msgr2=0x55fd50203ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50203ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 -- 172.21.15.116:0/3125528767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd500f4de0 con 0x55fd501f8d10 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b77fff640 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd501f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50203ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b600a2230 0x7f6b600a46f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6b70004af0 tx=0x7f6b700324c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b777fe640 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd502013b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c097740 tx=0x7f6b6c09a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267a2fdba5cfd2b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b5e7fc640 1 -- 172.21.15.116:0/3125528767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c09ad70 con 0x55fd501f8d10 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b5e7fc640 1 -- 172.21.15.116:0/3125528767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b6c09eec0 con 0x55fd501f8d10 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.552+0000 7f6b5e7fc640 1 -- 172.21.15.116:0/3125528767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c09b230 con 0x55fd501f8d10 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 -- 172.21.15.116:0/3125528767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 msgr2=0x55fd502013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd502013b0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c097740 tx=0x7f6b6c09a790 comp rx=0 tx=0).stop 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 -- 172.21.15.116:0/3125528767 shutdown_connections 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50203ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd502013b0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 --2- 172.21.15.116:0/3125528767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd501f87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 -- 172.21.15.116:0/3125528767 >> 172.21.15.116:0/3125528767 conn(0x55fd501f2b20 msgr2=0x55fd501f2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 -- 172.21.15.116:0/3125528767 shutdown_connections 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 -- 172.21.15.116:0/3125528767 wait complete. 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.553+0000 7f6b80fec740 1 -- start start 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 Processor -- start 2024-03-24T09:31:25.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501ff470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd4fe32430 con 0x55fd502018f0 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fd4fdeee00 con 0x55fd501f83f0 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fd500e4590 con 0x55fd501f8d10 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b80fec740 10 client.0 _ll_register_callbacks cb 0x55fd50153fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501ff470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501ff470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54746/0 (socket says 172.21.15.116:54746) 2024-03-24T09:31:25.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1042062152 learned_addr learned my addr 172.21.15.116:0/1042062152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b77fff640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b777fe640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 msgr2=0x55fd501fef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1042062152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 msgr2=0x55fd501fe9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fe9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1042062152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b781a4950 con 0x55fd502018f0 2024-03-24T09:31:25.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501ff470 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6b70036000 tx=0x7f6b7002c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ba52680e08530e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.554+0000 7f6b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b577fe640 1 -- 172.21.15.116:0/1042062152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70046030 con 0x55fd502018f0 2024-03-24T09:31:25.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b577fe640 1 -- 172.21.15.116:0/1042062152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f6b70047070 con 0x55fd502018f0 2024-03-24T09:31:25.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fd4fe5e7e0 con 0x55fd502018f0 2024-03-24T09:31:25.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b577fe640 1 -- 172.21.15.116:0/1042062152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6b70032680 con 0x55fd502018f0 2024-03-24T09:31:25.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 10 client.18279 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fd5015dbd0 con 0x55fd502018f0 2024-03-24T09:31:25.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b577fe640 1 -- 172.21.15.116:0/1042062152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fd4fe5e7e0 con 0x55fd502018f0 2024-03-24T09:31:25.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b577fe640 1 -- 172.21.15.116:0/1042062152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f6b70004410 con 0x55fd502018f0 2024-03-24T09:31:25.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 10 client.18279 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 10 client.18279 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:31:25.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 20 client.18279 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd50211aa0 0x55fd50213ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 4 client.18279 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:25.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd4fe9b720 con 0x55fd50211aa0 2024-03-24T09:31:25.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.555+0000 7f6b77fff640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd50211aa0 0x55fd50213ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b77fff640 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd50211aa0 0x55fd50213ec0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=82b907fb789452d5 server_cookie=b13f9ae848aa83b6 in_seq=0 out_seq=0 2024-03-24T09:31:25.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b577fe640 10 client.18279 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:25.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b577fe640 1 -- 172.21.15.116:0/1042062152 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fd4fe9b720 con 0x55fd50211aa0 2024-03-24T09:31:25.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b577fe640 10 client.18279 handle_command_reply: tid=0 2024-03-24T09:31:25.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 client.18279 shutdown 2024-03-24T09:31:25.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 msgr2=0x55fd501ff470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501ff470 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6b70036000 tx=0x7f6b7002c180 comp rx=0 tx=0).stop 2024-03-24T09:31:25.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 shutdown_connections 2024-03-24T09:31:25.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501ff470 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd50211aa0 0x55fd50213ec0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1042062152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b577fe640 0 client.18279 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.556+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 >> 172.21.15.116:0/1042062152 conn(0x55fd501f2b20 msgr2=0x55fd501f4870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.557+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 shutdown_connections 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.557+0000 7f6b80fec740 1 -- 172.21.15.116:0/1042062152 wait complete. 2024-03-24T09:31:25.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.557+0000 7f6b80fec740 20 client.18279 trim_cache size 0 max 16384 2024-03-24T09:31:25.576 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:25 smithi117 ceph-mon[35788]: pgmap v288: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-24T09:31:25.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.569+0000 7f6b80fec740 1 Processor -- start 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.569+0000 7f6b80fec740 1 -- start start 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd50209b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd5020d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50219770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b80fec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd5020e540 con 0x55fd501f8d10 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b80fec740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fd501f3d50 con 0x55fd502018f0 2024-03-24T09:31:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b80fec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fd501f7510 con 0x55fd501f83f0 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd5020d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd5020d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54756/0 (socket says 172.21.15.116:54756) 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 -- 172.21.15.116:0/759359604 learned_addr learned my addr 172.21.15.116:0/759359604 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50219770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b77fff640 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd50209b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 -- 172.21.15.116:0/759359604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 msgr2=0x55fd50209b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd50209b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 -- 172.21.15.116:0/759359604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 msgr2=0x55fd50219770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50219770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 -- 172.21.15.116:0/759359604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd500f9c80 con 0x55fd501f8d10 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b77fff640 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd50209b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b777fe640 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd5020d180 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c09aa70 tx=0x7f6b6c09a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52be81cf6783657 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b5dffb640 1 -- 172.21.15.116:0/759359604 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0ab070 con 0x55fd501f8d10 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b5dffb640 1 -- 172.21.15.116:0/759359604 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b6c0b92e0 con 0x55fd501f8d10 2024-03-24T09:31:25.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.570+0000 7f6b5dffb640 1 -- 172.21.15.116:0/759359604 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c09afd0 con 0x55fd501f8d10 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 -- 172.21.15.116:0/759359604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 msgr2=0x55fd5020d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd5020d180 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c09aa70 tx=0x7f6b6c09a800 comp rx=0 tx=0).stop 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 -- 172.21.15.116:0/759359604 shutdown_connections 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd502018f0 0x55fd50219770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd501f8d10 0x55fd5020d180 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 --2- 172.21.15.116:0/759359604 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f83f0 0x55fd50209b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 -- 172.21.15.116:0/759359604 >> 172.21.15.116:0/759359604 conn(0x55fd501f2b20 msgr2=0x55fd501f4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 -- 172.21.15.116:0/759359604 shutdown_connections 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 -- 172.21.15.116:0/759359604 wait complete. 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 -- start start 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.571+0000 7f6b80fec740 1 Processor -- start 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501fac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fd501fb380 con 0x55fd502018f0 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55fd501fb4f0 con 0x55fd501f83f0 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55fd501fb6b0 con 0x55fd501f8d10 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 10 client.0 _ll_register_callbacks cb 0x7f6b70044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501fac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501fac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54772/0 (socket says 172.21.15.116:54772) 2024-03-24T09:31:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1473731319 learned_addr learned my addr 172.21.15.116:0/1473731319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b77fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fc040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:41118/0 (socket says 172.21.15.116:41118) 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b777fe640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 msgr2=0x55fd501fc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1473731319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 msgr2=0x55fd501fc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 -- 172.21.15.116:0/1473731319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c0b7040 con 0x55fd502018f0 2024-03-24T09:31:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b77fff640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b7c8d7640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501fac80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6b7002c790 tx=0x7f6b70009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77e9fa6764fb6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b577fe640 1 -- 172.21.15.116:0/1473731319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7004cc40 con 0x55fd502018f0 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b577fe640 1 -- 172.21.15.116:0/1473731319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f6b7004cda0 con 0x55fd502018f0 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fd4feea890 con 0x55fd502018f0 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b577fe640 1 -- 172.21.15.116:0/1473731319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6b700348d0 con 0x55fd502018f0 2024-03-24T09:31:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 10 client.18285 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.572+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fd5011f920 con 0x55fd502018f0 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b577fe640 1 -- 172.21.15.116:0/1473731319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55fd4feea890 con 0x55fd502018f0 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b577fe640 1 -- 172.21.15.116:0/1473731319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f6b70003f00 con 0x55fd502018f0 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 10 client.18285 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 10 client.18285 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 20 client.18285 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd5027cd40 0x55fd5027f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 4 client.18285 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd4fe0a700 con 0x55fd5027cd40 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b77fff640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd5027cd40 0x55fd5027f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b577fe640 10 client.18285 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b77fff640 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd5027cd40 0x55fd5027f160 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1f6df0ca5839908 server_cookie=effec4ed3e6dfbd1 in_seq=0 out_seq=0 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b577fe640 1 -- 172.21.15.116:0/1473731319 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd4fe0a700 con 0x55fd5027cd40 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b577fe640 10 client.18285 handle_command_reply: tid=0 2024-03-24T09:31:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.573+0000 7f6b80fec740 1 client.18285 shutdown 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 msgr2=0x55fd501fac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501fac80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6b7002c790 tx=0x7f6b70009580 comp rx=0 tx=0).stop 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 shutdown_connections 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fd502018f0 0x55fd501fac80 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55fd501f8d10 0x55fd501fc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55fd5027cd40 0x55fd5027f160 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 --2- 172.21.15.116:0/1473731319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55fd501f83f0 0x55fd501fc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b577fe640 0 client.18285 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 >> 172.21.15.116:0/1473731319 conn(0x55fd501f2b20 msgr2=0x55fd501f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 shutdown_connections 2024-03-24T09:31:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 1 -- 172.21.15.116:0/1473731319 wait complete. 2024-03-24T09:31:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b80fec740 20 client.18285 trim_cache size 0 max 16384 2024-03-24T09:31:25.594 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b600a2230 msgr2=0x7f6b600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b600a2230 0x7f6b600a46f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6b70004af0 tx=0x7f6b700324c0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 msgr2=0x7f6b7819ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 0x7f6b7819ddf0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00e9f0 tx=0x7f6b6c00edf0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 shutdown_connections 2024-03-24T09:31:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b600a2230 0x7f6b600a46f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b7810bd50 0x7f6b78197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b781032a0 0x7f6b7819ddf0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 --2- 172.21.15.116:0/4061636773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b78102980 0x7f6b7819d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 >> 172.21.15.116:0/4061636773 conn(0x7f6b78077770 msgr2=0x7f6b780fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 shutdown_connections 2024-03-24T09:31:25.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.574+0000 7f6b7e33a640 1 -- 172.21.15.116:0/4061636773 wait complete. 2024-03-24T09:31:25.597 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:31:25.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 -- 172.21.15.116:0/135390614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae81059d0 msgr2=0x7faae810d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 --2- 172.21.15.116:0/135390614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae81059d0 0x7faae810d510 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faae0009cd0 tx=0x7faae002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 -- 172.21.15.116:0/135390614 shutdown_connections 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 --2- 172.21.15.116:0/135390614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae81059d0 0x7faae810d510 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 --2- 172.21.15.116:0/135390614 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae8101d00 0x7faae81052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 --2- 172.21.15.116:0/135390614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae8101350 0x7faae8101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 -- 172.21.15.116:0/135390614 >> 172.21.15.116:0/135390614 conn(0x7faae8075950 msgr2=0x7faae8075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 -- 172.21.15.116:0/135390614 shutdown_connections 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 -- 172.21.15.116:0/135390614 wait complete. 2024-03-24T09:31:25.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.700+0000 7faaef055640 1 Processor -- start 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 -- start start 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 0x7faae8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae81059d0 0x7faae8199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae81995d0 0x7faae81a3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faae81a4220 con 0x7faae81995d0 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faae81a4390 con 0x7faae8101350 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7faae81a4550 con 0x7faae81059d0 2024-03-24T09:31:25.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaecdf1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 0x7faae8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae81995d0 0x7faae81a3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faae7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae81059d0 0x7faae8199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaecdf1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 0x7faae8198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:41144/0 (socket says 172.21.15.116:41144) 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaecdf1640 1 -- 172.21.15.116:0/4203914326 learned_addr learned my addr 172.21.15.116:0/4203914326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 -- 172.21.15.116:0/4203914326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae81059d0 msgr2=0x7faae8199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae81059d0 0x7faae8199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 -- 172.21.15.116:0/4203914326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 msgr2=0x7faae8198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 0x7faae8198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 -- 172.21.15.116:0/4203914326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad8009490 con 0x7faae81995d0 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaecdf1640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 0x7faae8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae81995d0 0x7faae81a3b80 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7faae002c630 tx=0x7faae0035d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faae5ffb640 1 -- 172.21.15.116:0/4203914326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae00329e0 con 0x7faae81995d0 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.701+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae0009900 con 0x7faae81995d0 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.702+0000 7faae5ffb640 1 -- 172.21.15.116:0/4203914326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faae0003e20 con 0x7faae81995d0 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.702+0000 7faae5ffb640 1 -- 172.21.15.116:0/4203914326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae00046f0 con 0x7faae81995d0 2024-03-24T09:31:25.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.702+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae81a4b10 con 0x7faae81995d0 2024-03-24T09:31:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faae5ffb640 1 -- 172.21.15.116:0/4203914326 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7faae003b040 con 0x7faae81995d0 2024-03-24T09:31:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 Processor -- start 2024-03-24T09:31:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 -- start start 2024-03-24T09:31:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faae5ffb640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faad00a2230 0x7faad00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b73f0 0x5617e59b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59c03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59c08f0 0x5617e59c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617e5925840 con 0x5617e59b7d10 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617e5922d70 con 0x5617e59c08f0 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaf1c34740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5617e591e550 con 0x5617e59b73f0 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faae7fff640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faad00a2230 0x7faad00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaed5f2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59c08f0 0x5617e59c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaecdf1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b73f0 0x5617e59b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaecdf1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b73f0 0x5617e59b77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:60192/0 (socket says 172.21.15.116:60192) 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.704+0000 7faaecdf1640 1 -- 172.21.15.116:0/4290775798 learned_addr learned my addr 172.21.15.116:0/4290775798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faae7fff640 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59c03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faae5ffb640 1 -- 172.21.15.116:0/4203914326 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7faae00f0cc0 con 0x7faae81995d0 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaed5f2640 1 -- 172.21.15.116:0/4290775798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b73f0 msgr2=0x5617e59b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b73f0 0x5617e59b77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaed5f2640 1 -- 172.21.15.116:0/4290775798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 msgr2=0x5617e59c03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59c03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaed5f2640 1 -- 172.21.15.116:0/4290775798 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617e58b3de0 con 0x5617e59c08f0 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59c08f0 0x5617e59c2ce0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faae00ee110 tx=0x7faae0032b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ffc20b8c3a082f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faae7fff640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faad00a2230 0x7faad00a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5617e59b6ec0 tx=0x7faadc0098c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaceffd640 1 -- 172.21.15.116:0/4290775798 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae00bcef0 con 0x5617e59c08f0 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaceffd640 1 -- 172.21.15.116:0/4290775798 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faae00bcd10 con 0x5617e59c08f0 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.705+0000 7faaceffd640 1 -- 172.21.15.116:0/4290775798 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae00bb500 con 0x5617e59c08f0 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 -- 172.21.15.116:0/4290775798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59c08f0 msgr2=0x5617e59c2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59c08f0 0x5617e59c2ce0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faae00ee110 tx=0x7faae0032b40 comp rx=0 tx=0).stop 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 -- 172.21.15.116:0/4290775798 shutdown_connections 2024-03-24T09:31:25.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59c08f0 0x5617e59c2ce0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59c03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4290775798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b73f0 0x5617e59b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 -- 172.21.15.116:0/4290775798 >> 172.21.15.116:0/4290775798 conn(0x5617e59b1b20 msgr2=0x5617e59b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 -- 172.21.15.116:0/4290775798 shutdown_connections 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 -- 172.21.15.116:0/4290775798 wait complete. 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 -- start start 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.706+0000 7faaf1c34740 1 Processor -- start 2024-03-24T09:31:25.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59bd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59bde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617e55f1430 con 0x5617e59b7d10 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617e55ade00 con 0x5617e59b73f0 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5617e58a3590 con 0x5617e59c08f0 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 10 client.0 _ll_register_callbacks cb 0x5617e5912fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faae7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59bde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59bd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faae7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59bde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54804/0 (socket says 172.21.15.116:54804) 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faae7fff640 1 -- 172.21.15.116:0/4252792810 learned_addr learned my addr 172.21.15.116:0/4252792810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaed5f2640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 -- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 msgr2=0x5617e59be3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 -- 172.21.15.116:0/4252792810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 msgr2=0x5617e59bde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59bde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 -- 172.21.15.116:0/4252792810 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae81a4b10 con 0x5617e59b73f0 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faae7fff640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59bde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaecdf1640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59bd930 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faad8009fd0 tx=0x7faad800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a29a3ff4ef534e27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faac7fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faac7fff640 1 -- 172.21.15.116:0/4252792810 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8017070 con 0x5617e59b73f0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5617e561d7e0 con 0x5617e59b73f0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.707+0000 7faac7fff640 1 -- 172.21.15.116:0/4252792810 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7faad80120d0 con 0x5617e59b73f0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faac7fff640 1 -- 172.21.15.116:0/4252792810 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faad8012290 con 0x5617e59b73f0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 10 client.28112 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5617e591cbd0 con 0x5617e59b73f0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faac7fff640 1 -- 172.21.15.116:0/4252792810 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5617e561d7e0 con 0x5617e59b73f0 2024-03-24T09:31:25.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faac7fff640 1 -- 172.21.15.116:0/4252792810 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7faad80105f0 con 0x5617e59b73f0 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 10 client.28112 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 10 client.28112 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 20 client.28112 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e59d0790 0x5617e59d2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 4 client.28112 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617e565a720 con 0x5617e59d0790 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faae7fff640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e59d0790 0x5617e59d2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faac7fff640 10 client.28112 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.708+0000 7faae7fff640 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e59d0790 0x5617e59d2bd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a1b1707e89e1cbe server_cookie=ee611436156b4043 in_seq=0 out_seq=0 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faac7fff640 1 -- 172.21.15.116:0/4252792810 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617e565a720 con 0x5617e59d0790 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faac7fff640 10 client.28112 handle_command_reply: tid=0 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 client.28112 shutdown 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 msgr2=0x5617e59bd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59bd930 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faad8009fd0 tx=0x7faad800f790 comp rx=0 tx=0).stop 2024-03-24T09:31:25.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 shutdown_connections 2024-03-24T09:31:25.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e59d0790 0x5617e59d2bd0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59bde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 --2- 172.21.15.116:0/4252792810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59bd930 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 >> 172.21.15.116:0/4252792810 conn(0x5617e59b1b20 msgr2=0x5617e59b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faac7fff640 0 client.28112 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:25.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 shutdown_connections 2024-03-24T09:31:25.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 1 -- 172.21.15.116:0/4252792810 wait complete. 2024-03-24T09:31:25.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.709+0000 7faaf1c34740 20 client.28112 trim_cache size 0 max 16384 2024-03-24T09:31:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.722+0000 7faaf1c34740 1 Processor -- start 2024-03-24T09:31:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.722+0000 7faaf1c34740 1 -- start start 2024-03-24T09:31:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59d84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaf1c34740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617e59cc480 con 0x5617e59b7d10 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaf1c34740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617e565a750 con 0x5617e59b73f0 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaf1c34740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5617e59cef90 con 0x5617e59c08f0 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59d84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaecdf1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59d84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54820/0 (socket says 172.21.15.116:54820) 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 -- 172.21.15.116:0/347795203 learned_addr learned my addr 172.21.15.116:0/347795203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaed5f2640 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 -- 172.21.15.116:0/347795203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 msgr2=0x5617e59be1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 -- 172.21.15.116:0/347795203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 msgr2=0x5617e59cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 -- 172.21.15.116:0/347795203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617e58b8c80 con 0x5617e59b7d10 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faaecdf1640 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faae7fff640 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59d84e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x5617e59bd190 tx=0x7faadc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b18e8e9747e0a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faace7fc640 1 -- 172.21.15.116:0/347795203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc023040 con 0x5617e59b7d10 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faace7fc640 1 -- 172.21.15.116:0/347795203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faadc00fe20 con 0x5617e59b7d10 2024-03-24T09:31:25.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.723+0000 7faace7fc640 1 -- 172.21.15.116:0/347795203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc004e10 con 0x5617e59b7d10 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 -- 172.21.15.116:0/347795203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 msgr2=0x5617e59d84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59d84e0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x5617e59bd190 tx=0x7faadc005f90 comp rx=0 tx=0).stop 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 -- 172.21.15.116:0/347795203 shutdown_connections 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59c08f0 0x5617e59be1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59b7d10 0x5617e59d84e0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 --2- 172.21.15.116:0/347795203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 -- 172.21.15.116:0/347795203 >> 172.21.15.116:0/347795203 conn(0x5617e59b1b20 msgr2=0x5617e59b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 -- 172.21.15.116:0/347795203 shutdown_connections 2024-03-24T09:31:25.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 -- 172.21.15.116:0/347795203 wait complete. 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 -- start start 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 Processor -- start 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.724+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b7d10 0x5617e59c4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 0x5617e59c4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617e5a35b10 con 0x5617e59c08f0 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617e5a35c50 con 0x5617e59b73f0 2024-03-24T09:31:25.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5617e5a35d90 con 0x5617e59b7d10 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 10 client.0 _ll_register_callbacks cb 0x7faad8012a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 0x5617e59c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 0x5617e59c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54834/0 (socket says 172.21.15.116:54834) 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 -- 172.21.15.116:0/1380134871 learned_addr learned my addr 172.21.15.116:0/1380134871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaecdf1640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faae7fff640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b7d10 0x5617e59c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 -- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b7d10 msgr2=0x5617e59c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b7d10 0x5617e59c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 -- 172.21.15.116:0/1380134871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 msgr2=0x5617e59c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 -- 172.21.15.116:0/1380134871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617e591cbd0 con 0x5617e59c08f0 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaed5f2640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 0x5617e59c4940 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7faae00098d0 tx=0x7faae0035ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d963c6bae76b9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faac7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faac7fff640 1 -- 172.21.15.116:0/1380134871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae00bdab0 con 0x5617e59c08f0 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faac7fff640 1 -- 172.21.15.116:0/1380134871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7faae00c1500 con 0x5617e59c08f0 2024-03-24T09:31:25.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faae00bcd10 con 0x5617e59c08f0 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faac7fff640 1 -- 172.21.15.116:0/1380134871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faae00c5b50 con 0x5617e59c08f0 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 10 client.18303 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.725+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7faadc019040 con 0x5617e59c08f0 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faac7fff640 1 -- 172.21.15.116:0/1380134871 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faae00bcd10 con 0x5617e59c08f0 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faac7fff640 1 -- 172.21.15.116:0/1380134871 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7faae00bd610 con 0x5617e59c08f0 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 10 client.18303 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 10 client.18303 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 20 client.18303 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e5a37230 0x5617e5a39650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 4 client.18303 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617e56a9890 con 0x5617e5a37230 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaecdf1640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e5a37230 0x5617e5a39650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faac7fff640 10 client.18303 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaecdf1640 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e5a37230 0x5617e5a39650 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82736d6046580f1a server_cookie=c4898048ac67888b in_seq=0 out_seq=0 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faac7fff640 1 -- 172.21.15.116:0/1380134871 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617e56a9890 con 0x5617e5a37230 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faac7fff640 10 client.18303 handle_command_reply: tid=0 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 1 client.18303 shutdown 2024-03-24T09:31:25.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 msgr2=0x5617e59c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.726+0000 7faaf1c34740 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 0x5617e59c4940 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7faae00098d0 tx=0x7faae0035ca0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 shutdown_connections 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x5617e5a37230 0x5617e5a39650 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5617e59c08f0 0x5617e59c4940 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faac7fff640 0 client.18303 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5617e59b7d10 0x5617e59c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 --2- 172.21.15.116:0/1380134871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e59b73f0 0x5617e59c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 >> 172.21.15.116:0/1380134871 conn(0x5617e59b1b20 msgr2=0x5617e59b2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 shutdown_connections 2024-03-24T09:31:25.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 1 -- 172.21.15.116:0/1380134871 wait complete. 2024-03-24T09:31:25.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaf1c34740 20 client.18303 trim_cache size 0 max 16384 2024-03-24T09:31:25.742 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:25.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faad00a2230 msgr2=0x7faad00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faad00a2230 0x7faad00a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x5617e59b6ec0 tx=0x7faadc0098c0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae81995d0 msgr2=0x7faae81a3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae81995d0 0x7faae81a3b80 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7faae002c630 tx=0x7faae0035d20 comp rx=0 tx=0).stop 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 shutdown_connections 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faae81995d0 0x7faae81a3b80 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faad00a2230 0x7faad00a46f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faae81059d0 0x7faae8199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 --2- 172.21.15.116:0/4203914326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faae8101350 0x7faae8198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 >> 172.21.15.116:0/4203914326 conn(0x7faae8075950 msgr2=0x7faae80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 shutdown_connections 2024-03-24T09:31:25.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.727+0000 7faaef055640 1 -- 172.21.15.116:0/4203914326 wait complete. 2024-03-24T09:31:25.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:31:25.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 -- 172.21.15.116:0/1021400185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba0107690 msgr2=0x7f7ba0107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 --2- 172.21.15.116:0/1021400185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba0107690 0x7f7ba0107a70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009cd0 tx=0x7f7b9002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 -- 172.21.15.116:0/1021400185 shutdown_connections 2024-03-24T09:31:25.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 --2- 172.21.15.116:0/1021400185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba00692b0 0x7f7ba010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 --2- 172.21.15.116:0/1021400185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba00688f0 0x7f7ba0068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 --2- 172.21.15.116:0/1021400185 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba0107690 0x7f7ba0107a70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 -- 172.21.15.116:0/1021400185 >> 172.21.15.116:0/1021400185 conn(0x7f7ba0077710 msgr2=0x7f7ba0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 -- 172.21.15.116:0/1021400185 shutdown_connections 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.848+0000 7f7ba5902640 1 -- 172.21.15.116:0/1021400185 wait complete. 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 Processor -- start 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 -- start start 2024-03-24T09:31:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba00688f0 0x7f7ba019d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba00692b0 0x7f7ba019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 0x7f7ba0197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01980e0 con 0x7f7ba0107690 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0198250 con 0x7f7ba00692b0 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7ba5902640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0198410 con 0x7f7ba00688f0 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7b9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 0x7f7ba0197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.849+0000 7f7b9f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 0x7f7ba0197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54846/0 (socket says 172.21.15.116:54846) 2024-03-24T09:31:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 -- 172.21.15.116:0/2836693450 learned_addr learned my addr 172.21.15.116:0/2836693450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9effd640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba00688f0 0x7f7ba019d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba00692b0 0x7f7ba019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 -- 172.21.15.116:0/2836693450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba00688f0 msgr2=0x7f7ba019d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba00688f0 0x7f7ba019d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 -- 172.21.15.116:0/2836693450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba00692b0 msgr2=0x7f7ba019de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba00692b0 0x7f7ba019de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 -- 172.21.15.116:0/2836693450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b90009900 con 0x7f7ba0107690 2024-03-24T09:31:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7b9f7fe640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 0x7f7ba0197a90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800e970 tx=0x7f7b9800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7ba4900640 1 -- 172.21.15.116:0/2836693450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98009720 con 0x7f7ba0107690 2024-03-24T09:31:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba01986f0 con 0x7f7ba0107690 2024-03-24T09:31:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7ba4900640 1 -- 172.21.15.116:0/2836693450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b980054d0 con 0x7f7ba0107690 2024-03-24T09:31:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7ba4900640 1 -- 172.21.15.116:0/2836693450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98010430 con 0x7f7ba0107690 2024-03-24T09:31:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.850+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba01a4980 con 0x7f7ba0107690 2024-03-24T09:31:25.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.852+0000 7f7ba4900640 1 -- 172.21.15.116:0/2836693450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f7b98004f30 con 0x7f7ba0107690 2024-03-24T09:31:25.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.852+0000 7f7ba85bf740 1 Processor -- start 2024-03-24T09:31:25.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 -- start start 2024-03-24T09:31:25.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x561754290be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542997c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x56175429c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5617541fa840 con 0x561754299d00 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617541f7d70 con 0x561754290800 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5617541f3550 con 0x561754291120 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x56175429c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:60236/0 (socket says 172.21.15.116:60236) 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 -- 172.21.15.116:0/3284725339 learned_addr learned my addr 172.21.15.116:0/3284725339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x561754290be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba4900640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7b880a2230 0x7f7b880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 -- 172.21.15.116:0/3284725339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 msgr2=0x561754290be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x561754290be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 -- 172.21.15.116:0/3284725339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 msgr2=0x56175429c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9f7fe640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7b880a2230 0x7f7b880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x56175429c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 -- 172.21.15.116:0/3284725339 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561754188de0 con 0x561754291120 2024-03-24T09:31:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x56175429c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7ba4900640 1 -- 172.21.15.116:0/2836693450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f7b98014070 con 0x7f7ba0107690 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.853+0000 7f7b9effd640 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542997c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009ca0 tx=0x7f7b90005f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3697ff290f827826 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7b85ffb640 1 -- 172.21.15.116:0/3284725339 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90032c90 con 0x561754291120 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7b9f7fe640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7b880a2230 0x7f7b880a46f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7b980cc100 tx=0x7f7b980c5460 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7b85ffb640 1 -- 172.21.15.116:0/3284725339 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b90032df0 con 0x561754291120 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7b85ffb640 1 -- 172.21.15.116:0/3284725339 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90032990 con 0x561754291120 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 -- 172.21.15.116:0/3284725339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 msgr2=0x5617542997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542997c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7b90009ca0 tx=0x7f7b90005f80 comp rx=0 tx=0).stop 2024-03-24T09:31:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 -- 172.21.15.116:0/3284725339 shutdown_connections 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x56175429c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542997c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/3284725339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x561754290be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 -- 172.21.15.116:0/3284725339 >> 172.21.15.116:0/3284725339 conn(0x56175428af30 msgr2=0x56175428b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 -- 172.21.15.116:0/3284725339 shutdown_connections 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.854+0000 7f7ba85bf740 1 -- 172.21.15.116:0/3284725339 wait complete. 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 -- start start 2024-03-24T09:31:25.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 Processor -- start 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754290800 0x561754296d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x5617542972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754299d00 0x5617542977f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561753ec6430 con 0x561754291120 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561753e82e00 con 0x561754299d00 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561754178590 con 0x561754290800 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7ba85bf740 10 client.0 _ll_register_callbacks cb 0x5617541e7fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754290800 0x561754296d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x5617542972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754290800 0x561754296d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:60242/0 (socket says 172.21.15.116:60242) 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/160768042 learned_addr learned my addr 172.21.15.116:0/160768042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754299d00 0x5617542977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9effd640 1 -- 172.21.15.116:0/160768042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754290800 msgr2=0x561754296d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9effd640 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754290800 0x561754296d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9effd640 1 -- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754299d00 msgr2=0x5617542977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9effd640 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754299d00 0x5617542977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.855+0000 7f7b9effd640 1 -- 172.21.15.116:0/160768042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba010ee50 con 0x561754291120 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b9effd640 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x5617542972b0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f7b90006010 tx=0x7f7b90032980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84697549bd82d71c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b7effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b7effd640 1 -- 172.21.15.116:0/160768042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90032c90 con 0x561754291120 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b7effd640 1 -- 172.21.15.116:0/160768042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f7b90034c70 con 0x561754291120 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561753ef27e0 con 0x561754291120 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b7effd640 1 -- 172.21.15.116:0/160768042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b9002c7f0 con 0x561754291120 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 10 client.18315 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5617541f1bd0 con 0x561754291120 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b7effd640 1 -- 172.21.15.116:0/160768042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561753ef27e0 con 0x561754291120 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7b7effd640 1 -- 172.21.15.116:0/160768042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7b900042b0 con 0x561754291120 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 10 client.18315 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 10 client.18315 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 20 client.18315 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617542a9ba0 0x5617542abf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 4 client.18315 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.856+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561753f2f720 con 0x5617542a9ba0 2024-03-24T09:31:25.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.857+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617542a9ba0 0x5617542abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.857+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617542a9ba0 0x5617542abf90 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ade36d11a4a8147 server_cookie=3e93837212e85a16 in_seq=0 out_seq=0 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.857+0000 7f7b7effd640 10 client.18315 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7b7effd640 1 -- 172.21.15.116:0/160768042 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561753f2f720 con 0x5617542a9ba0 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7b7effd640 10 client.18315 handle_command_reply: tid=0 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 client.18315 shutdown 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 msgr2=0x5617542972b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x5617542972b0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f7b90006010 tx=0x7f7b90032980 comp rx=0 tx=0).stop 2024-03-24T09:31:25.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 shutdown_connections 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617542a9ba0 0x5617542abf90 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754299d00 0x5617542977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x5617542972b0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7b7effd640 0 client.18315 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/160768042 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754290800 0x561754296d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 >> 172.21.15.116:0/160768042 conn(0x56175428af30 msgr2=0x561754285cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 shutdown_connections 2024-03-24T09:31:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 1 -- 172.21.15.116:0/160768042 wait complete. 2024-03-24T09:31:25.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.858+0000 7f7ba85bf740 20 client.18315 trim_cache size 0 max 16384 2024-03-24T09:31:25.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 Processor -- start 2024-03-24T09:31:25.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 -- start start 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x5617542994a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542a5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x5617542b1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56175428ada0 con 0x561754299d00 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561754290240 con 0x561754290800 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561754277e20 con 0x561754291120 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x5617542b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x5617542994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x5617542b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54862/0 (socket says 172.21.15.116:54862) 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7b9effd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.871+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/1423489593 learned_addr learned my addr 172.21.15.116:0/1423489593 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/1423489593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 msgr2=0x5617542a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/1423489593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 msgr2=0x5617542994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x5617542994a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/1423489593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56175418dc80 con 0x561754299d00 2024-03-24T09:31:25.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x5617542b1860 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f7b94005eb0 tx=0x7f7b940096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff271021be8fe7b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b857fa640 1 -- 172.21.15.116:0/1423489593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94016040 con 0x561754299d00 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b857fa640 1 -- 172.21.15.116:0/1423489593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b94013e20 con 0x561754299d00 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7b857fa640 1 -- 172.21.15.116:0/1423489593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94005280 con 0x561754299d00 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 -- 172.21.15.116:0/1423489593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 msgr2=0x5617542b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x5617542b1860 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f7b94005eb0 tx=0x7f7b940096c0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 -- 172.21.15.116:0/1423489593 shutdown_connections 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754299d00 0x5617542b1860 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754291120 0x5617542a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/1423489593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x5617542994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.872+0000 7f7ba85bf740 1 -- 172.21.15.116:0/1423489593 >> 172.21.15.116:0/1423489593 conn(0x56175428af30 msgr2=0x561754285cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 -- 172.21.15.116:0/1423489593 shutdown_connections 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 -- 172.21.15.116:0/1423489593 wait complete. 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 -- start start 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 Processor -- start 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x56175429d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x56175429dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754299d00 0x56175429e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56175430efb0 con 0x561754291120 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56175430f120 con 0x561754290800 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56175430f2e0 con 0x561754299d00 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7ba85bf740 10 client.0 _ll_register_callbacks cb 0x7f7b94009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7b9effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x56175429dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x56175429d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7b9e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x56175429d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:41218/0 (socket says 172.21.15.116:41218) 2024-03-24T09:31:25.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/2053154531 learned_addr learned my addr 172.21.15.116:0/2053154531 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.873+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754299d00 0x56175429e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/2053154531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754299d00 msgr2=0x56175429e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754299d00 0x56175429e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/2053154531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 msgr2=0x56175429dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x56175429dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9e7fc640 1 -- 172.21.15.116:0/2053154531 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b94013130 con 0x561754290800 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9effd640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x56175429dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b9e7fc640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x56175429d670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009e80 tx=0x7f7b940134c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb01fc09928f8ada server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:25.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b7effd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b7effd640 1 -- 172.21.15.116:0/2053154531 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94016030 con 0x561754290800 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b7effd640 1 -- 172.21.15.116:0/2053154531 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f7b94005250 con 0x561754290800 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b90032df0 con 0x561754290800 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b7effd640 1 -- 172.21.15.116:0/2053154531 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7b94005410 con 0x561754290800 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 10 client.28136 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5617541b3920 con 0x561754290800 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b7effd640 1 -- 172.21.15.116:0/2053154531 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7b90032df0 con 0x561754290800 2024-03-24T09:31:25.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7b7effd640 1 -- 172.21.15.116:0/2053154531 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f7b94011550 con 0x561754290800 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 10 client.28136 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 10 client.28136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 20 client.28136 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617543109a0 0x561754312d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 4 client.28136 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.874+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561753f7e890 con 0x5617543109a0 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7b9effd640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617543109a0 0x561754312d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:25.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7b9effd640 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617543109a0 0x561754312d90 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80a91c963a8554e0 server_cookie=f2527d23c1e5631b in_seq=0 out_seq=0 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7b7effd640 10 client.28136 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7b7effd640 1 -- 172.21.15.116:0/2053154531 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561753f7e890 con 0x5617543109a0 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7b7effd640 10 client.28136 handle_command_reply: tid=0 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 client.28136 shutdown 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 msgr2=0x56175429d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x56175429d670 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009e80 tx=0x7f7b940134c0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 shutdown_connections 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561754299d00 0x56175429e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561754291120 0x56175429dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x5617543109a0 0x561754312d90 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 --2- 172.21.15.116:0/2053154531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561754290800 0x56175429d670 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7b7effd640 0 client.28136 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 >> 172.21.15.116:0/2053154531 conn(0x56175428af30 msgr2=0x561754296ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 shutdown_connections 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 1 -- 172.21.15.116:0/2053154531 wait complete. 2024-03-24T09:31:25.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.875+0000 7f7ba85bf740 20 client.28136 trim_cache size 0 max 16384 2024-03-24T09:31:25.889 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:25.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7b880a2230 msgr2=0x7f7b880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7b880a2230 0x7f7b880a46f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7b980cc100 tx=0x7f7b980c5460 comp rx=0 tx=0).stop 2024-03-24T09:31:25.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 msgr2=0x7f7ba0197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:25.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 0x7f7ba0197a90 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800e970 tx=0x7f7b9800ed70 comp rx=0 tx=0).stop 2024-03-24T09:31:25.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 shutdown_connections 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ba0107690 0x7f7ba0197a90 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba00692b0 0x7f7ba019de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f7b880a2230 0x7f7b880a46f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 --2- 172.21.15.116:0/2836693450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7ba00688f0 0x7f7ba019d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 >> 172.21.15.116:0/2836693450 conn(0x7f7ba0077710 msgr2=0x7f7ba00fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 shutdown_connections 2024-03-24T09:31:25.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:25.876+0000 7f7ba5902640 1 -- 172.21.15.116:0/2836693450 wait complete. 2024-03-24T09:31:26.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:26 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3177567004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:31:26.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:26 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3177567004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:31:26.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:26 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3177567004' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:31:26.892 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:31:26.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:31:26.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:31:27.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 -- 172.21.15.116:0/2685056483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c1076c0 msgr2=0x7fd14c107aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 --2- 172.21.15.116:0/2685056483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c1076c0 0x7fd14c107aa0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd144009930 tx=0x7fd14402bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:27.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 -- 172.21.15.116:0/2685056483 shutdown_connections 2024-03-24T09:31:27.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 --2- 172.21.15.116:0/2685056483 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c102070 0x7fd14c10e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 --2- 172.21.15.116:0/2685056483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd14c1016b0 0x7fd14c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 --2- 172.21.15.116:0/2685056483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c1076c0 0x7fd14c107aa0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 -- 172.21.15.116:0/2685056483 >> 172.21.15.116:0/2685056483 conn(0x7fd14c0fd140 msgr2=0x7fd14c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 -- 172.21.15.116:0/2685056483 shutdown_connections 2024-03-24T09:31:27.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.006+0000 7fd153ff4640 1 -- 172.21.15.116:0/2685056483 wait complete. 2024-03-24T09:31:27.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 Processor -- start 2024-03-24T09:31:27.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 -- start start 2024-03-24T09:31:27.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c1016b0 0x7fd14c19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 0x7fd14c19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd14c1076c0 0x7fd14c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c1980f0 con 0x7fd14c102070 2024-03-24T09:31:27.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c198260 con 0x7fd14c1076c0 2024-03-24T09:31:27.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd153ff4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd14c198420 con 0x7fd14c1016b0 2024-03-24T09:31:27.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 0x7fd14c19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 0x7fd14c19de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41116/0 (socket says 172.21.15.116:41116) 2024-03-24T09:31:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 -- 172.21.15.116:0/1896454621 learned_addr learned my addr 172.21.15.116:0/1896454621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd152591640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd14c1076c0 0x7fd14c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd151d90640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c1016b0 0x7fd14c19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 -- 172.21.15.116:0/1896454621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c1016b0 msgr2=0x7fd14c19d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c1016b0 0x7fd14c19d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 -- 172.21.15.116:0/1896454621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd14c1076c0 msgr2=0x7fd14c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd14c1076c0 0x7fd14c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 -- 172.21.15.116:0/1896454621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd144009560 con 0x7fd14c102070 2024-03-24T09:31:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.007+0000 7fd15158f640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 0x7fd14c19de30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00d950 tx=0x7fd13c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.008+0000 7fd142ffd640 1 -- 172.21.15.116:0/1896454621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c009a30 con 0x7fd14c102070 2024-03-24T09:31:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.008+0000 7fd142ffd640 1 -- 172.21.15.116:0/1896454621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd13c0044d0 con 0x7fd14c102070 2024-03-24T09:31:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.008+0000 7fd142ffd640 1 -- 172.21.15.116:0/1896454621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0047a0 con 0x7fd14c102070 2024-03-24T09:31:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.008+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14c198700 con 0x7fd14c102070 2024-03-24T09:31:27.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.008+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14c1a4990 con 0x7fd14c102070 2024-03-24T09:31:27.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.010+0000 7fd142ffd640 1 -- 172.21.15.116:0/1896454621 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd13c009b90 con 0x7fd14c102070 2024-03-24T09:31:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.010+0000 7fd156bd3740 1 Processor -- start 2024-03-24T09:31:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.010+0000 7fd156bd3740 1 -- start start 2024-03-24T09:31:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd142ffd640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1380a22b0 0x7fd1380a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d5760 0x55bf624d5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624de720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624dec60 0x55bf624e1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd156bd3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf62443530 con 0x55bf624dec60 2024-03-24T09:31:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd156bd3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf62440a60 con 0x55bf624d6080 2024-03-24T09:31:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd156bd3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf6243c240 con 0x55bf624d5760 2024-03-24T09:31:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624de720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd152591640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1380a22b0 0x7fd1380a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624de720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:37116/0 (socket says 172.21.15.116:37116) 2024-03-24T09:31:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 -- 172.21.15.116:0/2495181386 learned_addr learned my addr 172.21.15.116:0/2495181386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd142ffd640 1 -- 172.21.15.116:0/1896454621 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd13c0cdda0 con 0x7fd14c102070 2024-03-24T09:31:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd151d90640 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d5760 0x55bf624d5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd152591640 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624dec60 0x55bf624e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 -- 172.21.15.116:0/2495181386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d5760 msgr2=0x55bf624d5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d5760 0x55bf624d5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 -- 172.21.15.116:0/2495181386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624dec60 msgr2=0x55bf624e1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624dec60 0x55bf624e1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 -- 172.21.15.116:0/2495181386 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf6243a8c0 con 0x55bf624d6080 2024-03-24T09:31:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd152591640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1380a22b0 0x7fd1380a4770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd148005fd0 tx=0x7fd148009e90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd15158f640 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624de720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0967e0 tx=0x7fd13c009f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=528eba0f66791243 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.011+0000 7fd137fff640 1 -- 172.21.15.116:0/2495181386 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c09a380 con 0x55bf624d6080 2024-03-24T09:31:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd137fff640 1 -- 172.21.15.116:0/2495181386 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd13c09e4d0 con 0x55bf624d6080 2024-03-24T09:31:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd137fff640 1 -- 172.21.15.116:0/2495181386 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0cb360 con 0x55bf624d6080 2024-03-24T09:31:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 -- 172.21.15.116:0/2495181386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 msgr2=0x55bf624de720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624de720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0967e0 tx=0x7fd13c009f40 comp rx=0 tx=0).stop 2024-03-24T09:31:27.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 -- 172.21.15.116:0/2495181386 shutdown_connections 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624dec60 0x55bf624e1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624de720 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2495181386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d5760 0x55bf624d5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 -- 172.21.15.116:0/2495181386 >> 172.21.15.116:0/2495181386 conn(0x55bf624cfe90 msgr2=0x55bf624d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 -- 172.21.15.116:0/2495181386 shutdown_connections 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.012+0000 7fd156bd3740 1 -- 172.21.15.116:0/2495181386 wait complete. 2024-03-24T09:31:27.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 -- start start 2024-03-24T09:31:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 Processor -- start 2024-03-24T09:31:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624dbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624dc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624dc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf623e0f80 con 0x55bf624d5760 2024-03-24T09:31:27.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf6210f4b0 con 0x55bf624d6080 2024-03-24T09:31:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf623c1ad0 con 0x55bf624dec60 2024-03-24T09:31:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd156bd3740 10 client.0 _ll_register_callbacks cb 0x55bf623f4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd151d90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624dbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd15158f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624dc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd15158f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624dc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:37124/0 (socket says 172.21.15.116:37124) 2024-03-24T09:31:27.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd151d90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624dbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41128/0 (socket says 172.21.15.116:41128) 2024-03-24T09:31:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd15158f640 1 -- 172.21.15.116:0/3275138182 learned_addr learned my addr 172.21.15.116:0/3275138182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.013+0000 7fd152591640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624dc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd151d90640 1 -- 172.21.15.116:0/3275138182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 msgr2=0x55bf624dc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd151d90640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624dc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd151d90640 1 -- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 msgr2=0x55bf624dc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd151d90640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624dc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd151d90640 1 -- 172.21.15.116:0/3275138182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd14c1a4990 con 0x55bf624d5760 2024-03-24T09:31:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd152591640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624dc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd15158f640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624dc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd151d90640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624dbbd0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fd144009230 tx=0x7fd14402c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37de7eb68128c0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd1357fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd1357fa640 1 -- 172.21.15.116:0/3275138182 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14402cbd0 con 0x55bf624d5760 2024-03-24T09:31:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd1357fa640 1 -- 172.21.15.116:0/3275138182 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd144004250 con 0x55bf624d5760 2024-03-24T09:31:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bf624607a0 con 0x55bf624d5760 2024-03-24T09:31:27.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd1357fa640 1 -- 172.21.15.116:0/3275138182 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd144004410 con 0x55bf624d5760 2024-03-24T09:31:27.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 10 client.18333 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:27.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bf623f26d0 con 0x55bf624d5760 2024-03-24T09:31:27.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd1357fa640 1 -- 172.21.15.116:0/3275138182 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bf624607a0 con 0x55bf624d5760 2024-03-24T09:31:27.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd1357fa640 1 -- 172.21.15.116:0/3275138182 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd14403c040 con 0x55bf624d5760 2024-03-24T09:31:27.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 10 client.18333 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 10 client.18333 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 20 client.18333 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf624eecb0 0x55bf624f10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 4 client.18333 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.014+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf624321f0 con 0x55bf624eecb0 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.015+0000 7fd15158f640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf624eecb0 0x55bf624f10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.015+0000 7fd15158f640 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf624eecb0 0x55bf624f10c0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8da59a62b00e86f server_cookie=d3c8113018a3f199 in_seq=0 out_seq=0 2024-03-24T09:31:27.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.015+0000 7fd1357fa640 10 client.18333 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd1357fa640 1 -- 172.21.15.116:0/3275138182 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf624321f0 con 0x55bf624eecb0 2024-03-24T09:31:27.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd1357fa640 10 client.18333 handle_command_reply: tid=0 2024-03-24T09:31:27.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 client.18333 shutdown 2024-03-24T09:31:27.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 msgr2=0x55bf624dbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624dbbd0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fd144009230 tx=0x7fd14402c9a0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 shutdown_connections 2024-03-24T09:31:27.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624dc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf624eecb0 0x55bf624f10c0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d6080 0x55bf624dc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 --2- 172.21.15.116:0/3275138182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624dbbd0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd1357fa640 0 client.18333 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 >> 172.21.15.116:0/3275138182 conn(0x55bf624cfe90 msgr2=0x55bf624d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 shutdown_connections 2024-03-24T09:31:27.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 1 -- 172.21.15.116:0/3275138182 wait complete. 2024-03-24T09:31:27.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.016+0000 7fd156bd3740 20 client.18333 trim_cache size 0 max 16384 2024-03-24T09:31:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.033+0000 7fd156bd3740 1 Processor -- start 2024-03-24T09:31:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.033+0000 7fd156bd3740 1 -- start start 2024-03-24T09:31:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.033+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624ea4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.033+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d6080 0x55bf624dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.033+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624dec60 0x55bf624f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd156bd3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf624eaa30 con 0x55bf624d5760 2024-03-24T09:31:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd156bd3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf624cb620 con 0x55bf624dec60 2024-03-24T09:31:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd156bd3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf624eea60 con 0x55bf624d6080 2024-03-24T09:31:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624ea4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624ea4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41144/0 (socket says 172.21.15.116:41144) 2024-03-24T09:31:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 -- 172.21.15.116:0/2048974545 learned_addr learned my addr 172.21.15.116:0/2048974545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd152591640 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624dec60 0x55bf624f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd15158f640 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d6080 0x55bf624dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 -- 172.21.15.116:0/2048974545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d6080 msgr2=0x55bf624dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d6080 0x55bf624dc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 -- 172.21.15.116:0/2048974545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624dec60 msgr2=0x55bf624f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624dec60 0x55bf624f69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 -- 172.21.15.116:0/2048974545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf623d6c60 con 0x55bf624d5760 2024-03-24T09:31:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd151d90640 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624ea4f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fd144009850 tx=0x7fd144000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3443ebfedd666d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd1377fe640 1 -- 172.21.15.116:0/2048974545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14402cbd0 con 0x55bf624d5760 2024-03-24T09:31:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd1377fe640 1 -- 172.21.15.116:0/2048974545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd14402de90 con 0x55bf624d5760 2024-03-24T09:31:27.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.034+0000 7fd1377fe640 1 -- 172.21.15.116:0/2048974545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144003aa0 con 0x55bf624d5760 2024-03-24T09:31:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- 172.21.15.116:0/2048974545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 msgr2=0x55bf624ea4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624ea4f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fd144009850 tx=0x7fd144000bc0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- 172.21.15.116:0/2048974545 shutdown_connections 2024-03-24T09:31:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624dec60 0x55bf624f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624d6080 0x55bf624dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- 172.21.15.116:0/2048974545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d5760 0x55bf624ea4f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- 172.21.15.116:0/2048974545 >> 172.21.15.116:0/2048974545 conn(0x55bf624cfe90 msgr2=0x55bf624ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- 172.21.15.116:0/2048974545 shutdown_connections 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- 172.21.15.116:0/2048974545 wait complete. 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- start start 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 Processor -- start 2024-03-24T09:31:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d5760 0x55bf624d8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 0x55bf624d8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624d9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bf625541f0 con 0x55bf624d6080 2024-03-24T09:31:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bf62554360 con 0x55bf624d5760 2024-03-24T09:31:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.035+0000 7fd156bd3740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bf62554520 con 0x55bf624dec60 2024-03-24T09:31:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd156bd3740 10 client.0 _ll_register_callbacks cb 0x7fd144005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 0x55bf624d8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 0x55bf624d8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41160/0 (socket says 172.21.15.116:41160) 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 -- 172.21.15.116:0/1954353520 learned_addr learned my addr 172.21.15.116:0/1954353520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd151d90640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d5760 0x55bf624d8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd152591640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624d9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 -- 172.21.15.116:0/1954353520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 msgr2=0x55bf624d9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624d9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 -- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d5760 msgr2=0x55bf624d8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d5760 0x55bf624d8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 -- 172.21.15.116:0/1954353520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf623f26d0 con 0x55bf624d6080 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd152591640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624d9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd15158f640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 0x55bf624d8fa0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0967e0 tx=0x7fd13c00df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369e8022e565bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd1357fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd1357fa640 1 -- 172.21.15.116:0/1954353520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c096620 con 0x55bf624d6080 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd1357fa640 1 -- 172.21.15.116:0/1954353520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd13c09a380 con 0x55bf624d6080 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd13c09e4d0 con 0x55bf624d6080 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd1357fa640 1 -- 172.21.15.116:0/1954353520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd13c09e160 con 0x55bf624d6080 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd156bd3740 10 client.18339 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bf623fc970 con 0x55bf624d6080 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.036+0000 7fd1357fa640 1 -- 172.21.15.116:0/1954353520 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd13c09e4d0 con 0x55bf624d6080 2024-03-24T09:31:27.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd1357fa640 1 -- 172.21.15.116:0/1954353520 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd13c0c91c0 con 0x55bf624d6080 2024-03-24T09:31:27.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd156bd3740 10 client.18339 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:27.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd156bd3740 10 client.18339 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:27.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd156bd3740 20 client.18339 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:27.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd156bd3740 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf62555ac0 0x55bf62557ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd156bd3740 4 client.18339 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:31:27.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bf6213b860 con 0x55bf62555ac0 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd151d90640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf62555ac0 0x55bf62557ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd1357fa640 10 client.18339 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.037+0000 7fd151d90640 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf62555ac0 0x55bf62557ee0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf7a081ecb71078f server_cookie=3d0f24d8e63c73a8 in_seq=0 out_seq=0 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd1357fa640 1 -- 172.21.15.116:0/1954353520 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bf6213b860 con 0x55bf62555ac0 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd1357fa640 10 client.18339 handle_command_reply: tid=0 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 client.18339 shutdown 2024-03-24T09:31:27.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 msgr2=0x55bf624d8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 0x55bf624d8fa0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0967e0 tx=0x7fd13c00df40 comp rx=0 tx=0).stop 2024-03-24T09:31:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 shutdown_connections 2024-03-24T09:31:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bf624dec60 0x55bf624d9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bf624d6080 0x55bf624d8fa0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55bf62555ac0 0x55bf62557ee0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 --2- 172.21.15.116:0/1954353520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bf624d5760 0x55bf624d8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 >> 172.21.15.116:0/1954353520 conn(0x55bf624cfe90 msgr2=0x55bf624d1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.038+0000 7fd1357fa640 0 client.18339 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 shutdown_connections 2024-03-24T09:31:27.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd156bd3740 1 -- 172.21.15.116:0/1954353520 wait complete. 2024-03-24T09:31:27.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd156bd3740 20 client.18339 trim_cache size 0 max 16384 2024-03-24T09:31:27.062 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:31:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:31:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "e90465b5-cdff-4488-b3b3-a2fe678c3371", 2024-03-24T09:31:27.062 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:31:27.062 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:31:27.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1380a22b0 msgr2=0x7fd1380a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1380a22b0 0x7fd1380a4770 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd148005fd0 tx=0x7fd148009e90 comp rx=0 tx=0).stop 2024-03-24T09:31:27.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 msgr2=0x7fd14c19de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 0x7fd14c19de30 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00d950 tx=0x7fd13c00de10 comp rx=0 tx=0).stop 2024-03-24T09:31:27.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 shutdown_connections 2024-03-24T09:31:27.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd14c1076c0 0x7fd14c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd14c102070 0x7fd14c19de30 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd1380a22b0 0x7fd1380a4770 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 --2- 172.21.15.116:0/1896454621 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd14c1016b0 0x7fd14c19d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 >> 172.21.15.116:0/1896454621 conn(0x7fd14c0fd140 msgr2=0x7fd14c0fda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 shutdown_connections 2024-03-24T09:31:27.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.039+0000 7fd153ff4640 1 -- 172.21.15.116:0/1896454621 wait complete. 2024-03-24T09:31:27.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:31:27.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:31:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 -- 172.21.15.116:0/2439374079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c40692b0 msgr2=0x7f27c410e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 --2- 172.21.15.116:0/2439374079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c40692b0 0x7f27c410e550 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009cd0 tx=0x7f27bc02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 -- 172.21.15.116:0/2439374079 shutdown_connections 2024-03-24T09:31:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 --2- 172.21.15.116:0/2439374079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c40692b0 0x7f27c410e550 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 --2- 172.21.15.116:0/2439374079 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 0x7f27c4068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 --2- 172.21.15.116:0/2439374079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27c4107690 0x7f27c4107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 -- 172.21.15.116:0/2439374079 >> 172.21.15.116:0/2439374079 conn(0x7f27c4077710 msgr2=0x7f27c4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 -- 172.21.15.116:0/2439374079 shutdown_connections 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.188+0000 7f27cb720640 1 -- 172.21.15.116:0/2439374079 wait complete. 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 Processor -- start 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 -- start start 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 0x7f27c41005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27c40692b0 0x7f27c4100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 0x7f27c4101050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a5fe0 con 0x7f27c4107690 2024-03-24T09:31:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a6150 con 0x7f27c40692b0 2024-03-24T09:31:27.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.189+0000 7f27cb720640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a6310 con 0x7f27c40688f0 2024-03-24T09:31:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 0x7f27c4101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c94bc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 0x7f27c41005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 0x7f27c4101050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41192/0 (socket says 172.21.15.116:41192) 2024-03-24T09:31:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 -- 172.21.15.116:0/2874477572 learned_addr learned my addr 172.21.15.116:0/2874477572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c94bc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 0x7f27c41005d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:54686/0 (socket says 172.21.15.116:54686) 2024-03-24T09:31:27.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27c40692b0 0x7f27c4100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 -- 172.21.15.116:0/2874477572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 msgr2=0x7f27c41005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 0x7f27c41005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 -- 172.21.15.116:0/2874477572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27c40692b0 msgr2=0x7f27c4100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27c40692b0 0x7f27c4100b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 -- 172.21.15.116:0/2874477572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27bc009900 con 0x7f27c4107690 2024-03-24T09:31:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.190+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 0x7f27c4101050 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f27bc005e30 tx=0x7f27bc035ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.191+0000 7f27ba7fc640 1 -- 172.21.15.116:0/2874477572 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0329e0 con 0x7f27c4107690 2024-03-24T09:31:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.191+0000 7f27ba7fc640 1 -- 172.21.15.116:0/2874477572 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27bc035e90 con 0x7f27c4107690 2024-03-24T09:31:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.191+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c41a6590 con 0x7f27c4107690 2024-03-24T09:31:27.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.191+0000 7f27ba7fc640 1 -- 172.21.15.116:0/2874477572 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0046f0 con 0x7f27c4107690 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.191+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c41a6a90 con 0x7f27c4107690 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ba7fc640 1 -- 172.21.15.116:0/2874477572 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f27bc03b040 con 0x7f27c4107690 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ce3d6740 1 Processor -- start 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ce3d6740 1 -- start start 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c1845b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ba7fc640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27b00a22b0 0x7f27b00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c18d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.194+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c18d6d0 0x56390c18fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56390c0a8f30 con 0x56390c18d6d0 2024-03-24T09:31:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56390c0f2810 con 0x56390c184af0 2024-03-24T09:31:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56390c0efd40 con 0x56390c1841d0 2024-03-24T09:31:27.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27ba7fc640 1 -- 172.21.15.116:0/2874477572 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f27bc0f0cc0 con 0x7f27c4107690 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c18d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c94bc640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27b00a22b0 0x7f27b00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c9cbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c18d6d0 0x56390c18fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c18d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:37170/0 (socket says 172.21.15.116:37170) 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 -- 172.21.15.116:0/2891747294 learned_addr learned my addr 172.21.15.116:0/2891747294 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c1845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 -- 172.21.15.116:0/2891747294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 msgr2=0x56390c1845b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c1845b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 -- 172.21.15.116:0/2891747294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c18d6d0 msgr2=0x56390c18fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c18d6d0 0x56390c18fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c8cbb640 1 -- 172.21.15.116:0/2891747294 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56390c0acea0 con 0x56390c184af0 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.195+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c18d6d0 0x56390c18fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.196+0000 7f27c94bc640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27b00a22b0 0x7f27b00a4770 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27c000bb60 tx=0x7f27c000b390 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.196+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c18d190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f27b40058c0 tx=0x7f27b4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4313df6c8f146644 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.196+0000 7f27af7fe640 1 -- 172.21.15.116:0/2891747294 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4005020 con 0x56390c184af0 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.196+0000 7f27af7fe640 1 -- 172.21.15.116:0/2891747294 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27b4005630 con 0x56390c184af0 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.196+0000 7f27af7fe640 1 -- 172.21.15.116:0/2891747294 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b400fa20 con 0x56390c184af0 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 -- 172.21.15.116:0/2891747294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 msgr2=0x56390c18d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c18d190 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f27b40058c0 tx=0x7f27b4005850 comp rx=0 tx=0).stop 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 -- 172.21.15.116:0/2891747294 shutdown_connections 2024-03-24T09:31:27.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c18d6d0 0x56390c18fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c18d190 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/2891747294 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c1845b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 -- 172.21.15.116:0/2891747294 >> 172.21.15.116:0/2891747294 conn(0x56390c17e900 msgr2=0x56390c17ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 -- 172.21.15.116:0/2891747294 shutdown_connections 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 -- 172.21.15.116:0/2891747294 wait complete. 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 -- start start 2024-03-24T09:31:27.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.197+0000 7f27ce3d6740 1 Processor -- start 2024-03-24T09:31:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c18a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 0x56390c18aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c18d6d0 0x56390c18b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56390c0eb520 con 0x56390c184af0 2024-03-24T09:31:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56390bdbe430 con 0x56390c18d6d0 2024-03-24T09:31:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56390bd7ae00 con 0x56390c1841d0 2024-03-24T09:31:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27ce3d6740 10 client.0 _ll_register_callbacks cb 0x56390c089d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27c8cbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 0x56390c18aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27c8cbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 0x56390c18aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41220/0 (socket says 172.21.15.116:41220) 2024-03-24T09:31:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27c8cbb640 1 -- 172.21.15.116:0/3698442744 learned_addr learned my addr 172.21.15.116:0/3698442744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c18d6d0 0x56390c18b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.198+0000 7f27c94bc640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c18a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27c8cbb640 1 -- 172.21.15.116:0/3698442744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 msgr2=0x56390c18a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c18a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27c8cbb640 1 -- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c18d6d0 msgr2=0x56390c18b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c18d6d0 0x56390c18b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27c8cbb640 1 -- 172.21.15.116:0/3698442744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b4019040 con 0x56390c184af0 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 0x56390c18aca0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f27b4006010 tx=0x7f27b4004b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351197c7d7c1f18c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27acff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27acff9640 1 -- 172.21.15.116:0/3698442744 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4022040 con 0x56390c184af0 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56390bdea7e0 con 0x56390c184af0 2024-03-24T09:31:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27acff9640 1 -- 172.21.15.116:0/3698442744 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f27b4000e40 con 0x56390c184af0 2024-03-24T09:31:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27acff9640 1 -- 172.21.15.116:0/3698442744 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f27b4009d00 con 0x56390c184af0 2024-03-24T09:31:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27ce3d6740 10 client.18351 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.199+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56390c0e9ba0 con 0x56390c184af0 2024-03-24T09:31:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27acff9640 1 -- 172.21.15.116:0/3698442744 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x56390bdea7e0 con 0x56390c184af0 2024-03-24T09:31:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27acff9640 1 -- 172.21.15.116:0/3698442744 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f27b401e3f0 con 0x56390c184af0 2024-03-24T09:31:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27ce3d6740 10 client.18351 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27ce3d6740 10 client.18351 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27ce3d6740 20 client.18351 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c19db50 0x56390c19ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27ce3d6740 4 client.18351 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56390be27720 con 0x56390c19db50 2024-03-24T09:31:27.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.200+0000 7f27c94bc640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c19db50 0x56390c19ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.201+0000 7f27acff9640 10 client.18351 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.201+0000 7f27c94bc640 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c19db50 0x56390c19ff70 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37379aaf6a671b1 server_cookie=b491ae82be6c4147 in_seq=0 out_seq=0 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27acff9640 1 -- 172.21.15.116:0/3698442744 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56390be27720 con 0x56390c19db50 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27acff9640 10 client.18351 handle_command_reply: tid=0 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 client.18351 shutdown 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 msgr2=0x56390c18aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 0x56390c18aca0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f27b4006010 tx=0x7f27b4004b90 comp rx=0 tx=0).stop 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 shutdown_connections 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c19db50 0x56390c19ff70 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c18d6d0 0x56390c18b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c184af0 0x56390c18aca0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/3698442744 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c1841d0 0x56390c18a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 >> 172.21.15.116:0/3698442744 conn(0x56390c17e900 msgr2=0x56390c17db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.202+0000 7f27acff9640 0 client.18351 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.203+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 shutdown_connections 2024-03-24T09:31:27.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.203+0000 7f27ce3d6740 1 -- 172.21.15.116:0/3698442744 wait complete. 2024-03-24T09:31:27.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.203+0000 7f27ce3d6740 20 client.18351 trim_cache size 0 max 16384 2024-03-24T09:31:27.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 Processor -- start 2024-03-24T09:31:27.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 -- start start 2024-03-24T09:31:27.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c18ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c1898c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c198f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56390c070430 con 0x56390c1841d0 2024-03-24T09:31:27.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56390c17a1d0 con 0x56390c184af0 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56390c189e00 con 0x56390c18d6d0 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c18ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c18ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41224/0 (socket says 172.21.15.116:41224) 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 -- 172.21.15.116:0/4151487101 learned_addr learned my addr 172.21.15.116:0/4151487101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c1898c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 -- 172.21.15.116:0/4151487101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 msgr2=0x56390c198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 -- 172.21.15.116:0/4151487101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 msgr2=0x56390c1898c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c1898c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.227+0000 7f27c94bc640 1 -- 172.21.15.116:0/4151487101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56390c085b20 con 0x56390c1841d0 2024-03-24T09:31:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.228+0000 7f27c94bc640 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c18ce90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f27c0011540 tx=0x7f27c0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3af5763ea80e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.228+0000 7f27aeffd640 1 -- 172.21.15.116:0/4151487101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0024040 con 0x56390c1841d0 2024-03-24T09:31:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.228+0000 7f27aeffd640 1 -- 172.21.15.116:0/4151487101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27c0015070 con 0x56390c1841d0 2024-03-24T09:31:27.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.228+0000 7f27aeffd640 1 -- 172.21.15.116:0/4151487101 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c001fae0 con 0x56390c1841d0 2024-03-24T09:31:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.228+0000 7f27ce3d6740 1 -- 172.21.15.116:0/4151487101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 msgr2=0x56390c18ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.228+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c18ce90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f27c0011540 tx=0x7f27c0013000 comp rx=0 tx=0).stop 2024-03-24T09:31:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 -- 172.21.15.116:0/4151487101 shutdown_connections 2024-03-24T09:31:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c1898c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/4151487101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c18ce90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 -- 172.21.15.116:0/4151487101 >> 172.21.15.116:0/4151487101 conn(0x56390c17e900 msgr2=0x56390c17d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 -- 172.21.15.116:0/4151487101 shutdown_connections 2024-03-24T09:31:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 -- 172.21.15.116:0/4151487101 wait complete. 2024-03-24T09:31:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 -- start start 2024-03-24T09:31:27.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 Processor -- start 2024-03-24T09:31:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.229+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c1957f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c186fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27ce3d6740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c187510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56390c187c10 con 0x56390c1841d0 2024-03-24T09:31:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56390c187d80 con 0x56390c184af0 2024-03-24T09:31:27.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27ce3d6740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x56390c202e70 con 0x56390c18d6d0 2024-03-24T09:31:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27ce3d6740 10 client.0 _ll_register_callbacks cb 0x7f27bc0b95f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c1957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c1957f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41226/0 (socket says 172.21.15.116:41226) 2024-03-24T09:31:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 -- 172.21.15.116:0/121595443 learned_addr learned my addr 172.21.15.116:0/121595443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c9cbd640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c187510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c186fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 -- 172.21.15.116:0/121595443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 msgr2=0x56390c187510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c187510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 -- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 msgr2=0x56390c186fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c186fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 -- 172.21.15.116:0/121595443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c0010cb0 con 0x56390c1841d0 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27c94bc640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c1957f0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c001f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15c7ea4198e511e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27acff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.230+0000 7f27acff9640 1 -- 172.21.15.116:0/121595443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0028dd0 con 0x56390c1841d0 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27b4005630 con 0x56390c1841d0 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27acff9640 1 -- 172.21.15.116:0/121595443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f27c0017070 con 0x56390c1841d0 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27acff9640 1 -- 172.21.15.116:0/121595443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f27c00283f0 con 0x56390c1841d0 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 10 client.18357 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56390c0ab700 con 0x56390c1841d0 2024-03-24T09:31:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27acff9640 1 -- 172.21.15.116:0/121595443 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f27b4005630 con 0x56390c1841d0 2024-03-24T09:31:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27acff9640 1 -- 172.21.15.116:0/121595443 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f27c00394c0 con 0x56390c1841d0 2024-03-24T09:31:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 10 client.18357 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 10 client.18357 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 20 client.18357 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c2044f0 0x56390c2068e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 4 client.18357 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56390be76890 con 0x56390c2044f0 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.231+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c2044f0 0x56390c2068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27c8cbb640 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c2044f0 0x56390c2068e0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d3191b8ade7e534 server_cookie=dbb1036b95eeb30f in_seq=0 out_seq=0 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27acff9640 10 client.18357 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27acff9640 1 -- 172.21.15.116:0/121595443 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56390be76890 con 0x56390c2044f0 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27acff9640 10 client.18357 handle_command_reply: tid=0 2024-03-24T09:31:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27ce3d6740 1 client.18357 shutdown 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 msgr2=0x56390c1957f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.232+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c1957f0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f27c0000c00 tx=0x7f27c001f430 comp rx=0 tx=0).stop 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 shutdown_connections 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56390c18d6d0 0x56390c187510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x56390c2044f0 0x56390c2068e0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27acff9640 0 client.18357 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56390c184af0 0x56390c186fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 --2- 172.21.15.116:0/121595443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56390c1841d0 0x56390c1957f0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 >> 172.21.15.116:0/121595443 conn(0x56390c17e900 msgr2=0x56390c18af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 shutdown_connections 2024-03-24T09:31:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 1 -- 172.21.15.116:0/121595443 wait complete. 2024-03-24T09:31:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27ce3d6740 20 client.18357 trim_cache size 0 max 16384 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: "e90465b5-cdff-4488-b3b3-a2fe678c3371": { 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "e90465b5-cdff-4488-b3b3-a2fe678c3371", 2024-03-24T09:31:27.247 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:31:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:31:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:31:27.248 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:31:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27b00a22b0 msgr2=0x7f27b00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27b00a22b0 0x7f27b00a4770 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f27c000bb60 tx=0x7f27c000b390 comp rx=0 tx=0).stop 2024-03-24T09:31:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 msgr2=0x7f27c4101050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 0x7f27c4101050 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f27bc005e30 tx=0x7f27bc035ca0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 shutdown_connections 2024-03-24T09:31:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27c4107690 0x7f27c4101050 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27c40692b0 0x7f27c4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27b00a22b0 0x7f27b00a4770 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 --2- 172.21.15.116:0/2874477572 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27c40688f0 0x7f27c41005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 >> 172.21.15.116:0/2874477572 conn(0x7f27c4077710 msgr2=0x7f27c40fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 shutdown_connections 2024-03-24T09:31:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:27.233+0000 7f27cb720640 1 -- 172.21.15.116:0/2874477572 wait complete. 2024-03-24T09:31:27.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:27 smithi116 ceph-mon[32207]: pgmap v289: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2024-03-24T09:31:27.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:27 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:31:27.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:27 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:27.251 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:27 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:31:27.252 INFO:tasks.cephfs.filesystem:scrub status for tag:e90465b5-cdff-4488-b3b3-a2fe678c3371 - {'path': '/', 'tag': 'e90465b5-cdff-4488-b3b3-a2fe678c3371', 'options': 'recursive,force'} 2024-03-24T09:31:27.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:27 smithi143 ceph-mon[35965]: pgmap v289: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2024-03-24T09:31:27.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:27 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:31:27.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:27 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:27.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:27 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:31:27.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:27 smithi117 ceph-mon[35788]: pgmap v289: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 89 B/s wr, 8 op/s 2024-03-24T09:31:27.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:27 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:31:27.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:27 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:27.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:27 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:31:28.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:31:28.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:31:28.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:31:28.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:31:28.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:31:28.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:28 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:31:28.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:28 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:31:28.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:31:28.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:31:28.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:31:28.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:31:28.569 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:28.570 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:28 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:31:29.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:29 smithi143 ceph-mon[35965]: pgmap v290: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:31:29.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:29 smithi117 ceph-mon[35788]: pgmap v290: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:31:29.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:29 smithi116 ceph-mon[32207]: pgmap v290: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-24T09:31:31.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:31 smithi143 ceph-mon[35965]: pgmap v291: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:31.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:31 smithi117 ceph-mon[35788]: pgmap v291: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:31.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:31 smithi116 ceph-mon[32207]: pgmap v291: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:33.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:33 smithi143 ceph-mon[35965]: pgmap v292: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:33.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:33 smithi117 ceph-mon[35788]: pgmap v292: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:33.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:33 smithi116 ceph-mon[32207]: pgmap v292: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:35.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:35 smithi143 ceph-mon[35965]: pgmap v293: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:35.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:35 smithi117 ceph-mon[35788]: pgmap v293: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:35.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:35 smithi116 ceph-mon[32207]: pgmap v293: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:37.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:37 smithi143 ceph-mon[35965]: pgmap v294: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:37.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:37 smithi117 ceph-mon[35788]: pgmap v294: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:37.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:37 smithi116 ceph-mon[32207]: pgmap v294: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:38.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:38 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:38.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:38 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:38.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:38 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:38.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:38 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:38 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:38.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:38 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:39.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:39 smithi143 ceph-mon[35965]: pgmap v295: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:39.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:39 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:39.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:39 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:39.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:39 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:31:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:39 smithi117 ceph-mon[35788]: pgmap v295: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:39.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:39 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:39.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:39 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:39.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:39 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:31:39.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:39 smithi116 ceph-mon[32207]: pgmap v295: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:39.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:39 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:39.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:39 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:39.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:39 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:31:41.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:41 smithi143 ceph-mon[35965]: pgmap v296: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:41.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:41 smithi117 ceph-mon[35788]: pgmap v296: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:41.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:41 smithi116 ceph-mon[32207]: pgmap v296: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-03-24T09:31:43.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:43 smithi143 ceph-mon[35965]: pgmap v297: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:43.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:43 smithi117 ceph-mon[35788]: pgmap v297: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:43.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:43 smithi116 ceph-mon[32207]: pgmap v297: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s 2024-03-24T09:31:45.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:45 smithi143 ceph-mon[35965]: pgmap v298: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:45.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:45 smithi117 ceph-mon[35788]: pgmap v298: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:45.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:45 smithi116 ceph-mon[32207]: pgmap v298: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:47.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:47 smithi143 ceph-mon[35965]: pgmap v299: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:47.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:47 smithi117 ceph-mon[35788]: pgmap v299: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:47.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:47 smithi116 ceph-mon[32207]: pgmap v299: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:49.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:49 smithi143 ceph-mon[35965]: pgmap v300: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:49.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:49 smithi117 ceph-mon[35788]: pgmap v300: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:49.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:49 smithi116 ceph-mon[32207]: pgmap v300: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:51 smithi143 ceph-mon[35965]: pgmap v301: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:51.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:51 smithi117 ceph-mon[35788]: pgmap v301: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:51.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:51 smithi116 ceph-mon[32207]: pgmap v301: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:53.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:53 smithi143 ceph-mon[35965]: pgmap v302: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:53.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:53 smithi117 ceph-mon[35788]: pgmap v302: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:53.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:53 smithi116 ceph-mon[32207]: pgmap v302: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:55 smithi143 ceph-mon[35965]: pgmap v303: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:55.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:55 smithi117 ceph-mon[35788]: pgmap v303: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:55.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:55 smithi116 ceph-mon[32207]: pgmap v303: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:57.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:31:57.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:31:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 -- 172.21.15.116:0/3029080731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100ff0 msgr2=0x7f3d9010d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/3029080731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100ff0 0x7f3d9010d500 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009cd0 tx=0x7f3d8802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 -- 172.21.15.116:0/3029080731 shutdown_connections 2024-03-24T09:31:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/3029080731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100ff0 0x7f3d9010d500 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/3029080731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d90100630 0x7f3d90100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/3029080731 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 -- 172.21.15.116:0/3029080731 >> 172.21.15.116:0/3029080731 conn(0x7f3d90077770 msgr2=0x7f3d90077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 -- 172.21.15.116:0/3029080731 shutdown_connections 2024-03-24T09:31:57.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.368+0000 7f3d94dd7640 1 -- 172.21.15.116:0/3029080731 wait complete. 2024-03-24T09:31:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 Processor -- start 2024-03-24T09:31:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 -- start start 2024-03-24T09:31:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100630 0x7f3d9019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d90100ff0 0x7f3d9019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901980a0 con 0x7f3d90106640 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3d90198210 con 0x7f3d90100ff0 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d94dd7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901983d0 con 0x7f3d90100630 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35332/0 (socket says 172.21.15.116:35332) 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.369+0000 7f3d8e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100630 0x7f3d9019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8ed9d640 1 -- 172.21.15.116:0/838298651 learned_addr learned my addr 172.21.15.116:0/838298651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d87fff640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d90100ff0 0x7f3d9019dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8e59c640 1 -- 172.21.15.116:0/838298651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d90100ff0 msgr2=0x7f3d9019dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d90100ff0 0x7f3d9019dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8e59c640 1 -- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 msgr2=0x7f3d90197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8e59c640 1 -- 172.21.15.116:0/838298651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d80009490 con 0x7f3d90100630 2024-03-24T09:31:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8ed9d640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100630 0x7f3d9019d8a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000e9f0 tx=0x7f3d8000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d877fe640 1 -- 172.21.15.116:0/838298651 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8000cd80 con 0x7f3d90100630 2024-03-24T09:31:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d877fe640 1 -- 172.21.15.116:0/838298651 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d800044d0 con 0x7f3d90100630 2024-03-24T09:31:57.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d88009900 con 0x7f3d90100630 2024-03-24T09:31:57.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d877fe640 1 -- 172.21.15.116:0/838298651 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d80010430 con 0x7f3d90100630 2024-03-24T09:31:57.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.370+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d901a4760 con 0x7f3d90100630 2024-03-24T09:31:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.372+0000 7f3d877fe640 1 -- 172.21.15.116:0/838298651 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f3d80003fa0 con 0x7f3d90100630 2024-03-24T09:31:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 Processor -- start 2024-03-24T09:31:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 -- start start 2024-03-24T09:31:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e639c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e63f00 0x559180e6c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 0x559180e6eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559180d83f30 con 0x559180e635e0 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559180dcd810 con 0x559180e63f00 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d979ba740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559180dcad40 con 0x559180e6cae0 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d877fe640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3d7c0a2230 0x7f3d7c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 0x559180e6eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d8e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e63f00 0x559180e6c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 0x559180e6eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:49872/0 (socket says 172.21.15.116:49872) 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d87fff640 1 -- 172.21.15.116:0/4173204553 learned_addr learned my addr 172.21.15.116:0/4173204553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.373+0000 7f3d8ed9d640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3d7c0a2230 0x7f3d7c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d877fe640 1 -- 172.21.15.116:0/838298651 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3d80014070 con 0x7f3d90100630 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d87fff640 1 -- 172.21.15.116:0/4173204553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e63f00 msgr2=0x559180e6c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d87fff640 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e63f00 0x559180e6c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d87fff640 1 -- 172.21.15.116:0/4173204553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 msgr2=0x559180e639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d87fff640 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e639c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d87fff640 1 -- 172.21.15.116:0/4173204553 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559180d87ea0 con 0x559180e6cae0 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e63f00 0x559180e6c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:57.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d87fff640 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 0x559180e6eed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009c60 tx=0x7f3d78011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad714ada9c1031c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d8ed9d640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3d7c0a2230 0x7f3d7c0a46f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3d88032570 tx=0x7f3d880095c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d84ff9640 1 -- 172.21.15.116:0/4173204553 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78004230 con 0x559180e6cae0 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d84ff9640 1 -- 172.21.15.116:0/4173204553 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d78004390 con 0x559180e6cae0 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.374+0000 7f3d84ff9640 1 -- 172.21.15.116:0/4173204553 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78019600 con 0x559180e6cae0 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- 172.21.15.116:0/4173204553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 msgr2=0x559180e6eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 0x559180e6eed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009c60 tx=0x7f3d78011810 comp rx=0 tx=0).stop 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- 172.21.15.116:0/4173204553 shutdown_connections 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e6cae0 0x559180e6eed0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e63f00 0x559180e6c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- 172.21.15.116:0/4173204553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- 172.21.15.116:0/4173204553 >> 172.21.15.116:0/4173204553 conn(0x559180e5dd10 msgr2=0x559180e5e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- 172.21.15.116:0/4173204553 shutdown_connections 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- 172.21.15.116:0/4173204553 wait complete. 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- start start 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 Processor -- start 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e69b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 0x559180e6a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e6a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559180dc6520 con 0x559180e635e0 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559180a99430 con 0x559180e6cae0 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559180a55e00 con 0x559180e63f00 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.375+0000 7f3d979ba740 10 client.0 _ll_register_callbacks cb 0x559180d64d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e69b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 0x559180e6a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 0x559180e6a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:49880/0 (socket says 172.21.15.116:49880) 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d8e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e69b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35338/0 (socket says 172.21.15.116:35338) 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 -- 172.21.15.116:0/1644241189 learned_addr learned my addr 172.21.15.116:0/1644241189 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d8ed9d640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e6a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 -- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 msgr2=0x559180e6a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e6a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 -- 172.21.15.116:0/1644241189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 msgr2=0x559180e69b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e69b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 -- 172.21.15.116:0/1644241189 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d78009890 con 0x559180e63f00 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e69b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 0x559180e6a040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d78003f60 tx=0x7f3d780038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b77508f2eb6bb89d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.143:3300/0 2024-03-24T09:31:57.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1644241189 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78005240 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559180ac57e0 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1644241189 <== mon.1 v2:172.21.15.143:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3d780053a0 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1644241189 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3d78003cc0 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d979ba740 10 client.28045 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.376+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559180dc4ba0 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1644241189 <== mon.1 v2:172.21.15.143:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x559180ac57e0 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1644241189 <== mon.1 v2:172.21.15.143:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3d7801a440 con 0x559180e63f00 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d979ba740 10 client.28045 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d979ba740 10 client.28045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d979ba740 20 client.28045 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180e7ce20 0x559180e7f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d979ba740 4 client.28045 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559180b02720 con 0x559180e7ce20 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180e7ce20 0x559180e7f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d75ffb640 10 client.28045 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.377+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180e7ce20 0x559180e7f280 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c47fb8ec288f9244 server_cookie=3b05e4ef13db39ed in_seq=0 out_seq=0 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.378+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1644241189 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559180b02720 con 0x559180e7ce20 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.378+0000 7f3d75ffb640 10 client.28045 handle_command_reply: tid=0 2024-03-24T09:31:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.378+0000 7f3d979ba740 1 client.28045 shutdown 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 msgr2=0x559180e6a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 0x559180e6a040 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d78003f60 tx=0x7f3d780038a0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 shutdown_connections 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180e7ce20 0x559180e7f280 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e6a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d75ffb640 0 client.28045 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:57.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e63f00 0x559180e6a040 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1644241189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e635e0 0x559180e69b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 >> 172.21.15.116:0/1644241189 conn(0x559180e5dd10 msgr2=0x559180e5f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 shutdown_connections 2024-03-24T09:31:57.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 1 -- 172.21.15.116:0/1644241189 wait complete. 2024-03-24T09:31:57.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.379+0000 7f3d979ba740 20 client.28045 trim_cache size 0 max 16384 2024-03-24T09:31:57.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 Processor -- start 2024-03-24T09:31:57.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 -- start start 2024-03-24T09:31:57.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e6c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e78370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e84b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559180d4b430 con 0x559180e63f00 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559180e5ed10 con 0x559180e6cae0 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d979ba740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559180e4b4d0 con 0x559180e635e0 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e78370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e78370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35352/0 (socket says 172.21.15.116:35352) 2024-03-24T09:31:57.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 -- 172.21.15.116:0/1722743623 learned_addr learned my addr 172.21.15.116:0/1722743623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e6c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d8ed9d640 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e84b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 -- 172.21.15.116:0/1722743623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 msgr2=0x559180e6c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e6c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 -- 172.21.15.116:0/1722743623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 msgr2=0x559180e84b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e84b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.396+0000 7f3d87fff640 1 -- 172.21.15.116:0/1722743623 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559180d60b20 con 0x559180e63f00 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e78370 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3d7801a820 tx=0x7f3d78012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe88e533a0dfb442 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d77fff640 1 -- 172.21.15.116:0/1722743623 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78011dc0 con 0x559180e63f00 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d77fff640 1 -- 172.21.15.116:0/1722743623 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d78003910 con 0x559180e63f00 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d77fff640 1 -- 172.21.15.116:0/1722743623 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d780052a0 con 0x559180e63f00 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 -- 172.21.15.116:0/1722743623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 msgr2=0x559180e78370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e78370 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f3d7801a820 tx=0x7f3d78012f90 comp rx=0 tx=0).stop 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 -- 172.21.15.116:0/1722743623 shutdown_connections 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e84b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e78370 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1722743623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e6c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 -- 172.21.15.116:0/1722743623 >> 172.21.15.116:0/1722743623 conn(0x559180e5dd10 msgr2=0x559180e5e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 -- 172.21.15.116:0/1722743623 shutdown_connections 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.397+0000 7f3d979ba740 1 -- 172.21.15.116:0/1722743623 wait complete. 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 -- start start 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 Processor -- start 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e701a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e70c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559180e75170 con 0x559180e63f00 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559180ee21c0 con 0x559180e6cae0 2024-03-24T09:31:57.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x559180ee2300 con 0x559180e635e0 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d979ba740 10 client.0 _ll_register_callbacks cb 0x7f3d80005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d8ed9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e70c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e706e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35366/0 (socket says 172.21.15.116:35366) 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 -- 172.21.15.116:0/1729031892 learned_addr learned my addr 172.21.15.116:0/1729031892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d8e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 -- 172.21.15.116:0/1729031892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 msgr2=0x559180e701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 -- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 msgr2=0x559180e70c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e70c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.398+0000 7f3d87fff640 1 -- 172.21.15.116:0/1729031892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559180dc4ba0 con 0x559180e63f00 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d87fff640 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e706e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3d78011850 tx=0x7f3d7801a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74014558aedc1451 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d75ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1729031892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78003c90 con 0x559180e63f00 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d78004390 con 0x559180e63f00 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1729031892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3d78000e40 con 0x559180e63f00 2024-03-24T09:31:57.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1729031892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3d78003df0 con 0x559180e63f00 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 10 client.18375 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559180d86700 con 0x559180e63f00 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1729031892 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3d78004390 con 0x559180e63f00 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1729031892 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3d78004500 con 0x559180e63f00 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 10 client.18375 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 10 client.18375 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 20 client.18375 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180ee3740 0x559180ee5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 4 client.18375 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.399+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559180b51890 con 0x559180ee3740 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180ee3740 0x559180ee5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d75ffb640 10 client.18375 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d8e59c640 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180ee3740 0x559180ee5b60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a89fc957823ab3f3 server_cookie=51f96d42d504390d in_seq=0 out_seq=0 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d75ffb640 1 -- 172.21.15.116:0/1729031892 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559180b51890 con 0x559180ee3740 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d75ffb640 10 client.18375 handle_command_reply: tid=0 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 client.18375 shutdown 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 msgr2=0x559180e706e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e706e0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3d78011850 tx=0x7f3d7801a6f0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 shutdown_connections 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x559180ee3740 0x559180ee5b60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559180e6cae0 0x559180e70c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d75ffb640 0 client.18375 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559180e63f00 0x559180e706e0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 --2- 172.21.15.116:0/1729031892 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x559180e635e0 0x559180e701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 >> 172.21.15.116:0/1729031892 conn(0x559180e5dd10 msgr2=0x559180e582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 shutdown_connections 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 1 -- 172.21.15.116:0/1729031892 wait complete. 2024-03-24T09:31:57.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.400+0000 7f3d979ba740 20 client.18375 trim_cache size 0 max 16384 2024-03-24T09:31:57.411 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:31:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:31:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:31:57.411 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3d7c0a2230 msgr2=0x7f3d7c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3d7c0a2230 0x7f3d7c0a46f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3d88032570 tx=0x7f3d880095c0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100630 msgr2=0x7f3d9019d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100630 0x7f3d9019d8a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d8000e9f0 tx=0x7f3d8000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 shutdown_connections 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d90106640 0x7f3d90197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d90100ff0 0x7f3d9019dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3d7c0a2230 0x7f3d7c0a46f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 --2- 172.21.15.116:0/838298651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3d90100630 0x7f3d9019d8a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 >> 172.21.15.116:0/838298651 conn(0x7f3d90077770 msgr2=0x7f3d900ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 shutdown_connections 2024-03-24T09:31:57.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.401+0000 7f3d94dd7640 1 -- 172.21.15.116:0/838298651 wait complete. 2024-03-24T09:31:57.414 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:31:57.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:31:57.438 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:57 smithi116 ceph-mon[32207]: pgmap v304: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 -- 172.21.15.116:0/1836373204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 msgr2=0x7fd56010d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 --2- 172.21.15.116:0/1836373204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56010d450 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fd558009ca0 tx=0x7fd55802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 -- 172.21.15.116:0/1836373204 shutdown_connections 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 --2- 172.21.15.116:0/1836373204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56010d450 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 --2- 172.21.15.116:0/1836373204 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd5601005d0 0x7fd560100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 --2- 172.21.15.116:0/1836373204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd560106690 0x7fd560106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 -- 172.21.15.116:0/1836373204 >> 172.21.15.116:0/1836373204 conn(0x7fd560076990 msgr2=0x7fd560076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 -- 172.21.15.116:0/1836373204 shutdown_connections 2024-03-24T09:31:57.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.516+0000 7fd5662ae640 1 -- 172.21.15.116:0/1836373204 wait complete. 2024-03-24T09:31:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 Processor -- start 2024-03-24T09:31:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 -- start start 2024-03-24T09:31:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5601005d0 0x7fd56019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd560106690 0x7fd5601979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd560198040 con 0x7fd560100f90 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5601981b0 con 0x7fd5601005d0 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd5662ae640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd560198370 con 0x7fd560106690 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd55f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd55f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56019dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35384/0 (socket says 172.21.15.116:35384) 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd55f7fe640 1 -- 172.21.15.116:0/4177099548 learned_addr learned my addr 172.21.15.116:0/4177099548 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd55ffff640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5601005d0 0x7fd56019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.517+0000 7fd56484b640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd560106690 0x7fd5601979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55f7fe640 1 -- 172.21.15.116:0/4177099548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd560106690 msgr2=0x7fd5601979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55f7fe640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd560106690 0x7fd5601979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55f7fe640 1 -- 172.21.15.116:0/4177099548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5601005d0 msgr2=0x7fd56019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55f7fe640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5601005d0 0x7fd56019d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55f7fe640 1 -- 172.21.15.116:0/4177099548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd558009900 con 0x7fd560100f90 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55ffff640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5601005d0 0x7fd56019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55f7fe640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56019dd30 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fd55400e9f0 tx=0x7fd55400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554009720 con 0x7fd560100f90 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd560198650 con 0x7fd560100f90 2024-03-24T09:31:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd55400cd80 con 0x7fd560100f90 2024-03-24T09:31:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554010430 con 0x7fd560100f90 2024-03-24T09:31:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.518+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5601a4890 con 0x7fd560100f90 2024-03-24T09:31:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.519+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd560101cf0 con 0x7fd560100f90 2024-03-24T09:31:57.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.520+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd554005020 con 0x7fd560100f90 2024-03-24T09:31:57.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.521+0000 7fd55d7fa640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5480a2230 0x7fd5480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.521+0000 7fd55ffff640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5480a2230 0x7fd5480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.521+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd554014070 con 0x7fd560100f90 2024-03-24T09:31:57.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.521+0000 7fd55ffff640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5480a2230 0x7fd5480a46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd560105760 tx=0x7fd550005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:57 smithi143 ceph-mon[35965]: pgmap v304: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:57.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.525+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd554097050 con 0x7fd560100f90 2024-03-24T09:31:57.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:57 smithi117 ceph-mon[35788]: pgmap v304: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:31:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.735+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd560068950 con 0x7fd560100f90 2024-03-24T09:31:57.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.736+0000 7fd55d7fa640 1 -- 172.21.15.116:0/4177099548 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7fd55409a710 con 0x7fd560100f90 2024-03-24T09:31:57.737 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:31:57.737 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:31:57.737 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:31:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5480a2230 msgr2=0x7fd5480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5480a2230 0x7fd5480a46f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd560105760 tx=0x7fd550005f90 comp rx=0 tx=0).stop 2024-03-24T09:31:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 msgr2=0x7fd56019dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56019dd30 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fd55400e9f0 tx=0x7fd55400edf0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 shutdown_connections 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd5480a2230 0x7fd5480a46f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd560106690 0x7fd5601979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd560100f90 0x7fd56019dd30 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 --2- 172.21.15.116:0/4177099548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5601005d0 0x7fd56019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.738+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 >> 172.21.15.116:0/4177099548 conn(0x7fd560076990 msgr2=0x7fd5600fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.739+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 shutdown_connections 2024-03-24T09:31:57.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.739+0000 7fd5662ae640 1 -- 172.21.15.116:0/4177099548 wait complete. 2024-03-24T09:31:57.750 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:31:57.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 -- 172.21.15.116:0/3259753570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a201016a0 msgr2=0x7f3a20101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 --2- 172.21.15.116:0/3259753570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a201016a0 0x7f3a20101b20 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009930 tx=0x7f3a1002bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 -- 172.21.15.116:0/3259753570 shutdown_connections 2024-03-24T09:31:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 --2- 172.21.15.116:0/3259753570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a20102060 0x7f3a2010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 --2- 172.21.15.116:0/3259753570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a201016a0 0x7f3a20101b20 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 --2- 172.21.15.116:0/3259753570 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a201076b0 0x7f3a20107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 -- 172.21.15.116:0/3259753570 >> 172.21.15.116:0/3259753570 conn(0x7f3a20078b10 msgr2=0x7f3a20078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 -- 172.21.15.116:0/3259753570 shutdown_connections 2024-03-24T09:31:57.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 -- 172.21.15.116:0/3259753570 wait complete. 2024-03-24T09:31:57.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 Processor -- start 2024-03-24T09:31:57.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.893+0000 7f3a2762e640 1 -- start start 2024-03-24T09:31:57.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a201016a0 0x7f3a2019d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 0x7f3a2019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a201076b0 0x7f3a20197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a201980e0 con 0x7f3a20102060 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20198250 con 0x7f3a201016a0 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3a20198410 con 0x7f3a201076b0 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 0x7f3a2019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a201016a0 0x7f3a2019d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 0x7f3a2019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35410/0 (socket says 172.21.15.116:35410) 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 -- 172.21.15.116:0/434178180 learned_addr learned my addr 172.21.15.116:0/434178180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a201076b0 0x7f3a20197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 -- 172.21.15.116:0/434178180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a201076b0 msgr2=0x7f3a20197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a201076b0 0x7f3a20197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 -- 172.21.15.116:0/434178180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a201016a0 msgr2=0x7f3a2019d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a201016a0 0x7f3a2019d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 -- 172.21.15.116:0/434178180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a10009560 con 0x7f3a20102060 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 0x7f3a2019de20 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009230 tx=0x7f3a10005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a167fc640 1 -- 172.21.15.116:0/434178180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10004350 con 0x7f3a20102060 2024-03-24T09:31:57.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a20198690 con 0x7f3a20102060 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a167fc640 1 -- 172.21.15.116:0/434178180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a1002de90 con 0x7f3a20102060 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a167fc640 1 -- 172.21.15.116:0/434178180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10005220 con 0x7f3a20102060 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.894+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a201a4980 con 0x7f3a20102060 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.897+0000 7f3a167fc640 1 -- 172.21.15.116:0/434178180 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f3a10003c30 con 0x7f3a20102060 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.897+0000 7f3a2a215740 1 Processor -- start 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.897+0000 7f3a2a215740 1 -- start start 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a167fc640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3a0c0a2230 0x7f3a0c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a063d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0648ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a2a215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5642a05ab840 con 0x5642a063d3f0 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a2a215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5642a05a8d70 con 0x5642a06468f0 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a2a215740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5642a05a4550 con 0x5642a063dd10 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a063d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3a0c0a2230 0x7f3a0c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a063d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35422/0 (socket says 172.21.15.116:35422) 2024-03-24T09:31:57.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a253ca640 1 -- 172.21.15.116:0/1951795405 learned_addr learned my addr 172.21.15.116:0/1951795405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a167fc640 1 -- 172.21.15.116:0/434178180 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3a100f0340 con 0x7f3a20102060 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0648ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 -- 172.21.15.116:0/1951795405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 msgr2=0x5642a0648ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0648ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 -- 172.21.15.116:0/1951795405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 msgr2=0x5642a063d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a063d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 -- 172.21.15.116:0/1951795405 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642a0539de0 con 0x5642a063dd10 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a253ca640 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a063d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:57.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3a0c0a2230 0x7f3a0c0a46f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a18005fd0 tx=0x7f3a18009e60 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06463b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a100ed730 tx=0x7f3a100bc5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f9cefe9923c2f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a0b7fe640 1 -- 172.21.15.116:0/1951795405 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a10079070 con 0x5642a063dd10 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a0b7fe640 1 -- 172.21.15.116:0/1951795405 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a100dc040 con 0x5642a063dd10 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.898+0000 7f3a0b7fe640 1 -- 172.21.15.116:0/1951795405 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a100bc9d0 con 0x5642a063dd10 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 -- 172.21.15.116:0/1951795405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 msgr2=0x5642a06463b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06463b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3a100ed730 tx=0x7f3a100bc5d0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 -- 172.21.15.116:0/1951795405 shutdown_connections 2024-03-24T09:31:57.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0648ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06463b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1951795405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a063d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 -- 172.21.15.116:0/1951795405 >> 172.21.15.116:0/1951795405 conn(0x5642a0637b20 msgr2=0x5642a0637f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 -- 172.21.15.116:0/1951795405 shutdown_connections 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 -- 172.21.15.116:0/1951795405 wait complete. 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.899+0000 7f3a2a215740 1 -- start start 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 Processor -- start 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06439b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a0643ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0644430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5642a0277430 con 0x5642a063d3f0 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5642a0233e00 con 0x5642a06468f0 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5642a0529590 con 0x5642a063dd10 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a2a215740 10 client.0 _ll_register_callbacks cb 0x5642a0598fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35430/0 (socket says 172.21.15.116:35430) 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 -- 172.21.15.116:0/1260560884 learned_addr learned my addr 172.21.15.116:0/1260560884 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a0643ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0644430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 -- 172.21.15.116:0/1260560884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 msgr2=0x5642a0643ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a0643ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 -- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 msgr2=0x5642a0644430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0644430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 -- 172.21.15.116:0/1260560884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a201a4980 con 0x5642a063d3f0 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a253ca640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06439b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c012720 tx=0x7f3a1c012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d42512f24bca96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.900+0000 7f3a08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a08ff9640 1 -- 172.21.15.116:0/1260560884 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c017070 con 0x5642a063d3f0 2024-03-24T09:31:57.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a08ff9640 1 -- 172.21.15.116:0/1260560884 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3a1c004050 con 0x5642a063d3f0 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5642a02a37e0 con 0x5642a063d3f0 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a08ff9640 1 -- 172.21.15.116:0/1260560884 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3a1c004210 con 0x5642a063d3f0 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 10 client.18393 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5642a05a2bd0 con 0x5642a063d3f0 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a08ff9640 1 -- 172.21.15.116:0/1260560884 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x5642a02a37e0 con 0x5642a063d3f0 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a08ff9640 1 -- 172.21.15.116:0/1260560884 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3a1c003990 con 0x5642a063d3f0 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 10 client.18393 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 10 client.18393 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:31:57.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 20 client.18393 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06568c0 0x5642a0658ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 4 client.18393 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642a02e0720 con 0x5642a06568c0 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.901+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06568c0 0x5642a0658ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a08ff9640 10 client.18393 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06568c0 0x5642a0658ce0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3bef8607e4d39717 server_cookie=60ae4fe4a546b0f7 in_seq=0 out_seq=0 2024-03-24T09:31:57.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a08ff9640 1 -- 172.21.15.116:0/1260560884 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642a02e0720 con 0x5642a06568c0 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a08ff9640 10 client.18393 handle_command_reply: tid=0 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 client.18393 shutdown 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 msgr2=0x5642a06439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06439b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c012720 tx=0x7f3a1c012be0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 shutdown_connections 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a0644430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a0643ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06568c0 0x5642a0658ce0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 --2- 172.21.15.116:0/1260560884 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06439b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a08ff9640 0 client.18393 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:57.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.902+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 >> 172.21.15.116:0/1260560884 conn(0x5642a0637b20 msgr2=0x5642a0639870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.903+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 shutdown_connections 2024-03-24T09:31:57.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.903+0000 7f3a2a215740 1 -- 172.21.15.116:0/1260560884 wait complete. 2024-03-24T09:31:57.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.903+0000 7f3a2a215740 20 client.18393 trim_cache size 0 max 16384 2024-03-24T09:31:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 Processor -- start 2024-03-24T09:31:57.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 -- start start 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a0651f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a063dd10 0x5642a0644250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a06468f0 0x5642a065e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5642a063cd40 con 0x5642a063d3f0 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5642a0633210 con 0x5642a063dd10 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a2a215740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5642a06373d0 con 0x5642a06468f0 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a0651f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a0651f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35442/0 (socket says 172.21.15.116:35442) 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a24bc9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a063dd10 0x5642a0644250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 -- 172.21.15.116:0/216526502 learned_addr learned my addr 172.21.15.116:0/216526502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a06468f0 0x5642a065e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 -- 172.21.15.116:0/216526502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a06468f0 msgr2=0x5642a065e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a06468f0 0x5642a065e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 -- 172.21.15.116:0/216526502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a063dd10 msgr2=0x5642a0644250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a063dd10 0x5642a0644250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 -- 172.21.15.116:0/216526502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642a053ec80 con 0x5642a063d3f0 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a253ca640 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a0651f40 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c003f10 tx=0x7f3a1c009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af365537736ba96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a0affd640 1 -- 172.21.15.116:0/216526502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c017070 con 0x5642a063d3f0 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a0affd640 1 -- 172.21.15.116:0/216526502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a1c003aa0 con 0x5642a063d3f0 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.916+0000 7f3a0affd640 1 -- 172.21.15.116:0/216526502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c00fcb0 con 0x5642a063d3f0 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 -- 172.21.15.116:0/216526502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 msgr2=0x5642a0651f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a0651f40 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c003f10 tx=0x7f3a1c009f40 comp rx=0 tx=0).stop 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 -- 172.21.15.116:0/216526502 shutdown_connections 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a06468f0 0x5642a065e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a063dd10 0x5642a0644250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 --2- 172.21.15.116:0/216526502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a0651f40 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 -- 172.21.15.116:0/216526502 >> 172.21.15.116:0/216526502 conn(0x5642a0637b20 msgr2=0x5642a0636cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 -- 172.21.15.116:0/216526502 shutdown_connections 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 -- 172.21.15.116:0/216526502 wait complete. 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 -- start start 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.917+0000 7f3a2a215740 1 Processor -- start 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06b6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a064a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5642a064ac20 con 0x5642a063d3f0 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5642a064ad90 con 0x5642a06468f0 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5642a064af50 con 0x5642a063dd10 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 10 client.0 _ll_register_callbacks cb 0x7f3a1c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06b5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35456/0 (socket says 172.21.15.116:35456) 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 -- 172.21.15.116:0/3236312853 learned_addr learned my addr 172.21.15.116:0/3236312853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a25bcb640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a064a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06b6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 -- 172.21.15.116:0/3236312853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 msgr2=0x5642a06b6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06b6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 -- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 msgr2=0x5642a064a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a064a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 -- 172.21.15.116:0/3236312853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c021040 con 0x5642a063d3f0 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a253ca640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06b5f40 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c003d20 tx=0x7f3a1c0106c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e9779351ad02d78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a08ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a08ff9640 1 -- 172.21.15.116:0/3236312853 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c017070 con 0x5642a063d3f0 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a08ff9640 1 -- 172.21.15.116:0/3236312853 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f3a1c000e40 con 0x5642a063d3f0 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.918+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a100dc040 con 0x5642a063d3f0 2024-03-24T09:31:57.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a08ff9640 1 -- 172.21.15.116:0/3236312853 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3a1c00f920 con 0x5642a063d3f0 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 10 client.18399 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5642a0564920 con 0x5642a063d3f0 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a08ff9640 1 -- 172.21.15.116:0/3236312853 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3a100dc040 con 0x5642a063d3f0 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a08ff9640 1 -- 172.21.15.116:0/3236312853 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f3a1c031030 con 0x5642a063d3f0 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 10 client.18399 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 10 client.18399 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 20 client.18399 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06bda90 0x5642a06bfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:57.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 4 client.18399 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642a032f890 con 0x5642a06bda90 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06bda90 0x5642a06bfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a24bc9640 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06bda90 0x5642a06bfe80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=228f334544a1c7b1 server_cookie=189f66802f87bd8e in_seq=0 out_seq=0 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a08ff9640 10 client.18399 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a08ff9640 1 -- 172.21.15.116:0/3236312853 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642a032f890 con 0x5642a06bda90 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.919+0000 7f3a08ff9640 10 client.18399 handle_command_reply: tid=0 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 client.18399 shutdown 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 msgr2=0x5642a06b5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06b5f40 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c003d20 tx=0x7f3a1c0106c0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 shutdown_connections 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5642a06468f0 0x5642a064a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x5642a063dd10 0x5642a06b6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x5642a06bda90 0x5642a06bfe80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 --2- 172.21.15.116:0/3236312853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5642a063d3f0 0x5642a06b5f40 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a08ff9640 0 client.18399 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 >> 172.21.15.116:0/3236312853 conn(0x5642a0637b20 msgr2=0x5642a0638ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 shutdown_connections 2024-03-24T09:31:57.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 1 -- 172.21.15.116:0/3236312853 wait complete. 2024-03-24T09:31:57.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2a215740 20 client.18399 trim_cache size 0 max 16384 2024-03-24T09:31:57.943 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:57.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3a0c0a2230 msgr2=0x7f3a0c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3a0c0a2230 0x7f3a0c0a46f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a18005fd0 tx=0x7f3a18009e60 comp rx=0 tx=0).stop 2024-03-24T09:31:57.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 msgr2=0x7f3a2019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:57.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 0x7f3a2019de20 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3a10009230 tx=0x7f3a10005dd0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 shutdown_connections 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3a0c0a2230 0x7f3a0c0a46f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a201076b0 0x7f3a20197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a20102060 0x7f3a2019de20 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 --2- 172.21.15.116:0/434178180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a201016a0 0x7f3a2019d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 >> 172.21.15.116:0/434178180 conn(0x7f3a20078b10 msgr2=0x7f3a20104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 shutdown_connections 2024-03-24T09:31:57.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:57.920+0000 7f3a2762e640 1 -- 172.21.15.116:0/434178180 wait complete. 2024-03-24T09:31:57.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:31:58.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 -- 172.21.15.116:0/3242305637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 msgr2=0x7f13a0068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/3242305637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a0068d90 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f1388009930 tx=0x7f138802bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:58.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 -- 172.21.15.116:0/3242305637 shutdown_connections 2024-03-24T09:31:58.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/3242305637 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a00692d0 0x7f13a010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/3242305637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a0068d90 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/3242305637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a0107760 0x7f13a0107b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 -- 172.21.15.116:0/3242305637 >> 172.21.15.116:0/3242305637 conn(0x7f13a0077710 msgr2=0x7f13a0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 -- 172.21.15.116:0/3242305637 shutdown_connections 2024-03-24T09:31:58.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.049+0000 7f13a4cd0640 1 -- 172.21.15.116:0/3242305637 wait complete. 2024-03-24T09:31:58.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 Processor -- start 2024-03-24T09:31:58.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 -- start start 2024-03-24T09:31:58.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a00692d0 0x7f13a019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a0107760 0x7f13a0197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13a01980f0 con 0x7f13a0068910 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13a0198260 con 0x7f13a0107760 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f13a4cd0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f13a0198420 con 0x7f13a00692d0 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f139e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f139e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a019d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35486/0 (socket says 172.21.15.116:35486) 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f139e59c640 1 -- 172.21.15.116:0/929631893 learned_addr learned my addr 172.21.15.116:0/929631893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f139ed9d640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a0107760 0x7f13a0197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.050+0000 7f1397fff640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a00692d0 0x7f13a019dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f139e59c640 1 -- 172.21.15.116:0/929631893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a00692d0 msgr2=0x7f13a019dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f139e59c640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a00692d0 0x7f13a019dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f139e59c640 1 -- 172.21.15.116:0/929631893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a0107760 msgr2=0x7f13a0197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f139e59c640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a0107760 0x7f13a0197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f139e59c640 1 -- 172.21.15.116:0/929631893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1388009560 con 0x7f13a0068910 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f139e59c640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a019d8a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f139000e9f0 tx=0x7f139000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f13977fe640 1 -- 172.21.15.116:0/929631893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390009700 con 0x7f13a0068910 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f13977fe640 1 -- 172.21.15.116:0/929631893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f139000cd80 con 0x7f13a0068910 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a0198700 con 0x7f13a0068910 2024-03-24T09:31:58.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f13977fe640 1 -- 172.21.15.116:0/929631893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390018840 con 0x7f13a0068910 2024-03-24T09:31:58.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.051+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a01a4940 con 0x7f13a0068910 2024-03-24T09:31:58.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13977fe640 1 -- 172.21.15.116:0/929631893 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f13900189a0 con 0x7f13a0068910 2024-03-24T09:31:58.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 Processor -- start 2024-03-24T09:31:58.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 -- start start 2024-03-24T09:31:58.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab0977d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab097d10 0x561aab0a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 0x561aab0a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561aab005840 con 0x561aab0a08f0 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561aab002d70 con 0x561aab0973f0 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13a7981740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561aaaffe550 con 0x561aab097d10 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f13977fe640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f138c0a2230 0x7f138c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 0x561aab0a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 0x561aab0a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35494/0 (socket says 172.21.15.116:35494) 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f139e59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab097d10 0x561aab0a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f1397fff640 1 -- 172.21.15.116:0/1691125105 learned_addr learned my addr 172.21.15.116:0/1691125105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.054+0000 7f139ed9d640 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab0977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f139ed9d640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f138c0a2230 0x7f138c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f13977fe640 1 -- 172.21.15.116:0/929631893 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f1390014070 con 0x7f13a0068910 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1397fff640 1 -- 172.21.15.116:0/1691125105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab097d10 msgr2=0x561aab0a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1397fff640 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab097d10 0x561aab0a03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1397fff640 1 -- 172.21.15.116:0/1691125105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 msgr2=0x561aab0977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1397fff640 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab0977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1397fff640 1 -- 172.21.15.116:0/1691125105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aaaf93de0 con 0x561aab0a08f0 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f139ed9d640 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab0977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1397fff640 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 0x561aab0a2ce0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f1388036040 tx=0x7f1388005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c41708fa786a573 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f139ed9d640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f138c0a2230 0x7f138c0a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1398005c40 tx=0x7f1398009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1394ff9640 1 -- 172.21.15.116:0/1691125105 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388004740 con 0x561aab0a08f0 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1394ff9640 1 -- 172.21.15.116:0/1691125105 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f13880048a0 con 0x561aab0a08f0 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.055+0000 7f1394ff9640 1 -- 172.21.15.116:0/1691125105 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388003850 con 0x561aab0a08f0 2024-03-24T09:31:58.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 -- 172.21.15.116:0/1691125105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 msgr2=0x561aab0a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 0x561aab0a2ce0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f1388036040 tx=0x7f1388005d50 comp rx=0 tx=0).stop 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 -- 172.21.15.116:0/1691125105 shutdown_connections 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab0a08f0 0x561aab0a2ce0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab097d10 0x561aab0a03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 --2- 172.21.15.116:0/1691125105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab0977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 -- 172.21.15.116:0/1691125105 >> 172.21.15.116:0/1691125105 conn(0x561aab091b20 msgr2=0x561aab091f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 -- 172.21.15.116:0/1691125105 shutdown_connections 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 -- 172.21.15.116:0/1691125105 wait complete. 2024-03-24T09:31:58.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 -- start start 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.056+0000 7f13a7981740 1 Processor -- start 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab09d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab09de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0a08f0 0x561aab09e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561aaacd1430 con 0x561aab097d10 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561aaac8de00 con 0x561aab0973f0 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561aaaf83590 con 0x561aab0a08f0 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 10 client.0 _ll_register_callbacks cb 0x561aaaff2fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab09de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab09de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35500/0 (socket says 172.21.15.116:35500) 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 -- 172.21.15.116:0/696694705 learned_addr learned my addr 172.21.15.116:0/696694705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f139e59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab09d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f139ed9d640 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0a08f0 0x561aab09e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 -- 172.21.15.116:0/696694705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0a08f0 msgr2=0x561aab09e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0a08f0 0x561aab09e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 -- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 msgr2=0x561aab09d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab09d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 -- 172.21.15.116:0/696694705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a010ee70 con 0x561aab097d10 2024-03-24T09:31:58.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1397fff640 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab09de10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f138802c170 tx=0x7f13880045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a93cf9d3235b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1385ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1385ffb640 1 -- 172.21.15.116:0/696694705 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880051d0 con 0x561aab097d10 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1385ffb640 1 -- 172.21.15.116:0/696694705 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f138804d070 con 0x561aab097d10 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561aaacfd7e0 con 0x561aab097d10 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f1385ffb640 1 -- 172.21.15.116:0/696694705 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f1388005330 con 0x561aab097d10 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 10 client.18411 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.057+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561aaaffcbd0 con 0x561aab097d10 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f1385ffb640 1 -- 172.21.15.116:0/696694705 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561aaacfd7e0 con 0x561aab097d10 2024-03-24T09:31:58.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f1385ffb640 1 -- 172.21.15.116:0/696694705 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f138803c050 con 0x561aab097d10 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f13a7981740 10 client.18411 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f13a7981740 10 client.18411 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f13a7981740 20 client.18411 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f13a7981740 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab0b0570 0x561aab0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f13a7981740 4 client.18411 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aaad3a720 con 0x561aab0b0570 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f139e59c640 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab0b0570 0x561aab0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f1385ffb640 10 client.18411 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.058+0000 7f139e59c640 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab0b0570 0x561aab0b29d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4dcdf1be603db2f9 server_cookie=fd78b09f07484118 in_seq=0 out_seq=0 2024-03-24T09:31:58.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f1385ffb640 1 -- 172.21.15.116:0/696694705 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561aaad3a720 con 0x561aab0b0570 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f1385ffb640 10 client.18411 handle_command_reply: tid=0 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 client.18411 shutdown 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 msgr2=0x561aab09de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab09de10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f138802c170 tx=0x7f13880045b0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 shutdown_connections 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab0b0570 0x561aab0b29d0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0a08f0 0x561aab09e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f1385ffb640 0 client.18411 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:58.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab09de10 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 --2- 172.21.15.116:0/696694705 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0973f0 0x561aab09d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 >> 172.21.15.116:0/696694705 conn(0x561aab091b20 msgr2=0x561aab093870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 shutdown_connections 2024-03-24T09:31:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 1 -- 172.21.15.116:0/696694705 wait complete. 2024-03-24T09:31:58.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.059+0000 7f13a7981740 20 client.18411 trim_cache size 0 max 16384 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.072+0000 7f13a7981740 1 Processor -- start 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.072+0000 7f13a7981740 1 -- start start 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f13a7981740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0ac120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f13a7981740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0b82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f13a7981740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f13a7981740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561aab0a32a0 con 0x561aab097d10 2024-03-24T09:31:58.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f13a7981740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561aab097050 con 0x561aab0a08f0 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f13a7981740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561aab0926f0 con 0x561aab0973f0 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0b82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0b82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35502/0 (socket says 172.21.15.116:35502) 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 -- 172.21.15.116:0/3289704401 learned_addr learned my addr 172.21.15.116:0/3289704401 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f139e59c640 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f139ed9d640 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 -- 172.21.15.116:0/3289704401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 msgr2=0x561aab0ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0ac120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 -- 172.21.15.116:0/3289704401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 msgr2=0x561aab0a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 -- 172.21.15.116:0/3289704401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aaaf98c80 con 0x561aab097d10 2024-03-24T09:31:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1397fff640 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0b82e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f138802bc70 tx=0x7f1388000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b85a10b16115e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1387fff640 1 -- 172.21.15.116:0/3289704401 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388034640 con 0x561aab097d10 2024-03-24T09:31:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1387fff640 1 -- 172.21.15.116:0/3289704401 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f138802c510 con 0x561aab097d10 2024-03-24T09:31:58.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.073+0000 7f1387fff640 1 -- 172.21.15.116:0/3289704401 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388005400 con 0x561aab097d10 2024-03-24T09:31:58.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- 172.21.15.116:0/3289704401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 msgr2=0x561aab0b82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0b82e0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f138802bc70 tx=0x7f1388000bc0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- 172.21.15.116:0/3289704401 shutdown_connections 2024-03-24T09:31:58.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0b82e0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- 172.21.15.116:0/3289704401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- 172.21.15.116:0/3289704401 >> 172.21.15.116:0/3289704401 conn(0x561aab091b20 msgr2=0x561aab09d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- 172.21.15.116:0/3289704401 shutdown_connections 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- 172.21.15.116:0/3289704401 wait complete. 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- start start 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 Processor -- start 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0a4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0a4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561aab115ad0 con 0x561aab097d10 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561aab115c40 con 0x561aab0a08f0 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.074+0000 7f13a7981740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x561aab115e00 con 0x561aab0973f0 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f13a7981740 10 client.0 _ll_register_callbacks cb 0x7f139009ac50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0a4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35510/0 (socket says 172.21.15.116:35510) 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 -- 172.21.15.116:0/2782607441 learned_addr learned my addr 172.21.15.116:0/2782607441 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f139e59c640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0a4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f139ed9d640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 -- 172.21.15.116:0/2782607441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 msgr2=0x561aab0a4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0a4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 -- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 msgr2=0x561aab0a4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 -- 172.21.15.116:0/2782607441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aaaffcbd0 con 0x561aab097d10 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1397fff640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0a4880 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1388005c40 tx=0x7f13880053f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51e15ffdff77509 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1385ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1385ffb640 1 -- 172.21.15.116:0/2782607441 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388004220 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1385ffb640 1 -- 172.21.15.116:0/2782607441 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f1388000e40 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561aab09d700 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1385ffb640 1 -- 172.21.15.116:0/2782607441 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f13880046c0 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f13a7981740 10 client.18417 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561aaafbe920 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.075+0000 7f1385ffb640 1 -- 172.21.15.116:0/2782607441 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x561aab09d700 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f1385ffb640 1 -- 172.21.15.116:0/2782607441 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f138803c070 con 0x561aab097d10 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 10 client.18417 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 10 client.18417 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:31:58.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 20 client.18417 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab1170f0 0x561aab1194e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 4 client.18417 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f13880048a0 con 0x561aab1170f0 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f139e59c640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab1170f0 0x561aab1194e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f139e59c640 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab1170f0 0x561aab1194e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=45dc4cffed169042 server_cookie=2719fec09ae0d08d in_seq=0 out_seq=0 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f1385ffb640 10 client.18417 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f1385ffb640 1 -- 172.21.15.116:0/2782607441 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f13880048a0 con 0x561aab1170f0 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f1385ffb640 10 client.18417 handle_command_reply: tid=0 2024-03-24T09:31:58.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 client.18417 shutdown 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 msgr2=0x561aab0a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0a4880 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f1388005c40 tx=0x7f13880053f0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 shutdown_connections 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x561aab1170f0 0x561aab1194e0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561aab0a08f0 0x561aab0a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aab097d10 0x561aab0a4880 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f1385ffb640 0 client.18417 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:31:58.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 --2- 172.21.15.116:0/2782607441 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x561aab0973f0 0x561aab0a4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 >> 172.21.15.116:0/2782607441 conn(0x561aab091b20 msgr2=0x561aab09e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.076+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 shutdown_connections 2024-03-24T09:31:58.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a7981740 1 -- 172.21.15.116:0/2782607441 wait complete. 2024-03-24T09:31:58.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a7981740 20 client.18417 trim_cache size 0 max 16384 2024-03-24T09:31:58.087 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:58.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f138c0a2230 msgr2=0x7f138c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f138c0a2230 0x7f138c0a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1398005c40 tx=0x7f1398009500 comp rx=0 tx=0).stop 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 msgr2=0x7f13a019d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a019d8a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f139000e9f0 tx=0x7f139000eeb0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 shutdown_connections 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13a0107760 0x7f13a0197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f138c0a2230 0x7f138c0a46f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13a00692d0 0x7f13a019dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 --2- 172.21.15.116:0/929631893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13a0068910 0x7f13a019d8a0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 >> 172.21.15.116:0/929631893 conn(0x7f13a0077710 msgr2=0x7f13a00fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 shutdown_connections 2024-03-24T09:31:58.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.077+0000 7f13a4cd0640 1 -- 172.21.15.116:0/929631893 wait complete. 2024-03-24T09:31:58.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:31:58.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- 172.21.15.116:0/3537811767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41016a0 msgr2=0x7f77a4101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- 172.21.15.116:0/3537811767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41016a0 0x7f77a4101b20 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f7790009930 tx=0x7f779002bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- 172.21.15.116:0/3537811767 shutdown_connections 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- 172.21.15.116:0/3537811767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a4102060 0x7f77a410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- 172.21.15.116:0/3537811767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41016a0 0x7f77a4101b20 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- 172.21.15.116:0/3537811767 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77a41076b0 0x7f77a4107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- 172.21.15.116:0/3537811767 >> 172.21.15.116:0/3537811767 conn(0x7f77a40fd1b0 msgr2=0x7f77a40ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- 172.21.15.116:0/3537811767 shutdown_connections 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- 172.21.15.116:0/3537811767 wait complete. 2024-03-24T09:31:58.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 Processor -- start 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- start start 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77a41016a0 0x7f77a419da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a4102060 0x7f77a419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 0x7f77a4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f77a41982a0 con 0x7f77a41076b0 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f77a4198410 con 0x7f77a4102060 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.195+0000 7f77aab84640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f77a41985d0 con 0x7f77a41016a0 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 0x7f77a4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 0x7f77a4197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35528/0 (socket says 172.21.15.116:35528) 2024-03-24T09:31:58.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 -- 172.21.15.116:0/3726603514 learned_addr learned my addr 172.21.15.116:0/3726603514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a3fff640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a4102060 0x7f77a419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a8920640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77a41016a0 0x7f77a419da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 -- 172.21.15.116:0/3726603514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77a41016a0 msgr2=0x7f77a419da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77a41016a0 0x7f77a419da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 -- 172.21.15.116:0/3726603514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a4102060 msgr2=0x7f77a419df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a4102060 0x7f77a419df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 -- 172.21.15.116:0/3726603514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779c009840 con 0x7f77a41076b0 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a9121640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 0x7f77a4197b10 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f779c00efd0 tx=0x7f779c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a1ffb640 1 -- 172.21.15.116:0/3726603514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c009b40 con 0x7f77a41076b0 2024-03-24T09:31:58.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7790009560 con 0x7f77a41076b0 2024-03-24T09:31:58.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a1ffb640 1 -- 172.21.15.116:0/3726603514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f779c003b70 con 0x7f77a41076b0 2024-03-24T09:31:58.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77a1ffb640 1 -- 172.21.15.116:0/3726603514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0107e0 con 0x7f77a41076b0 2024-03-24T09:31:58.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.196+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a41a49d0 con 0x7f77a41076b0 2024-03-24T09:31:58.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a1ffb640 1 -- 172.21.15.116:0/3726603514 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f779c003ce0 con 0x7f77a41076b0 2024-03-24T09:31:58.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 Processor -- start 2024-03-24T09:31:58.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 -- start start 2024-03-24T09:31:58.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a1ffb640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f77940a2230 0x7f77940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a3fff640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f77940a2230 0x7f77940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516ef480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a1ffb640 1 -- 172.21.15.116:0/3726603514 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f779c014070 con 0x7f77a41076b0 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef9c0 0x55ac516fc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516fcd00 0x55ac516ff0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ac5165d840 con 0x55ac516ef0a0 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ac5165ad70 con 0x55ac516fcd00 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77ad835740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac51656550 con 0x55ac516ef9c0 2024-03-24T09:31:58.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a8920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516ef480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a8920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516ef480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35536/0 (socket says 172.21.15.116:35536) 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a8920640 1 -- 172.21.15.116:0/2388893848 learned_addr learned my addr 172.21.15.116:0/2388893848 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a9121640 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516fcd00 0x55ac516ff0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.199+0000 7f77a3fff640 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef9c0 0x55ac516fc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a8920640 1 -- 172.21.15.116:0/2388893848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef9c0 msgr2=0x55ac516fc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a8920640 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef9c0 0x55ac516fc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a8920640 1 -- 172.21.15.116:0/2388893848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516fcd00 msgr2=0x55ac516ff0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a8920640 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516fcd00 0x55ac516ff0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a8920640 1 -- 172.21.15.116:0/2388893848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac515ebde0 con 0x55ac516ef0a0 2024-03-24T09:31:58.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a3fff640 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef9c0 0x55ac516fc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a3fff640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f77940a2230 0x7f77940a46f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7790005ab0 tx=0x7f7790004b00 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f77a8920640 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516ef480 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f7798009ca0 tx=0x7f7798011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67013b7568005fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f778effd640 1 -- 172.21.15.116:0/2388893848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7798009e50 con 0x55ac516ef0a0 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f778effd640 1 -- 172.21.15.116:0/2388893848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77980198c0 con 0x55ac516ef0a0 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.200+0000 7f778effd640 1 -- 172.21.15.116:0/2388893848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7798018840 con 0x55ac516ef0a0 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 -- 172.21.15.116:0/2388893848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 msgr2=0x55ac516ef480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516ef480 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f7798009ca0 tx=0x7f7798011810 comp rx=0 tx=0).stop 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 -- 172.21.15.116:0/2388893848 shutdown_connections 2024-03-24T09:31:58.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516fcd00 0x55ac516ff0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef9c0 0x55ac516fc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 --2- 172.21.15.116:0/2388893848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516ef480 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 -- 172.21.15.116:0/2388893848 >> 172.21.15.116:0/2388893848 conn(0x55ac516e9b20 msgr2=0x55ac516e9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 -- 172.21.15.116:0/2388893848 shutdown_connections 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 -- 172.21.15.116:0/2388893848 wait complete. 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 -- start start 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 Processor -- start 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516f9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.201+0000 7f77ad835740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac516fa230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac516fa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ac51329430 con 0x55ac516ef0a0 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ac512e5e00 con 0x55ac516ef9c0 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac515db590 con 0x55ac516fcd00 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 10 client.0 _ll_register_callbacks cb 0x55ac5164afc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516f9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516f9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35542/0 (socket says 172.21.15.116:35542) 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 -- 172.21.15.116:0/1375482326 learned_addr learned my addr 172.21.15.116:0/1375482326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a3fff640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac516fa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a9121640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac516fa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 -- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 msgr2=0x55ac516fa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac516fa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 -- 172.21.15.116:0/1375482326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 msgr2=0x55ac516fa230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac516fa230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 -- 172.21.15.116:0/1375482326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77980098d0 con 0x55ac516ef0a0 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77a8920640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516f9cf0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7798005890 tx=0x7f7798019c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbcf058f39ccbe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f7783fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f7783fff640 1 -- 172.21.15.116:0/1375482326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77980183b0 con 0x55ac516ef0a0 2024-03-24T09:31:58.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f7783fff640 1 -- 172.21.15.116:0/1375482326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f7798009e50 con 0x55ac516ef0a0 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ac513557e0 con 0x55ac516ef0a0 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f7783fff640 1 -- 172.21.15.116:0/1375482326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f7798005260 con 0x55ac516ef0a0 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 10 client.18429 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ac51654bd0 con 0x55ac516ef0a0 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.202+0000 7f7783fff640 1 -- 172.21.15.116:0/1375482326 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ac513557e0 con 0x55ac516ef0a0 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f7783fff640 1 -- 172.21.15.116:0/1375482326 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f77980185e0 con 0x55ac516ef0a0 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77ad835740 10 client.18429 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77ad835740 10 client.18429 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:31:58.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77ad835740 20 client.18429 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77ad835740 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac5170cdd0 0x55ac5170f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77ad835740 4 client.18429 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac51392720 con 0x55ac5170cdd0 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77a3fff640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac5170cdd0 0x55ac5170f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f7783fff640 10 client.18429 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.203+0000 7f77a3fff640 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac5170cdd0 0x55ac5170f1e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9210e4bede1d10e server_cookie=49f8d596373ad1d5 in_seq=0 out_seq=0 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f7783fff640 1 -- 172.21.15.116:0/1375482326 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ac51392720 con 0x55ac5170cdd0 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f7783fff640 10 client.18429 handle_command_reply: tid=0 2024-03-24T09:31:58.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 client.18429 shutdown 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 msgr2=0x55ac516f9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516f9cf0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7798005890 tx=0x7f7798019c60 comp rx=0 tx=0).stop 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 shutdown_connections 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac516fa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac5170cdd0 0x55ac5170f1e0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac516fa230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f7783fff640 0 client.18429 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 --2- 172.21.15.116:0/1375482326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516f9cf0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 >> 172.21.15.116:0/1375482326 conn(0x55ac516e9b20 msgr2=0x55ac516eb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 shutdown_connections 2024-03-24T09:31:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 1 -- 172.21.15.116:0/1375482326 wait complete. 2024-03-24T09:31:58.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.204+0000 7f77ad835740 20 client.18429 trim_cache size 0 max 16384 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 Processor -- start 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 -- start start 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51708650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac51714af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ac516ff0f0 con 0x55ac516ef0a0 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ac516ee710 con 0x55ac516ef9c0 2024-03-24T09:31:58.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77ad835740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac516f8240 con 0x55ac516fcd00 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.217+0000 7f77a8920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516e4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35546/0 (socket says 172.21.15.116:35546) 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 -- 172.21.15.116:0/1191317652 learned_addr learned my addr 172.21.15.116:0/1191317652 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a9121640 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac51714af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a3fff640 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51708650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 -- 172.21.15.116:0/1191317652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 msgr2=0x55ac51714af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac51714af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 -- 172.21.15.116:0/1191317652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 msgr2=0x55ac51708650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51708650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 -- 172.21.15.116:0/1191317652 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac515f0c80 con 0x55ac516ef0a0 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f77a8920640 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516e4e30 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7798011d10 tx=0x7f7798019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc38d7221ea42d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f778e7fc640 1 -- 172.21.15.116:0/1191317652 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7798011dc0 con 0x55ac516ef0a0 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f778e7fc640 1 -- 172.21.15.116:0/1191317652 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7798019cd0 con 0x55ac516ef0a0 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.218+0000 7f778e7fc640 1 -- 172.21.15.116:0/1191317652 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77980035f0 con 0x55ac516ef0a0 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- 172.21.15.116:0/1191317652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 msgr2=0x55ac516e4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516e4e30 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7798011d10 tx=0x7f7798019790 comp rx=0 tx=0).stop 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- 172.21.15.116:0/1191317652 shutdown_connections 2024-03-24T09:31:58.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516fcd00 0x55ac51714af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51708650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- 172.21.15.116:0/1191317652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516ef0a0 0x55ac516e4e30 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- 172.21.15.116:0/1191317652 >> 172.21.15.116:0/1191317652 conn(0x55ac516e9b20 msgr2=0x55ac516f4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- 172.21.15.116:0/1191317652 shutdown_connections 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- 172.21.15.116:0/1191317652 wait complete. 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- start start 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 Processor -- start 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef0a0 0x55ac517050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51700410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 0x55ac51700b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ac51701250 con 0x55ac516fcd00 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ac51772120 con 0x55ac516ef9c0 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ac51772260 con 0x55ac516ef0a0 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.219+0000 7f77ad835740 10 client.0 _ll_register_callbacks cb 0x7f779801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 0x55ac51700b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a8920640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef0a0 0x55ac517050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 0x55ac51700b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35560/0 (socket says 172.21.15.116:35560) 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51700410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 -- 172.21.15.116:0/1546268219 learned_addr learned my addr 172.21.15.116:0/1546268219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 -- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef0a0 msgr2=0x55ac517050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef0a0 0x55ac517050a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 -- 172.21.15.116:0/1546268219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 msgr2=0x55ac51700410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51700410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 -- 172.21.15.116:0/1546268219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac51654bd0 con 0x55ac516fcd00 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77a9121640 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 0x55ac51700b50 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f779c0974a0 tx=0x7f779c09b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b389c71a66b8e283 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f7783fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f7783fff640 1 -- 172.21.15.116:0/1546268219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f779c0b9500 con 0x55ac516fcd00 2024-03-24T09:31:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f7783fff640 1 -- 172.21.15.116:0/1546268219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f779c09f190 con 0x55ac516fcd00 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77980198c0 con 0x55ac516fcd00 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f7783fff640 1 -- 172.21.15.116:0/1546268219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f779c0a37e0 con 0x55ac516fcd00 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77ad835740 10 client.18435 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ac51616920 con 0x55ac516fcd00 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.220+0000 7f7783fff640 1 -- 172.21.15.116:0/1546268219 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f77980198c0 con 0x55ac516fcd00 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f7783fff640 1 -- 172.21.15.116:0/1546268219 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f779c0b9020 con 0x55ac516fcd00 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 10 client.18435 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 10 client.18435 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 20 client.18435 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac51773ab0 0x55ac51775ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 4 client.18435 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ac513e1890 con 0x55ac51773ab0 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77a8920640 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac51773ab0 0x55ac51775ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f7783fff640 10 client.18435 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77a8920640 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac51773ab0 0x55ac51775ed0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=774b9ac0ac0d4167 server_cookie=8afbf3ab1a874a89 in_seq=0 out_seq=0 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f7783fff640 1 -- 172.21.15.116:0/1546268219 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ac513e1890 con 0x55ac51773ab0 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f7783fff640 10 client.18435 handle_command_reply: tid=0 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 client.18435 shutdown 2024-03-24T09:31:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 msgr2=0x55ac51700b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 0x55ac51700b50 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f779c0974a0 tx=0x7f779c09b7b0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 shutdown_connections 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ac516fcd00 0x55ac51700b50 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55ac51773ab0 0x55ac51775ed0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ac516ef9c0 0x55ac51700410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 --2- 172.21.15.116:0/1546268219 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ac516ef0a0 0x55ac517050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f7783fff640 0 client.18435 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.221+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 >> 172.21.15.116:0/1546268219 conn(0x55ac516e9b20 msgr2=0x55ac516f9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 shutdown_connections 2024-03-24T09:31:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77ad835740 1 -- 172.21.15.116:0/1546268219 wait complete. 2024-03-24T09:31:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77ad835740 20 client.18435 trim_cache size 0 max 16384 2024-03-24T09:31:58.232 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f77940a2230 msgr2=0x7f77940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f77940a2230 0x7f77940a46f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7790005ab0 tx=0x7f7790004b00 comp rx=0 tx=0).stop 2024-03-24T09:31:58.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 msgr2=0x7f77a4197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 0x7f77a4197b10 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f779c00efd0 tx=0x7f779c00c6a0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 shutdown_connections 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77a41076b0 0x7f77a4197b10 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a4102060 0x7f77a419df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f77940a2230 0x7f77940a46f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 --2- 172.21.15.116:0/3726603514 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f77a41016a0 0x7f77a419da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 >> 172.21.15.116:0/3726603514 conn(0x7f77a40fd1b0 msgr2=0x7f77a40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 shutdown_connections 2024-03-24T09:31:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.222+0000 7f77aab84640 1 -- 172.21.15.116:0/3726603514 wait complete. 2024-03-24T09:31:58.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:31:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/3831519377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 msgr2=0x7fd3c010d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/3831519377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c010d460 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009c10 tx=0x7fd3b802bf80 comp rx=0 tx=0).stop 2024-03-24T09:31:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/3831519377 shutdown_connections 2024-03-24T09:31:58.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/3831519377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c010d460 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/3831519377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/3831519377 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c01066a0 0x7fd3c0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/3831519377 >> 172.21.15.116:0/3831519377 conn(0x7fd3c0076bd0 msgr2=0x7fd3c00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/3831519377 shutdown_connections 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/3831519377 wait complete. 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 Processor -- start 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- start start 2024-03-24T09:31:58.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c019d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c019ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c01066a0 0x7fd3c0197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0198090 con 0x7fd3c0100d20 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c0198200 con 0x7fd3c01016e0 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.338+0000 7fd3c6fa0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c01983c0 con 0x7fd3c01066a0 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c019d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c019d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35582/0 (socket says 172.21.15.116:35582) 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c019ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/2860827677 learned_addr learned my addr 172.21.15.116:0/2860827677 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c553d640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c01066a0 0x7fd3c0197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/2860827677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c01066a0 msgr2=0x7fd3c0197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c01066a0 0x7fd3c0197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/2860827677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 msgr2=0x7fd3c019ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c019ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/2860827677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac009560 con 0x7fd3c0100d20 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3bffff640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c019d890 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400e9f0 tx=0x7fd3b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3bdffb640 1 -- 172.21.15.116:0/2860827677 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4009700 con 0x7fd3c0100d20 2024-03-24T09:31:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b8009840 con 0x7fd3c0100d20 2024-03-24T09:31:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3bdffb640 1 -- 172.21.15.116:0/2860827677 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3b400cd80 con 0x7fd3c0100d20 2024-03-24T09:31:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3bdffb640 1 -- 172.21.15.116:0/2860827677 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4018840 con 0x7fd3c0100d20 2024-03-24T09:31:58.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.339+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c0198890 con 0x7fd3c0100d20 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.341+0000 7fd3bdffb640 1 -- 172.21.15.116:0/2860827677 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd3b40189a0 con 0x7fd3c0100d20 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 Processor -- start 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 -- start start 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f599480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 0x55f90f5a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f90f507840 con 0x55f90f5a6d00 2024-03-24T09:31:58.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f90f504d70 con 0x55f90f5990a0 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f90f500550 con 0x55f90f5999c0 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bdffb640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3b00a2230 0x7fd3b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 0x55f90f5a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 0x55f90f5a90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35592/0 (socket says 172.21.15.116:35592) 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c553d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f599480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 -- 172.21.15.116:0/711176951 learned_addr learned my addr 172.21.15.116:0/711176951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c553d640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3b00a2230 0x7fd3b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bdffb640 1 -- 172.21.15.116:0/2860827677 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd3b4014070 con 0x7fd3c0100d20 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 -- 172.21.15.116:0/711176951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 msgr2=0x55f90f5a67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 -- 172.21.15.116:0/711176951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 msgr2=0x55f90f599480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f599480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3bffff640 1 -- 172.21.15.116:0/711176951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f90f495de0 con 0x55f90f5a6d00 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.342+0000 7fd3c553d640 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f599480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c553d640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3b00a2230 0x7fd3b00a46f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8004af0 tx=0x7fd3b8009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3bffff640 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 0x55f90f5a90f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac011600 tx=0x7fd3ac011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f957637dccdc0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3aaffd640 1 -- 172.21.15.116:0/711176951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00f6f0 con 0x55f90f5a6d00 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3aaffd640 1 -- 172.21.15.116:0/711176951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3ac00fd00 con 0x55f90f5a6d00 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3aaffd640 1 -- 172.21.15.116:0/711176951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac019e10 con 0x55f90f5a6d00 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 -- 172.21.15.116:0/711176951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 msgr2=0x55f90f5a90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 0x55f90f5a90f0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac011600 tx=0x7fd3ac011ac0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 -- 172.21.15.116:0/711176951 shutdown_connections 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5a6d00 0x55f90f5a90f0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/711176951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f599480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 -- 172.21.15.116:0/711176951 >> 172.21.15.116:0/711176951 conn(0x55f90f593b20 msgr2=0x55f90f593f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.343+0000 7fd3c9b77740 1 -- 172.21.15.116:0/711176951 shutdown_connections 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 -- 172.21.15.116:0/711176951 wait complete. 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 -- start start 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 Processor -- start 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5a6d00 0x55f90f5a47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f90f1d3430 con 0x55f90f5990a0 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f90f18fe00 con 0x55f90f5a6d00 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f90f485590 con 0x55f90f5999c0 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c9b77740 10 client.0 _ll_register_callbacks cb 0x55f90f4f4fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35598/0 (socket says 172.21.15.116:35598) 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/853814721 learned_addr learned my addr 172.21.15.116:0/853814721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3bffff640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.344+0000 7fd3c553d640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5a6d00 0x55f90f5a47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 msgr2=0x55f90f5a4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/853814721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5a6d00 msgr2=0x55f90f5a47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5a6d00 0x55f90f5a47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/853814721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac009df0 con 0x55f90f5990a0 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a3d40 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4097230 tx=0x7fd3b400aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462e5b3a02165b5a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd39ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd39ffff640 1 -- 172.21.15.116:0/853814721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b40a34c0 con 0x55f90f5990a0 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd39ffff640 1 -- 172.21.15.116:0/853814721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd3b409add0 con 0x55f90f5990a0 2024-03-24T09:31:58.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f90f1ff7e0 con 0x55f90f5990a0 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd39ffff640 1 -- 172.21.15.116:0/853814721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3b4097440 con 0x55f90f5990a0 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 10 client.18447 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f90f4febd0 con 0x55f90f5990a0 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd39ffff640 1 -- 172.21.15.116:0/853814721 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f90f1ff7e0 con 0x55f90f5990a0 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd39ffff640 1 -- 172.21.15.116:0/853814721 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd3b4096b20 con 0x55f90f5990a0 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 10 client.18447 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 10 client.18447 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 20 client.18447 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f5b6ea0 0x55f90f5b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 4 client.18447 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.345+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f90f23c720 con 0x55f90f5b6ea0 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.346+0000 7fd3bffff640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f5b6ea0 0x55f90f5b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.346+0000 7fd39ffff640 10 client.18447 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.346+0000 7fd3bffff640 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f5b6ea0 0x55f90f5b92b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3981095da7188d75 server_cookie=96a979e3465f0c59 in_seq=0 out_seq=0 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd39ffff640 1 -- 172.21.15.116:0/853814721 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f90f23c720 con 0x55f90f5b6ea0 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd39ffff640 10 client.18447 handle_command_reply: tid=0 2024-03-24T09:31:58.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 client.18447 shutdown 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 msgr2=0x55f90f5a3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a3d40 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd3b4097230 tx=0x7fd3b400aee0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 shutdown_connections 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f5b6ea0 0x55f90f5b92b0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5a6d00 0x55f90f5a47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd39ffff640 0 client.18447 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5999c0 0x55f90f5a4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/853814721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a3d40 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 >> 172.21.15.116:0/853814721 conn(0x55f90f593b20 msgr2=0x55f90f595870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 shutdown_connections 2024-03-24T09:31:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 1 -- 172.21.15.116:0/853814721 wait complete. 2024-03-24T09:31:58.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.347+0000 7fd3c9b77740 20 client.18447 trim_cache size 0 max 16384 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 Processor -- start 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 -- start start 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5999c0 0x55f90f5a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5bebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f90f5b3960 con 0x55f90f5990a0 2024-03-24T09:31:58.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f90f599820 con 0x55f90f5999c0 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f90f598740 con 0x55f90f5a6d00 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35602/0 (socket says 172.21.15.116:35602) 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/3340667454 learned_addr learned my addr 172.21.15.116:0/3340667454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3bffff640 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5999c0 0x55f90f5a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c553d640 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5bebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/3340667454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 msgr2=0x55f90f5bebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5bebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/3340667454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5999c0 msgr2=0x55f90f5a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5999c0 0x55f90f5a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 -- 172.21.15.116:0/3340667454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f90f49ac80 con 0x55f90f5990a0 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.360+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a6470 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd3b409f030 tx=0x7fd3b401df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ae44f37fa9e23e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3aa7fc640 1 -- 172.21.15.116:0/3340667454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b409a730 con 0x55f90f5990a0 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3aa7fc640 1 -- 172.21.15.116:0/3340667454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3b409add0 con 0x55f90f5990a0 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3aa7fc640 1 -- 172.21.15.116:0/3340667454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3b4097440 con 0x55f90f5990a0 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 -- 172.21.15.116:0/3340667454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 msgr2=0x55f90f5a6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a6470 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd3b409f030 tx=0x7fd3b401df80 comp rx=0 tx=0).stop 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 -- 172.21.15.116:0/3340667454 shutdown_connections 2024-03-24T09:31:58.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5bebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5999c0 0x55f90f5a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/3340667454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5990a0 0x55f90f5a6470 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 -- 172.21.15.116:0/3340667454 >> 172.21.15.116:0/3340667454 conn(0x55f90f593b20 msgr2=0x55f90f594e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 -- 172.21.15.116:0/3340667454 shutdown_connections 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.361+0000 7fd3c9b77740 1 -- 172.21.15.116:0/3340667454 wait complete. 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 -- start start 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 Processor -- start 2024-03-24T09:31:58.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f5a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 0x55f90f5a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5a1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f90f61c1f0 con 0x55f90f5999c0 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f90f61c330 con 0x55f90f5990a0 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f90f61c4a0 con 0x55f90f5a6d00 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c9b77740 10 client.0 _ll_register_callbacks cb 0x7fd3b400e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 0x55f90f5a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c4d3c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f5a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 0x55f90f5a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35604/0 (socket says 172.21.15.116:35604) 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 -- 172.21.15.116:0/1319570296 learned_addr learned my addr 172.21.15.116:0/1319570296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c553d640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 -- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 msgr2=0x55f90f5a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5a1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 -- 172.21.15.116:0/1319570296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 msgr2=0x55f90f5a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f5a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3bffff640 1 -- 172.21.15.116:0/1319570296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f90f4febd0 con 0x55f90f5999c0 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.362+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f5a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3bffff640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 0x55f90f5a0ef0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000c00 tx=0x7fd3ac00f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bad75b103689aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd39ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd39ffff640 1 -- 172.21.15.116:0/1319570296 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac02a070 con 0x55f90f5999c0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd39ffff640 1 -- 172.21.15.116:0/1319570296 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fd3ac009260 con 0x55f90f5999c0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3ac00fd00 con 0x55f90f5999c0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd39ffff640 1 -- 172.21.15.116:0/1319570296 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd3ac0197a0 con 0x55f90f5999c0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 10 client.18453 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f90f4c0920 con 0x55f90f5999c0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd39ffff640 1 -- 172.21.15.116:0/1319570296 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3ac00fd00 con 0x55f90f5999c0 2024-03-24T09:31:58.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd39ffff640 1 -- 172.21.15.116:0/1319570296 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fd3ac005040 con 0x55f90f5999c0 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 10 client.18453 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 10 client.18453 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 20 client.18453 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f61dc80 0x55f90f6200a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 4 client.18453 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f90f28b890 con 0x55f90f61dc80 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.363+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f61dc80 0x55f90f6200a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd39ffff640 10 client.18453 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c4d3c640 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f61dc80 0x55f90f6200a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8cb601794f5ff580 server_cookie=8987d5f843c41a4d in_seq=0 out_seq=0 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd39ffff640 1 -- 172.21.15.116:0/1319570296 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f90f28b890 con 0x55f90f61dc80 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd39ffff640 10 client.18453 handle_command_reply: tid=0 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 client.18453 shutdown 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 msgr2=0x55f90f5a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 0x55f90f5a0ef0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac000c00 tx=0x7fd3ac00f4c0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 shutdown_connections 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55f90f61dc80 0x55f90f6200a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f90f5a6d00 0x55f90f5a1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f90f5999c0 0x55f90f5a0ef0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd39ffff640 0 client.18453 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 --2- 172.21.15.116:0/1319570296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f90f5990a0 0x55f90f5a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 >> 172.21.15.116:0/1319570296 conn(0x55f90f593b20 msgr2=0x55f90f5955c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 shutdown_connections 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 1 -- 172.21.15.116:0/1319570296 wait complete. 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.364+0000 7fd3c9b77740 20 client.18453 trim_cache size 0 max 16384 2024-03-24T09:31:58.376 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3b00a2230 msgr2=0x7fd3b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3b00a2230 0x7fd3b00a46f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8004af0 tx=0x7fd3b8009500 comp rx=0 tx=0).stop 2024-03-24T09:31:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 msgr2=0x7fd3c019d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c019d890 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd3b400e9f0 tx=0x7fd3b400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 shutdown_connections 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3b00a2230 0x7fd3b00a46f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd3c01066a0 0x7fd3c0197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3c01016e0 0x7fd3c019ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 --2- 172.21.15.116:0/2860827677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3c0100d20 0x7fd3c019d890 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 >> 172.21.15.116:0/2860827677 conn(0x7fd3c0076bd0 msgr2=0x7fd3c00fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 shutdown_connections 2024-03-24T09:31:58.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.365+0000 7fd3c6fa0640 1 -- 172.21.15.116:0/2860827677 wait complete. 2024-03-24T09:31:58.379 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:58 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4177099548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:31:58.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:31:58.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 -- 172.21.15.116:0/2746430073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941032c0 msgr2=0x7fc79410b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 --2- 172.21.15.116:0/2746430073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941032c0 0x7fc79410b7a0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fc77c009900 tx=0x7fc77c02bcd0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 -- 172.21.15.116:0/2746430073 shutdown_connections 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 --2- 172.21.15.116:0/2746430073 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc79410bce0 0x7fc79410e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 --2- 172.21.15.116:0/2746430073 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941032c0 0x7fc79410b7a0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 --2- 172.21.15.116:0/2746430073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc7941029a0 0x7fc794102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 -- 172.21.15.116:0/2746430073 >> 172.21.15.116:0/2746430073 conn(0x7fc794077770 msgr2=0x7fc794077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 -- 172.21.15.116:0/2746430073 shutdown_connections 2024-03-24T09:31:58.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 -- 172.21.15.116:0/2746430073 wait complete. 2024-03-24T09:31:58.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.483+0000 7fc79973e640 1 Processor -- start 2024-03-24T09:31:58.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 -- start start 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 0x7fc79419d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7941032c0 0x7fc79419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79410bce0 0x7fc794197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc794198110 con 0x7fc7941029a0 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc794198280 con 0x7fc79410bce0 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc79973e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc794198440 con 0x7fc7941032c0 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 0x7fc79419d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 0x7fc79419d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35614/0 (socket says 172.21.15.116:35614) 2024-03-24T09:31:58.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc78a7fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7941032c0 0x7fc79419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 -- 172.21.15.116:0/3472092622 learned_addr learned my addr 172.21.15.116:0/3472092622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79410bce0 0x7fc794197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 -- 172.21.15.116:0/3472092622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7941032c0 msgr2=0x7fc79419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7941032c0 0x7fc79419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 -- 172.21.15.116:0/3472092622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79410bce0 msgr2=0x7fc794197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79410bce0 0x7fc794197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 -- 172.21.15.116:0/3472092622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77c009560 con 0x7fc7941029a0 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7941032c0 0x7fc79419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.484+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 0x7fc79419d8c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc78400e9f0 tx=0x7fc78400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.485+0000 7fc790ff9640 1 -- 172.21.15.116:0/3472092622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784009700 con 0x7fc7941029a0 2024-03-24T09:31:58.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.485+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc794198720 con 0x7fc7941029a0 2024-03-24T09:31:58.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.485+0000 7fc790ff9640 1 -- 172.21.15.116:0/3472092622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc78400cd80 con 0x7fc7941029a0 2024-03-24T09:31:58.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.485+0000 7fc790ff9640 1 -- 172.21.15.116:0/3472092622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc784018840 con 0x7fc7941029a0 2024-03-24T09:31:58.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.485+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7941a4960 con 0x7fc7941029a0 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc790ff9640 1 -- 172.21.15.116:0/3472092622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc7840189a0 con 0x7fc7941029a0 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 Processor -- start 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 -- start start 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b492fb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fbd10 0x558b493043b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b49306ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc790ff9640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7800a2230 0x7fc7800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b49269840 con 0x558b493048f0 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558b49266d70 con 0x558b492fbd10 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.487+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b49262550 con 0x558b492fb3f0 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b492fb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b49306ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b492fb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:50150/0 (socket says 172.21.15.116:50150) 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7937fe640 1 -- 172.21.15.116:0/4251704977 learned_addr learned my addr 172.21.15.116:0/4251704977 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc790ff9640 1 -- 172.21.15.116:0/3472092622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fc784014070 con 0x7fc7941029a0 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc792ffd640 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fbd10 0x558b493043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7800a2230 0x7fc7800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 -- 172.21.15.116:0/4251704977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 msgr2=0x558b492fb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b492fb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 -- 172.21.15.116:0/4251704977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fbd10 msgr2=0x558b493043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fbd10 0x558b493043b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 -- 172.21.15.116:0/4251704977 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b491f7de0 con 0x558b493048f0 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7800a2230 0x7fc7800a46f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005c40 tx=0x7fc78c009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b49306ce0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fc77c000c00 tx=0x7fc77c004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a3c1871b616c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7897fa640 1 -- 172.21.15.116:0/4251704977 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c02c810 con 0x558b493048f0 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7897fa640 1 -- 172.21.15.116:0/4251704977 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc77c0044f0 con 0x558b493048f0 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.488+0000 7fc7897fa640 1 -- 172.21.15.116:0/4251704977 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c003c00 con 0x558b493048f0 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 -- 172.21.15.116:0/4251704977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 msgr2=0x558b49306ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b49306ce0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fc77c000c00 tx=0x7fc77c004a70 comp rx=0 tx=0).stop 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 -- 172.21.15.116:0/4251704977 shutdown_connections 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b49306ce0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fbd10 0x558b493043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/4251704977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b492fb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 -- 172.21.15.116:0/4251704977 >> 172.21.15.116:0/4251704977 conn(0x558b492f5b20 msgr2=0x558b492f5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 -- 172.21.15.116:0/4251704977 shutdown_connections 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 -- 172.21.15.116:0/4251704977 wait complete. 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 -- start start 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.489+0000 7fc79c3f2740 1 Processor -- start 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b49301930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49301e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b493023b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b48f35430 con 0x558b492fbd10 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558b48ef1e00 con 0x558b493048f0 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b491e7590 con 0x558b492fb3f0 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 10 client.0 _ll_register_callbacks cb 0x558b49256fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49301e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49301e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35638/0 (socket says 172.21.15.116:35638) 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b49301930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b493023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 -- 172.21.15.116:0/2007094444 learned_addr learned my addr 172.21.15.116:0/2007094444 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 -- 172.21.15.116:0/2007094444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 msgr2=0x558b49301930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b49301930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 -- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 msgr2=0x558b493023b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b493023b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 -- 172.21.15.116:0/2007094444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc79410edd0 con 0x558b492fbd10 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc7937fe640 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b493023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49301e70 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fc77c009900 tx=0x7fc77c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b76bf98b49b4e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc77a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc77a7fc640 1 -- 172.21.15.116:0/2007094444 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c004fc0 con 0x558b492fbd10 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc77a7fc640 1 -- 172.21.15.116:0/2007094444 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fc77c005120 con 0x558b492fbd10 2024-03-24T09:31:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558b48f617e0 con 0x558b492fbd10 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.490+0000 7fc77a7fc640 1 -- 172.21.15.116:0/2007094444 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc77c0052e0 con 0x558b492fbd10 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 10 client.18465 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558b49260bd0 con 0x558b492fbd10 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc77a7fc640 1 -- 172.21.15.116:0/2007094444 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x558b48f617e0 con 0x558b492fbd10 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc77a7fc640 1 -- 172.21.15.116:0/2007094444 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc77c03c050 con 0x558b492fbd10 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 10 client.18465 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 10 client.18465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 20 client.18465 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b493146b0 0x558b49316ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 4 client.18465 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b48f9e720 con 0x558b493146b0 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc792ffd640 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b493146b0 0x558b49316ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc77a7fc640 10 client.18465 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.491+0000 7fc792ffd640 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b493146b0 0x558b49316ad0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9475c747af07d6a4 server_cookie=8c1baeb92a6b87bc in_seq=0 out_seq=0 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc77a7fc640 1 -- 172.21.15.116:0/2007094444 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558b48f9e720 con 0x558b493146b0 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc77a7fc640 10 client.18465 handle_command_reply: tid=0 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 client.18465 shutdown 2024-03-24T09:31:58.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 msgr2=0x558b49301e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49301e70 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fc77c009900 tx=0x7fc77c003820 comp rx=0 tx=0).stop 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 shutdown_connections 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b493146b0 0x558b49316ad0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b493023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49301e70 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc77a7fc640 0 client.18465 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/2007094444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b49301930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 >> 172.21.15.116:0/2007094444 conn(0x558b492f5b20 msgr2=0x558b492f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 shutdown_connections 2024-03-24T09:31:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 1 -- 172.21.15.116:0/2007094444 wait complete. 2024-03-24T09:31:58.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.492+0000 7fc79c3f2740 20 client.18465 trim_cache size 0 max 16384 2024-03-24T09:31:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.505+0000 7fc79c3f2740 1 Processor -- start 2024-03-24T09:31:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.505+0000 7fc79c3f2740 1 -- start start 2024-03-24T09:31:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 0x558b49310120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fbd10 0x558b4931c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b4930c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b492fa7c0 con 0x558b493048f0 2024-03-24T09:31:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558b492f6b20 con 0x558b492fb3f0 2024-03-24T09:31:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b492f13e0 con 0x558b492fbd10 2024-03-24T09:31:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b4930c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 0x558b49310120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b4930c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35652/0 (socket says 172.21.15.116:35652) 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc792ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 0x558b49310120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59224/0 (socket says 172.21.15.116:59224) 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 -- 172.21.15.116:0/3796031924 learned_addr learned my addr 172.21.15.116:0/3796031924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fbd10 0x558b4931c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 -- 172.21.15.116:0/3796031924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fbd10 msgr2=0x558b4931c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fbd10 0x558b4931c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 -- 172.21.15.116:0/3796031924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 msgr2=0x558b49310120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 0x558b49310120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 -- 172.21.15.116:0/3796031924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b491fcc80 con 0x558b493048f0 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fbd10 0x558b4931c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 0x558b49310120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b4930c740 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009e20 tx=0x7fc78c013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e5a208fe4cd469 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc788ff9640 1 -- 172.21.15.116:0/3796031924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c025040 con 0x558b493048f0 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.506+0000 7fc788ff9640 1 -- 172.21.15.116:0/3796031924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc78c0044e0 con 0x558b493048f0 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc788ff9640 1 -- 172.21.15.116:0/3796031924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0209e0 con 0x558b493048f0 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3796031924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 msgr2=0x558b4930c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b4930c740 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fc78c009e20 tx=0x7fc78c013070 comp rx=0 tx=0).stop 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3796031924 shutdown_connections 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b493048f0 0x558b4930c740 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fbd10 0x558b4931c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3796031924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b492fb3f0 0x558b49310120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3796031924 >> 172.21.15.116:0/3796031924 conn(0x558b492f5b20 msgr2=0x558b493018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3796031924 shutdown_connections 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3796031924 wait complete. 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.507+0000 7fc79c3f2740 1 -- start start 2024-03-24T09:31:58.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 Processor -- start 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b4930c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49308a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b49308fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558b49309510 con 0x558b492fbd10 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558b49309680 con 0x558b493048f0 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x558b49309840 con 0x558b492fb3f0 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc79c3f2740 10 client.0 _ll_register_callbacks cb 0x7fc78400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49308a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49308a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35656/0 (socket says 172.21.15.116:35656) 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 -- 172.21.15.116:0/3618495658 learned_addr learned my addr 172.21.15.116:0/3618495658 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b4930c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc7937fe640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b49308fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 -- 172.21.15.116:0/3618495658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 msgr2=0x558b4930c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b4930c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 -- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 msgr2=0x558b49308fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b49308fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 -- 172.21.15.116:0/3618495658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7941a4960 con 0x558b492fbd10 2024-03-24T09:31:58.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc78a7fc640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49308a90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fc77c009800 tx=0x7fc77c04a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3b5d3f45e07d5a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc77a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.508+0000 7fc77a7fc640 1 -- 172.21.15.116:0/3618495658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c04bdb0 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc77a7fc640 1 -- 172.21.15.116:0/3618495658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7fc77c000e40 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc77c0044f0 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc77a7fc640 1 -- 172.21.15.116:0/3618495658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc77c004dc0 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 10 client.18471 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558b49260bd0 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc77a7fc640 1 -- 172.21.15.116:0/3618495658 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc77c0044f0 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc77a7fc640 1 -- 172.21.15.116:0/3618495658 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc77c03c040 con 0x558b492fbd10 2024-03-24T09:31:58.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 10 client.18471 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 10 client.18471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 20 client.18471 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b4937bb40 0x558b4937df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 4 client.18471 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b48fed890 con 0x558b4937bb40 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.509+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b4937bb40 0x558b4937df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc792ffd640 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b4937bb40 0x558b4937df60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=612f7456e041906f server_cookie=5f7b139a149d22b3 in_seq=0 out_seq=0 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc77a7fc640 10 client.18471 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:58.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc77a7fc640 1 -- 172.21.15.116:0/3618495658 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b48fed890 con 0x558b4937bb40 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc77a7fc640 10 client.18471 handle_command_reply: tid=0 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 client.18471 shutdown 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 msgr2=0x558b49308a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49308a90 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fc77c009800 tx=0x7fc77c04a7b0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 shutdown_connections 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x558b4937bb40 0x558b4937df60 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b493048f0 0x558b49308fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc77a7fc640 0 client.18471 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:58.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558b492fbd10 0x558b49308a90 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 --2- 172.21.15.116:0/3618495658 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x558b492fb3f0 0x558b4930c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 >> 172.21.15.116:0/3618495658 conn(0x558b492f5b20 msgr2=0x558b492f6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 shutdown_connections 2024-03-24T09:31:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 1 -- 172.21.15.116:0/3618495658 wait complete. 2024-03-24T09:31:58.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79c3f2740 20 client.18471 trim_cache size 0 max 16384 2024-03-24T09:31:58.521 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7800a2230 msgr2=0x7fc7800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7800a2230 0x7fc7800a46f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc78c005c40 tx=0x7fc78c009500 comp rx=0 tx=0).stop 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 msgr2=0x7fc79419d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 0x7fc79419d8c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc78400e9f0 tx=0x7fc78400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 shutdown_connections 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc79410bce0 0x7fc794197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc7800a2230 0x7fc7800a46f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7941032c0 0x7fc79419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 --2- 172.21.15.116:0/3472092622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7941029a0 0x7fc79419d8c0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.510+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 >> 172.21.15.116:0/3472092622 conn(0x7fc794077770 msgr2=0x7fc7940fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.511+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 shutdown_connections 2024-03-24T09:31:58.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:58.511+0000 7fc79973e640 1 -- 172.21.15.116:0/3472092622 wait complete. 2024-03-24T09:31:58.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:58 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4177099548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:31:58.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:58 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4177099548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:31:59.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:31:59 smithi143 ceph-mon[35965]: pgmap v305: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:59.524 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T09:31:59.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:31:59.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T09:31:59.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:31:59 smithi116 ceph-mon[32207]: pgmap v305: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:59.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:31:59 smithi117 ceph-mon[35788]: pgmap v305: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:31:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 -- 172.21.15.116:0/4046900804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 msgr2=0x7f29b8100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/4046900804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b8100ab0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009930 tx=0x7f29ac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 -- 172.21.15.116:0/4046900804 shutdown_connections 2024-03-24T09:31:59.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/4046900804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29b8100ff0 0x7f29b810d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/4046900804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b8100ab0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/4046900804 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b8106640 0x7f29b8106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 -- 172.21.15.116:0/4046900804 >> 172.21.15.116:0/4046900804 conn(0x7f29b8077770 msgr2=0x7f29b8077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 -- 172.21.15.116:0/4046900804 shutdown_connections 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 -- 172.21.15.116:0/4046900804 wait complete. 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.642+0000 7f29bdfb6640 1 Processor -- start 2024-03-24T09:31:59.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 -- start start 2024-03-24T09:31:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29b8100ff0 0x7f29b819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b8106640 0x7f29b81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8197040 con 0x7f29b8100630 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29b81971b0 con 0x7f29b8100ff0 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29bdfb6640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f29b8197370 con 0x7f29b8106640 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b819c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35684/0 (socket says 172.21.15.116:35684) 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 -- 172.21.15.116:0/1735325408 learned_addr learned my addr 172.21.15.116:0/1735325408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b7fff640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b8106640 0x7f29b81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29b8100ff0 0x7f29b819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 -- 172.21.15.116:0/1735325408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b8106640 msgr2=0x7f29b81969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b8106640 0x7f29b81969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 -- 172.21.15.116:0/1735325408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29b8100ff0 msgr2=0x7f29b819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29b8100ff0 0x7f29b819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.643+0000 7f29b77fe640 1 -- 172.21.15.116:0/1735325408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ac009560 con 0x7f29b8100630 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.644+0000 7f29b77fe640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b819c840 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e9f0 tx=0x7f29a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.644+0000 7f29b4ff9640 1 -- 172.21.15.116:0/1735325408 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8009720 con 0x7f29b8100630 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.644+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b8197650 con 0x7f29b8100630 2024-03-24T09:31:59.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.644+0000 7f29b4ff9640 1 -- 172.21.15.116:0/1735325408 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29a800cd80 con 0x7f29b8100630 2024-03-24T09:31:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.644+0000 7f29b4ff9640 1 -- 172.21.15.116:0/1735325408 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8010430 con 0x7f29b8100630 2024-03-24T09:31:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.644+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b81a38e0 con 0x7f29b8100630 2024-03-24T09:31:59.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.646+0000 7f29b4ff9640 1 -- 172.21.15.116:0/1735325408 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f29a8005020 con 0x7f29b8100630 2024-03-24T09:31:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.646+0000 7f29c0b93740 1 Processor -- start 2024-03-24T09:31:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.646+0000 7f29b4ff9640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29a00a2230 0x7f29a00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.646+0000 7f29c0b93740 1 -- start start 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.646+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29a00a2230 0x7f29a00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b4ff9640 1 -- 172.21.15.116:0/1735325408 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f29a8014070 con 0x7f29b8100630 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d120227f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d12030260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29c0b93740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562d11f90530 con 0x562d12022d30 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29c0b93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562d11f8da60 con 0x562d1202de70 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29c0b93740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d11f89240 con 0x562d12022410 2024-03-24T09:31:59.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d120227f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d120227f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:50230/0 (socket says 172.21.15.116:50230) 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 -- 172.21.15.116:0/1466532611 learned_addr learned my addr 172.21.15.116:0/1466532611 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b7fff640 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d12030260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 -- 172.21.15.116:0/1466532611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 msgr2=0x562d12030260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d12030260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 -- 172.21.15.116:0/1466532611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 msgr2=0x562d1202d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 -- 172.21.15.116:0/1466532611 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d11f878c0 con 0x562d12022410 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29a00a2230 0x7f29a00a46f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f29ac005ab0 tx=0x7f29ac0047a0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f29b77fe640 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d120227f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009380 tx=0x7f29a809a7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8800796e8e48fc7d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f299dffb640 1 -- 172.21.15.116:0/1466532611 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a809e070 con 0x562d12022410 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f299dffb640 1 -- 172.21.15.116:0/1466532611 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29a809ada0 con 0x562d12022410 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.647+0000 7f299dffb640 1 -- 172.21.15.116:0/1466532611 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a80cbf00 con 0x562d12022410 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 -- 172.21.15.116:0/1466532611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 msgr2=0x562d120227f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d120227f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009380 tx=0x7f29a809a7c0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 -- 172.21.15.116:0/1466532611 shutdown_connections 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d12030260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 --2- 172.21.15.116:0/1466532611 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d120227f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 -- 172.21.15.116:0/1466532611 >> 172.21.15.116:0/1466532611 conn(0x562d1201ce90 msgr2=0x562d1201d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 -- 172.21.15.116:0/1466532611 shutdown_connections 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 -- 172.21.15.116:0/1466532611 wait complete. 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.648+0000 7f29c0b93740 1 -- start start 2024-03-24T09:31:59.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 Processor -- start 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d1202ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d1202b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562d11f2df80 con 0x562d12022d30 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562d11c5c4b0 con 0x562d1202de70 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d11f0ead0 con 0x562d12022410 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29c0b93740 10 client.0 _ll_register_callbacks cb 0x562d11f41b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:59.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35706/0 (socket says 172.21.15.116:35706) 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 -- 172.21.15.116:0/3308675280 learned_addr learned my addr 172.21.15.116:0/3308675280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b7fff640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d1202b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d1202ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 -- 172.21.15.116:0/3308675280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 msgr2=0x562d1202ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d1202ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 -- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 msgr2=0x562d1202b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d1202b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.649+0000 7f29b6ffd640 1 -- 172.21.15.116:0/3308675280 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b81a38e0 con 0x562d12022d30 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202b370 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f29ac035000 tx=0x7f29ac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58b2cc552271b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f2996ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f2996ffd640 1 -- 172.21.15.116:0/3308675280 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0044c0 con 0x562d12022d30 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f2996ffd640 1 -- 172.21.15.116:0/3308675280 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f29ac003800 con 0x562d12022d30 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562d11fad7a0 con 0x562d12022d30 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f2996ffd640 1 -- 172.21.15.116:0/3308675280 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f29ac0039c0 con 0x562d12022d30 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 10 client.18483 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562d11f3f6d0 con 0x562d12022d30 2024-03-24T09:31:59.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f2996ffd640 1 -- 172.21.15.116:0/3308675280 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x562d11fad7a0 con 0x562d12022d30 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f2996ffd640 1 -- 172.21.15.116:0/3308675280 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f29ac041460 con 0x562d12022d30 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 10 client.18483 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 10 client.18483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 20 client.18483 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d1203e100 0x562d12040520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 4 client.18483 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d11f7f1f0 con 0x562d1203e100 2024-03-24T09:31:59.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.650+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d1203e100 0x562d12040520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.651+0000 7f2996ffd640 10 client.18483 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.651+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d1203e100 0x562d12040520 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=400b2e2a3bdc1e6e server_cookie=fd57cd5d297c0b88 in_seq=0 out_seq=0 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.651+0000 7f2996ffd640 1 -- 172.21.15.116:0/3308675280 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562d11f7f1f0 con 0x562d1203e100 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.651+0000 7f2996ffd640 10 client.18483 handle_command_reply: tid=0 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.651+0000 7f29c0b93740 1 client.18483 shutdown 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 msgr2=0x562d1202b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202b370 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f29ac035000 tx=0x7f29ac02bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 shutdown_connections 2024-03-24T09:31:59.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d1203e100 0x562d12040520 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d1202de70 0x562d1202b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022d30 0x562d1202b370 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f2996ffd640 0 client.18483 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3308675280 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d12022410 0x562d1202ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 >> 172.21.15.116:0/3308675280 conn(0x562d1201ce90 msgr2=0x562d1201ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 shutdown_connections 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 1 -- 172.21.15.116:0/3308675280 wait complete. 2024-03-24T09:31:59.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.652+0000 7f29c0b93740 20 client.18483 trim_cache size 0 max 16384 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 Processor -- start 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 -- start start 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d120397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d1202b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d12045d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562d12030bc0 con 0x562d12022410 2024-03-24T09:31:59.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562d12023140 con 0x562d12022d30 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29c0b93740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d12025c00 con 0x562d1202de70 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d120397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d120397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35720/0 (socket says 172.21.15.116:35720) 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 -- 172.21.15.116:0/3589482926 learned_addr learned my addr 172.21.15.116:0/3589482926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b7fff640 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d12045d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d1202b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 -- 172.21.15.116:0/3589482926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 msgr2=0x562d12045d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d12045d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 -- 172.21.15.116:0/3589482926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 msgr2=0x562d1202b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d1202b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 -- 172.21.15.116:0/3589482926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d11f23c60 con 0x562d12022410 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d1202b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.669+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d120397c0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f29a80cbbb0 tx=0x7f29a809a830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08d77a6ed22d428 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f299d7fa640 1 -- 172.21.15.116:0/3589482926 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a809e040 con 0x562d12022410 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f299d7fa640 1 -- 172.21.15.116:0/3589482926 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29a80a7070 con 0x562d12022410 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f299d7fa640 1 -- 172.21.15.116:0/3589482926 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a809b000 con 0x562d12022410 2024-03-24T09:31:59.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 -- 172.21.15.116:0/3589482926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 msgr2=0x562d120397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d120397c0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f29a80cbbb0 tx=0x7f29a809a830 comp rx=0 tx=0).stop 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 -- 172.21.15.116:0/3589482926 shutdown_connections 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d12045d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d1202b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3589482926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d120397c0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 -- 172.21.15.116:0/3589482926 >> 172.21.15.116:0/3589482926 conn(0x562d1201ce90 msgr2=0x562d1201bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 -- 172.21.15.116:0/3589482926 shutdown_connections 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 -- 172.21.15.116:0/3589482926 wait complete. 2024-03-24T09:31:59.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.670+0000 7f29c0b93740 1 -- start start 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 Processor -- start 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d12035e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d120321b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d120326f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562d12032c30 con 0x562d12022410 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562d12032da0 con 0x562d12022d30 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x562d12032f60 con 0x562d1202de70 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29c0b93740 10 client.0 _ll_register_callbacks cb 0x7f29ac02c820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:59.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d12035e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d12035e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35722/0 (socket says 172.21.15.116:35722) 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 -- 172.21.15.116:0/3702509754 learned_addr learned my addr 172.21.15.116:0/3702509754 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d120321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b7fff640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d120326f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 -- 172.21.15.116:0/3702509754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 msgr2=0x562d120326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d120326f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 -- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 msgr2=0x562d120321b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d120321b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 -- 172.21.15.116:0/3702509754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d11f878c0 con 0x562d12022410 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f29b77fe640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d12035e30 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f29a809a800 tx=0x7f29a80cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b457378cc2c64bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.671+0000 7f2996ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f2996ffd640 1 -- 172.21.15.116:0/3702509754 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a809e040 con 0x562d12022410 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f2996ffd640 1 -- 172.21.15.116:0/3702509754 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f29a80a9070 con 0x562d12022410 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29a809ada0 con 0x562d12022410 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f2996ffd640 1 -- 172.21.15.116:0/3702509754 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f29a80cbd80 con 0x562d12022410 2024-03-24T09:31:59.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 10 client.18489 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562d11f49970 con 0x562d12022410 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f2996ffd640 1 -- 172.21.15.116:0/3702509754 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29a809ada0 con 0x562d12022410 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f2996ffd640 1 -- 172.21.15.116:0/3702509754 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f29a801f060 con 0x562d12022410 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 10 client.18489 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 10 client.18489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 20 client.18489 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d120a54b0 0x562d120a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 4 client.18489 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T09:31:59.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562d11c88860 con 0x562d120a54b0 2024-03-24T09:31:59.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.672+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d120a54b0 0x562d120a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.673+0000 7f29b6ffd640 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d120a54b0 0x562d120a78d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8dc07c0e795c1c8 server_cookie=ae9e009453fae8bb in_seq=0 out_seq=0 2024-03-24T09:31:59.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.673+0000 7f2996ffd640 10 client.18489 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f2996ffd640 1 -- 172.21.15.116:0/3702509754 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562d11c88860 con 0x562d120a54b0 2024-03-24T09:31:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f2996ffd640 10 client.18489 handle_command_reply: tid=0 2024-03-24T09:31:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 client.18489 shutdown 2024-03-24T09:31:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 msgr2=0x562d12035e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d12035e30 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f29a809a800 tx=0x7f29a80cc040 comp rx=0 tx=0).stop 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 shutdown_connections 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x562d1202de70 0x562d120326f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x562d120a54b0 0x562d120a78d0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562d12022d30 0x562d120321b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 --2- 172.21.15.116:0/3702509754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562d12022410 0x562d12035e30 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 >> 172.21.15.116:0/3702509754 conn(0x562d1201ce90 msgr2=0x562d1201e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f2996ffd640 0 client.18489 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 shutdown_connections 2024-03-24T09:31:59.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 1 -- 172.21.15.116:0/3702509754 wait complete. 2024-03-24T09:31:59.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.674+0000 7f29c0b93740 20 client.18489 trim_cache size 0 max 16384 2024-03-24T09:31:59.685 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:31:59.685 INFO:teuthology.orchestra.run.smithi116.stdout: "return_code": 0, 2024-03-24T09:31:59.685 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_tag": "62dafd0c-eb48-4d45-af26-a2ee61ac37db", 2024-03-24T09:31:59.685 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": "asynchronous" 2024-03-24T09:31:59.685 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:31:59.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29a00a2230 msgr2=0x7f29a00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29a00a2230 0x7f29a00a46f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f29ac005ab0 tx=0x7f29ac0047a0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 msgr2=0x7f29b819c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b819c840 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f29a800e9f0 tx=0x7f29a800edf0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 shutdown_connections 2024-03-24T09:31:59.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f29a00a2230 0x7f29a00a46f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29b8106640 0x7f29b81969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29b8100ff0 0x7f29b819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 --2- 172.21.15.116:0/1735325408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29b8100630 0x7f29b819c840 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 >> 172.21.15.116:0/1735325408 conn(0x7f29b8077770 msgr2=0x7f29b8103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 shutdown_connections 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.675+0000 7f29bdfb6640 1 -- 172.21.15.116:0/1735325408 wait complete. 2024-03-24T09:31:59.688 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:31:59.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:31:59.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 -- 172.21.15.116:0/3462085481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 msgr2=0x7f55a0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/3462085481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a0106a40 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f559c009900 tx=0x7f559c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 -- 172.21.15.116:0/3462085481 shutdown_connections 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/3462085481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a0100f60 0x7f55a010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/3462085481 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55a01005a0 0x7f55a0100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/3462085481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a0106a40 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 -- 172.21.15.116:0/3462085481 >> 172.21.15.116:0/3462085481 conn(0x7f55a0077710 msgr2=0x7f55a0077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 -- 172.21.15.116:0/3462085481 shutdown_connections 2024-03-24T09:31:59.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.854+0000 7f55a7c8f640 1 -- 172.21.15.116:0/3462085481 wait complete. 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 Processor -- start 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 -- start start 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 0x7f55a019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55a0100f60 0x7f55a019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a01968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0196f40 con 0x7f55a0106660 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55a01970b0 con 0x7f55a01005a0 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a7c8f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f55a0197270 con 0x7f55a0100f60 2024-03-24T09:31:59.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a5a2b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 0x7f55a019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a5a2b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 0x7f55a019c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59312/0 (socket says 172.21.15.116:59312) 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a5a2b640 1 -- 172.21.15.116:0/2155368692 learned_addr learned my addr 172.21.15.116:0/2155368692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a622c640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a01968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a522a640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55a0100f60 0x7f55a019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a622c640 1 -- 172.21.15.116:0/2155368692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55a0100f60 msgr2=0x7f55a019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a622c640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55a0100f60 0x7f55a019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a622c640 1 -- 172.21.15.116:0/2155368692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 msgr2=0x7f55a019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a622c640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 0x7f55a019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a622c640 1 -- 172.21.15.116:0/2155368692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f559c009560 con 0x7f55a0106660 2024-03-24T09:31:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.855+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 0x7f55a019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.856+0000 7f55a622c640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a01968f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f5598009fd0 tx=0x7f559800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.856+0000 7f5596ffd640 1 -- 172.21.15.116:0/2155368692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559800ce20 con 0x7f55a0106660 2024-03-24T09:31:59.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.856+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a0197550 con 0x7f55a0106660 2024-03-24T09:31:59.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.856+0000 7f5596ffd640 1 -- 172.21.15.116:0/2155368692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5598005020 con 0x7f55a0106660 2024-03-24T09:31:59.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.856+0000 7f5596ffd640 1 -- 172.21.15.116:0/2155368692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598010690 con 0x7f55a0106660 2024-03-24T09:31:59.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.856+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a01a37e0 con 0x7f55a0106660 2024-03-24T09:31:59.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.859+0000 7f5596ffd640 1 -- 172.21.15.116:0/2155368692 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f5598005190 con 0x7f55a0106660 2024-03-24T09:31:59.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.859+0000 7f5596ffd640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f558c0a22b0 0x7f558c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.859+0000 7f55aa874740 1 Processor -- start 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f5596ffd640 1 -- 172.21.15.116:0/2155368692 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f5598014070 con 0x7f55a0106660 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 -- start start 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f558c0a22b0 0x7f558c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f578731ab0 0x55f578731e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f5787323d0 0x55f57873f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f5787420c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f578656f30 con 0x55f578731ab0 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f5786a0810 con 0x55f57873fcd0 2024-03-24T09:31:59.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55aa874740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f57869dd40 con 0x55f5787323d0 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a622c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f5787420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a5a2b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f578731ab0 0x55f578731e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a622c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f5787420c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:59316/0 (socket says 172.21.15.116:59316) 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a622c640 1 -- 172.21.15.116:0/3272465249 learned_addr learned my addr 172.21.15.116:0/3272465249 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a522a640 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f5787323d0 0x55f57873f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.860+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f558c0a22b0 0x7f558c0a4770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f559c009230 tx=0x7f559c004bb0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a5a2b640 1 -- 172.21.15.116:0/3272465249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f5787323d0 msgr2=0x55f57873f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f5787323d0 0x55f57873f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a5a2b640 1 -- 172.21.15.116:0/3272465249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 msgr2=0x55f5787420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f5787420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a5a2b640 1 -- 172.21.15.116:0/3272465249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f57865aea0 con 0x55f578731ab0 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a622c640 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f5787420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f578731ab0 0x55f578731e90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f559c02c740 tx=0x7f559c045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3268693001412f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f558bfff640 1 -- 172.21.15.116:0/3272465249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c02cbc0 con 0x55f578731ab0 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f558bfff640 1 -- 172.21.15.116:0/3272465249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f559c005270 con 0x55f578731ab0 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.861+0000 7f558bfff640 1 -- 172.21.15.116:0/3272465249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c003ca0 con 0x55f578731ab0 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 -- 172.21.15.116:0/3272465249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f578731ab0 msgr2=0x55f578731e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f578731ab0 0x55f578731e90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f559c02c740 tx=0x7f559c045000 comp rx=0 tx=0).stop 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 -- 172.21.15.116:0/3272465249 shutdown_connections 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f5787420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f5787323d0 0x55f57873f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 --2- 172.21.15.116:0/3272465249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f578731ab0 0x55f578731e90 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 -- 172.21.15.116:0/3272465249 >> 172.21.15.116:0/3272465249 conn(0x55f57872c900 msgr2=0x55f57872cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 -- 172.21.15.116:0/3272465249 shutdown_connections 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.862+0000 7f55aa874740 1 -- 172.21.15.116:0/3272465249 wait complete. 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 -- start start 2024-03-24T09:31:59.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 Processor -- start 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5787323d0 0x55f57873d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 0x55f57873d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f578699520 con 0x55f57873fcd0 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f57836c430 con 0x55f5787323d0 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f578328e00 con 0x55f578731ab0 2024-03-24T09:31:59.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55aa874740 10 client.0 _ll_register_callbacks cb 0x55f578637d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55a622c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 0x55f57873d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55a622c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 0x55f57873d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35754/0 (socket says 172.21.15.116:35754) 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55a622c640 1 -- 172.21.15.116:0/3239725624 learned_addr learned my addr 172.21.15.116:0/3239725624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.863+0000 7f55a522a640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5787323d0 0x55f57873d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55a622c640 1 -- 172.21.15.116:0/3239725624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 msgr2=0x55f57873cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55a622c640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55a622c640 1 -- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5787323d0 msgr2=0x55f57873d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55a622c640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5787323d0 0x55f57873d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55a622c640 1 -- 172.21.15.116:0/3239725624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a010dd20 con 0x55f57873fcd0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55a622c640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 0x55f57873d720 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f55980cbf20 tx=0x7f559809ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a7203b07cf42fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55897fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55897fa640 1 -- 172.21.15.116:0/3239725624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559809f260 con 0x55f57873fcd0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55897fa640 1 -- 172.21.15.116:0/3239725624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f55980b3e40 con 0x55f57873fcd0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f5783987e0 con 0x55f57873fcd0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55897fa640 1 -- 172.21.15.116:0/3239725624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f55980a2370 con 0x55f57873fcd0 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55aa874740 10 client.18501 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:59.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f578697ba0 con 0x55f57873fcd0 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55897fa640 1 -- 172.21.15.116:0/3239725624 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55f5783987e0 con 0x55f57873fcd0 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.864+0000 7f55897fa640 1 -- 172.21.15.116:0/3239725624 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f559801f070 con 0x55f57873fcd0 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55aa874740 10 client.18501 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55aa874740 10 client.18501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55aa874740 20 client.18501 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55aa874740 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f57874ff50 0x55f578752370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55aa874740 4 client.18501 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5783d5720 con 0x55f57874ff50 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f57874ff50 0x55f578752370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f57874ff50 0x55f578752370 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6929f1ac5cccd57b server_cookie=4d49896678928101 in_seq=0 out_seq=0 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.865+0000 7f55897fa640 10 client.18501 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.866+0000 7f55897fa640 1 -- 172.21.15.116:0/3239725624 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5783d5720 con 0x55f57874ff50 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.866+0000 7f55897fa640 10 client.18501 handle_command_reply: tid=0 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.866+0000 7f55aa874740 1 client.18501 shutdown 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 msgr2=0x55f57873d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 0x55f57873d720 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f55980cbf20 tx=0x7f559809ad10 comp rx=0 tx=0).stop 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 shutdown_connections 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f57873fcd0 0x55f57873d720 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f57874ff50 0x55f578752370 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f5787323d0 0x55f57873d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 --2- 172.21.15.116:0/3239725624 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 >> 172.21.15.116:0/3239725624 conn(0x55f57872c900 msgr2=0x55f57872e650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55897fa640 0 client.18501 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 shutdown_connections 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 1 -- 172.21.15.116:0/3239725624 wait complete. 2024-03-24T09:31:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.867+0000 7f55aa874740 20 client.18501 trim_cache size 0 max 16384 2024-03-24T09:31:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.887+0000 7f55aa874740 1 Processor -- start 2024-03-24T09:31:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 -- start start 2024-03-24T09:31:59.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f57873be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f57874b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f57861e430 con 0x55f5787323d0 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f57873aca0 con 0x55f57873fcd0 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55aa874740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f57872d6d0 con 0x55f578731ab0 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f57873be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f57873be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35760/0 (socket says 172.21.15.116:35760) 2024-03-24T09:31:59.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 -- 172.21.15.116:0/1983424035 learned_addr learned my addr 172.21.15.116:0/1983424035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a622c640 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f57874b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 -- 172.21.15.116:0/1983424035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 msgr2=0x55f57873a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 -- 172.21.15.116:0/1983424035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 msgr2=0x55f57874b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f57874b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 -- 172.21.15.116:0/1983424035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f578633b20 con 0x55f5787323d0 2024-03-24T09:31:59.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.888+0000 7f55a522a640 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f57873be00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5590005fd0 tx=0x7f559000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c9b040cb809c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f558b7fe640 1 -- 172.21.15.116:0/1983424035 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559001f070 con 0x55f5787323d0 2024-03-24T09:31:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f558b7fe640 1 -- 172.21.15.116:0/1983424035 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f5590017040 con 0x55f5787323d0 2024-03-24T09:31:59.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f558b7fe640 1 -- 172.21.15.116:0/1983424035 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5590016070 con 0x55f5787323d0 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 -- 172.21.15.116:0/1983424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 msgr2=0x55f57873be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f57873be00 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f5590005fd0 tx=0x7f559000be20 comp rx=0 tx=0).stop 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 -- 172.21.15.116:0/1983424035 shutdown_connections 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f57873fcd0 0x55f57874b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f57873be00 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 --2- 172.21.15.116:0/1983424035 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f578731ab0 0x55f57873a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.889+0000 7f55aa874740 1 -- 172.21.15.116:0/1983424035 >> 172.21.15.116:0/1983424035 conn(0x55f57872c900 msgr2=0x55f57872b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 -- 172.21.15.116:0/1983424035 shutdown_connections 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 -- 172.21.15.116:0/1983424035 wait complete. 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 -- start start 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 Processor -- start 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f578731ab0 0x55f578747ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f578733670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f57873fcd0 0x55f578733bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55a522a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f578733670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55a522a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f578733670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35762/0 (socket says 172.21.15.116:35762) 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55a522a640 1 -- 172.21.15.116:0/1463441450 learned_addr learned my addr 172.21.15.116:0/1463441450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f578731ab0 0x55f578747ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f5787342f0 con 0x55f5787323d0 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f578734460 con 0x55f578731ab0 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.890+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55f5787469e0 con 0x55f57873fcd0 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a622c640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f57873fcd0 0x55f578733bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55aa874740 10 client.0 _ll_register_callbacks cb 0x7f55980a3280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:31:59.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a5a2b640 1 -- 172.21.15.116:0/1463441450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f57873fcd0 msgr2=0x55f578733bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f57873fcd0 0x55f578733bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a5a2b640 1 -- 172.21.15.116:0/1463441450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 msgr2=0x55f578733670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f578733670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a5a2b640 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f559000b670 con 0x55f578731ab0 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a622c640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f57873fcd0 0x55f578733bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f559c005270 con 0x55f578731ab0 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a522a640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f578733670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55a5a2b640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f578731ab0 0x55f578747ce0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f559c006010 tx=0x7f559c02c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73e65ca0e1fee8ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:31:59.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55897fa640 10 client.28322 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55897fa640 1 -- 172.21.15.116:0/1463441450 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c05b040 con 0x55f578731ab0 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55897fa640 1 -- 172.21.15.116:0/1463441450 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9554+0+0 (secure 0 0 0) 0x7f559c063ba0 con 0x55f578731ab0 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55897fa640 1 -- 172.21.15.116:0/1463441450 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f559c063dc0 con 0x55f578731ab0 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55aa874740 10 client.28322 fetch_fsmap learned FSMap version 35 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.891+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55f578659700 con 0x55f578731ab0 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55897fa640 1 -- 172.21.15.116:0/1463441450 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f559c004dc0 con 0x55f578731ab0 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55aa874740 10 client.28322 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:31:59.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55aa874740 10 client.28322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55aa874740 20 client.28322 populate_metadata read hostname 'smithi116' 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55aa874740 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f5787b65b0 0x55f5787b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55aa874740 4 client.28322 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f578424890 con 0x55f5787b65b0 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.892+0000 7f55a522a640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f5787b65b0 0x55f5787b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55897fa640 10 client.28322 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55a522a640 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f5787b65b0 0x55f5787b89d0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f31f7a4965e6e7 server_cookie=8fc13e1ef0a830f6 in_seq=0 out_seq=0 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55897fa640 1 -- 172.21.15.116:0/1463441450 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f578424890 con 0x55f5787b65b0 2024-03-24T09:31:59.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55897fa640 10 client.28322 handle_command_reply: tid=0 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 client.28322 shutdown 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f578731ab0 msgr2=0x55f578747ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f578731ab0 0x55f578747ce0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f559c006010 tx=0x7f559c02c380 comp rx=0 tx=0).stop 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 shutdown_connections 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55f57873fcd0 0x55f578733bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f5787323d0 0x55f578733670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55f5787b65b0 0x55f5787b89d0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55897fa640 0 client.28322 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:31:59.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 --2- 172.21.15.116:0/1463441450 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f578731ab0 0x55f578747ce0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 >> 172.21.15.116:0/1463441450 conn(0x55f57872c900 msgr2=0x55f57873d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 shutdown_connections 2024-03-24T09:31:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 1 -- 172.21.15.116:0/1463441450 wait complete. 2024-03-24T09:31:59.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.893+0000 7f55aa874740 20 client.28322 trim_cache size 0 max 16384 2024-03-24T09:31:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:31:59.903 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": { 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: "62dafd0c-eb48-4d45-af26-a2ee61ac37db": { 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: "path": "/", 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tag": "62dafd0c-eb48-4d45-af26-a2ee61ac37db", 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: "options": "recursive,force" 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:31:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:31:59.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f558c0a22b0 msgr2=0x7f558c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f558c0a22b0 0x7f558c0a4770 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f559c009230 tx=0x7f559c004bb0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 msgr2=0x7f55a01968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a01968f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f5598009fd0 tx=0x7f559800ef10 comp rx=0 tx=0).stop 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 shutdown_connections 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a0106660 0x7f55a01968f0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f558c0a22b0 0x7f558c0a4770 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f55a0100f60 0x7f55a019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 --2- 172.21.15.116:0/2155368692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55a01005a0 0x7f55a019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:31:59.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 >> 172.21.15.116:0/2155368692 conn(0x7f55a0077710 msgr2=0x7f55a00ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:31:59.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 shutdown_connections 2024-03-24T09:31:59.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:31:59.894+0000 7f55a7c8f640 1 -- 172.21.15.116:0/2155368692 wait complete. 2024-03-24T09:31:59.908 INFO:tasks.cephfs.filesystem:scrub status for tag:62dafd0c-eb48-4d45-af26-a2ee61ac37db - {'path': '/', 'tag': '62dafd0c-eb48-4d45-af26-a2ee61ac37db', 'options': 'recursive,force'} 2024-03-24T09:32:00.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub queued for path: / 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:00 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub queued for path: / 2024-03-24T09:32:00.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:32:00.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:32:00.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:32:00.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:32:00.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:00 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub queued for path: / 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:00.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:00 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:01 smithi143 ceph-mon[35965]: pgmap v306: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:01.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:01 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:32:01.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:01 smithi143 ceph-mon[35965]: scrub summary: active paths [/] 2024-03-24T09:32:01.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:01 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:01 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:01 smithi143 ceph-mon[35965]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:01 smithi117 ceph-mon[35788]: pgmap v306: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:01 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:32:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:01 smithi117 ceph-mon[35788]: scrub summary: active paths [/] 2024-03-24T09:32:01.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:01 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:01 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:01 smithi117 ceph-mon[35788]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:01 smithi116 ceph-mon[32207]: pgmap v306: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:01 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:32:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:01 smithi116 ceph-mon[32207]: scrub summary: active paths [/] 2024-03-24T09:32:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:01 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:01 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:01.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:01 smithi116 ceph-mon[32207]: scrub summary: idle+waiting paths [/] 2024-03-24T09:32:03.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:03 smithi143 ceph-mon[35965]: pgmap v307: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:32:03.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:03 smithi117 ceph-mon[35788]: pgmap v307: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:32:03.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:03 smithi116 ceph-mon[32207]: pgmap v307: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:32:05.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:05 smithi143 ceph-mon[35965]: pgmap v308: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:32:05.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:05 smithi117 ceph-mon[35788]: pgmap v308: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:32:05.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:05 smithi116 ceph-mon[32207]: pgmap v308: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:32:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:07 smithi143 ceph-mon[35965]: pgmap v309: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:32:07.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:07 smithi117 ceph-mon[35788]: pgmap v309: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:32:07.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:07 smithi116 ceph-mon[32207]: pgmap v309: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:32:09.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:09 smithi143 ceph-mon[35965]: pgmap v310: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:32:09.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:09 smithi143 ceph-mon[35965]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T09:32:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:09 smithi117 ceph-mon[35788]: pgmap v310: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:32:09.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:09 smithi117 ceph-mon[35788]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T09:32:09.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:09 smithi116 ceph-mon[32207]: pgmap v310: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:32:09.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:09 smithi116 ceph-mon[32207]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T09:32:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:11 smithi143 ceph-mon[35965]: pgmap v312: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:32:11.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:11 smithi117 ceph-mon[35788]: pgmap v312: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:32:11.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:11 smithi116 ceph-mon[32207]: pgmap v312: 129 pgs: 129 active+clean; 150 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:32:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:13 smithi143 ceph-mon[35965]: pgmap v313: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:32:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:13 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:32:13.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:13 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:32:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:13 smithi117 ceph-mon[35788]: pgmap v313: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:32:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:13 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:32:13.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:13 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:32:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:13 smithi116 ceph-mon[32207]: pgmap v313: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:32:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:13 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:32:13.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:13 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:32:13.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.968+0000 7f27a4b0c640 1 -- 172.21.15.116:0/2854208319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 msgr2=0x7f27a010e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.968+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/2854208319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a010e530 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f2798009ca0 tx=0x7f279802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 -- 172.21.15.116:0/2854208319 shutdown_connections 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/2854208319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a010e530 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/2854208319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a0069f10 0x7f27a006a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/2854208319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a01076e0 0x7f27a0107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:13.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 -- 172.21.15.116:0/2854208319 >> 172.21.15.116:0/2854208319 conn(0x7f27a00773e0 msgr2=0x7f27a00777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 -- 172.21.15.116:0/2854208319 shutdown_connections 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 -- 172.21.15.116:0/2854208319 wait complete. 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 Processor -- start 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.969+0000 7f27a4b0c640 1 -- start start 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a0069f10 0x7f27a019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a01076e0 0x7f27a01979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:13.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27a0198030 con 0x7f27a006a8b0 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f27a01981a0 con 0x7f27a01076e0 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f27a0198360 con 0x7f27a0069f10 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34528/0 (socket says 172.21.15.116:34528) 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 -- 172.21.15.116:0/4104409628 learned_addr learned my addr 172.21.15.116:0/4104409628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279ed9d640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a01076e0 0x7f27a01979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279e59c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a0069f10 0x7f27a019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:13.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 -- 172.21.15.116:0/4104409628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a0069f10 msgr2=0x7f27a019d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a0069f10 0x7f27a019d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 -- 172.21.15.116:0/4104409628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a01076e0 msgr2=0x7f27a01979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a01076e0 0x7f27a01979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 -- 172.21.15.116:0/4104409628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2798009900 con 0x7f27a006a8b0 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279ed9d640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a01076e0 0x7f27a01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f279dd9b640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a019dd20 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f279400e9f0 tx=0x7f279400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794009700 con 0x7f27a006a8b0 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a0198640 con 0x7f27a006a8b0 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f279400cd80 con 0x7f27a006a8b0 2024-03-24T09:32:13.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2794018840 con 0x7f27a006a8b0 2024-03-24T09:32:13.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.970+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a01a4880 con 0x7f27a006a8b0 2024-03-24T09:32:13.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.971+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a0102dc0 con 0x7f27a006a8b0 2024-03-24T09:32:13.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.972+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f27940189a0 con 0x7f27a006a8b0 2024-03-24T09:32:13.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.973+0000 7f278f7fe640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27880a22d0 0x7f27880a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:13.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.973+0000 7f279e59c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27880a22d0 0x7f27880a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:13.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.973+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9642+0+0 (secure 0 0 0) 0x7f2794014070 con 0x7f27a006a8b0 2024-03-24T09:32:13.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.973+0000 7f279e59c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27880a22d0 0x7f27880a4790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27a00ffe20 tx=0x7f2790005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:13.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:13.978+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2794053280 con 0x7f27a006a8b0 2024-03-24T09:32:14.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.180+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f27a0109c30 con 0x7f27880a22d0 2024-03-24T09:32:14.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.191+0000 7f278f7fe640 1 -- 172.21.15.116:0/4104409628 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f2790011be0 con 0x7f27880a22d0 2024-03-24T09:32:14.191 INFO:teuthology.orchestra.run.smithi116.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-24T00:00:00", "created": "2024-03-24T09:22:12", "first": "2024-03-24T09:23:00", "last": "2024-03-24T09:32:00", "last_pruned": "2024-03-24T09:32:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-24T09:32:14.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27880a22d0 msgr2=0x7f27880a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27880a22d0 0x7f27880a4790 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f27a00ffe20 tx=0x7f2790005f90 comp rx=0 tx=0).stop 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 msgr2=0x7f27a019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a019dd20 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f279400e9f0 tx=0x7f279400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 shutdown_connections 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f27a01076e0 0x7f27a01979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27a006a8b0 0x7f27a019dd20 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f27880a22d0 0x7f27880a4790 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 --2- 172.21.15.116:0/4104409628 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f27a0069f10 0x7f27a019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 >> 172.21.15.116:0/4104409628 conn(0x7f27a00773e0 msgr2=0x7f27a00fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:14.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 shutdown_connections 2024-03-24T09:32:14.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.193+0000 7f27a4b0c640 1 -- 172.21.15.116:0/4104409628 wait complete. 2024-03-24T09:32:14.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-24T09:32:14.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 -- 172.21.15.116:0/3310820415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501038e0 msgr2=0x7fe050103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 --2- 172.21.15.116:0/3310820415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501038e0 0x7fe050103d60 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fe044009930 tx=0x7fe04402bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 -- 172.21.15.116:0/3310820415 shutdown_connections 2024-03-24T09:32:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 --2- 172.21.15.116:0/3310820415 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0501042a0 0x7fe0501107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 --2- 172.21.15.116:0/3310820415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501038e0 0x7fe050103d60 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 --2- 172.21.15.116:0/3310820415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501098f0 0x7fe050109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 -- 172.21.15.116:0/3310820415 >> 172.21.15.116:0/3310820415 conn(0x7fe0500ff390 msgr2=0x7fe0501017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 -- 172.21.15.116:0/3310820415 shutdown_connections 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 -- 172.21.15.116:0/3310820415 wait complete. 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.307+0000 7fe05548f640 1 Processor -- start 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 -- start start 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501038e0 0x7fe050199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 0x7fe05019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0501098f0 0x7fe05019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:14.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe05019ed30 con 0x7fe0501042a0 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe05019eea0 con 0x7fe0501098f0 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe05548f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe05019f060 con 0x7fe0501038e0 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 0x7fe05019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 0x7fe05019a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34544/0 (socket says 172.21.15.116:34544) 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 -- 172.21.15.116:0/1921478135 learned_addr learned my addr 172.21.15.116:0/1921478135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04effd640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501038e0 0x7fe050199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04f7fe640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0501098f0 0x7fe05019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 -- 172.21.15.116:0/1921478135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501038e0 msgr2=0x7fe050199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501038e0 0x7fe050199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 -- 172.21.15.116:0/1921478135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0501098f0 msgr2=0x7fe05019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0501098f0 0x7fe05019e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 -- 172.21.15.116:0/1921478135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe044009560 con 0x7fe0501042a0 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04effd640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501038e0 0x7fe050199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.308+0000 7fe04e7fc640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 0x7fe05019a3b0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fe044009230 tx=0x7fe044005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.309+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044004350 con 0x7fe0501042a0 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.309+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe04402de90 con 0x7fe0501042a0 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.309+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe05019f2e0 con 0x7fe0501042a0 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.309+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe044005220 con 0x7fe0501042a0 2024-03-24T09:32:14.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.309+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0501a6bc0 con 0x7fe0501042a0 2024-03-24T09:32:14.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.310+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe05004f470 con 0x7fe0501042a0 2024-03-24T09:32:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.311+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe044003c30 con 0x7fe0501042a0 2024-03-24T09:32:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.311+0000 7fe037fff640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe0380a22d0 0x7fe0380a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.312+0000 7fe04effd640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe0380a22d0 0x7fe0380a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.312+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fe0440f0310 con 0x7fe0501042a0 2024-03-24T09:32:14.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.312+0000 7fe04effd640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe0380a22d0 0x7fe0380a4790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe040009f40 tx=0x7fe040005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:14.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.316+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe0440b8ca0 con 0x7fe0501042a0 2024-03-24T09:32:14.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.513+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 --> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fe05010be80 con 0x7fe0380a22d0 2024-03-24T09:32:14.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.519+0000 7fe037fff640 1 -- 172.21.15.116:0/1921478135 <== mgr.15543 v2:172.21.15.116:6800/2615204634 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fe05010be80 con 0x7fe0380a22d0 2024-03-24T09:32:14.520 INFO:teuthology.orchestra.run.smithi116.stdout:/ 1m 6m3h 2024-03-24T09:32:14.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe0380a22d0 msgr2=0x7fe0380a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe0380a22d0 0x7fe0380a4790 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe040009f40 tx=0x7fe040005f90 comp rx=0 tx=0).stop 2024-03-24T09:32:14.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 msgr2=0x7fe05019a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:14.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 0x7fe05019a3b0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fe044009230 tx=0x7fe044005dd0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 shutdown_connections 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0501098f0 0x7fe05019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0501042a0 0x7fe05019a3b0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe0380a22d0 0x7fe0380a4790 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 --2- 172.21.15.116:0/1921478135 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe0501038e0 0x7fe050199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.521+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 >> 172.21.15.116:0/1921478135 conn(0x7fe0500ff390 msgr2=0x7fe0500ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.522+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 shutdown_connections 2024-03-24T09:32:14.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:14.522+0000 7fe05548f640 1 -- 172.21.15.116:0/1921478135 wait complete. 2024-03-24T09:32:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:14 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:32:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:14 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:32:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:14 smithi143 ceph-mon[35965]: scrub summary: idle 2024-03-24T09:32:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:14 smithi143 ceph-mon[35965]: pgmap v314: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:32:14.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:14 smithi143 ceph-mon[35965]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T09:32:14.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:14 smithi143 ceph-mon[35965]: from='client.18513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:32:14.531 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:14 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:32:14.531 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:14 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:32:14.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:14 smithi116 ceph-mon[32207]: scrub summary: idle 2024-03-24T09:32:14.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:14 smithi116 ceph-mon[32207]: pgmap v314: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:32:14.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:14 smithi116 ceph-mon[32207]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T09:32:14.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:14 smithi116 ceph-mon[32207]: from='client.18513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:32:14.533 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-24T09:32:14.544 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-24T09:32:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:14 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:32:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:14 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:32:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:14 smithi117 ceph-mon[35788]: scrub summary: idle 2024-03-24T09:32:14.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:14 smithi117 ceph-mon[35788]: pgmap v314: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:32:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:14 smithi117 ceph-mon[35788]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T09:32:14.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:14 smithi117 ceph-mon[35788]: from='client.18513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:32:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:15 smithi143 ceph-mon[35965]: from='client.18519 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:32:15.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:15 smithi117 ceph-mon[35788]: from='client.18519 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:32:15.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:15 smithi116 ceph-mon[32207]: from='client.18519 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T09:32:16.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:16 smithi143 ceph-mon[35965]: pgmap v316: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:32:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:16 smithi117 ceph-mon[35788]: pgmap v316: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:32:16.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:16 smithi116 ceph-mon[32207]: pgmap v316: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:32:18.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:18 smithi143 ceph-mon[35965]: pgmap v317: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 221 B/s wr, 14 op/s 2024-03-24T09:32:18.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:18 smithi117 ceph-mon[35788]: pgmap v317: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 221 B/s wr, 14 op/s 2024-03-24T09:32:18.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:18 smithi116 ceph-mon[32207]: pgmap v317: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 221 B/s wr, 14 op/s 2024-03-24T09:32:19.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:19 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:32:19.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:19 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:32:19.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:19 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:32:20.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:20 smithi143 ceph-mon[35965]: pgmap v318: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-24T09:32:20.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:20 smithi117 ceph-mon[35788]: pgmap v318: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-24T09:32:20.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:20 smithi116 ceph-mon[32207]: pgmap v318: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-24T09:32:22.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.276 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:32:22.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:32:22.277 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:21 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:32:22.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:21 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.320 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T09:32:22.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:32:22.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:32:22.321 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:21 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:32:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: pgmap v319: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T09:32:23.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-24T09:32:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-03-24T09:32:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: Unable to set osd_memory_target on smithi117 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-03-24T09:32:23.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:22 smithi143 ceph-mon[35965]: Unable to set osd_memory_target on smithi143 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: pgmap v319: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T09:32:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-24T09:32:23.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-03-24T09:32:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: Unable to set osd_memory_target on smithi117 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-03-24T09:32:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:22 smithi117 ceph-mon[35788]: Unable to set osd_memory_target on smithi143 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: pgmap v319: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: Unable to set osd_memory_target on smithi117 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: Adjusting osd_memory_target on smithi143 to 176.2M 2024-03-24T09:32:23.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:22 smithi116 ceph-mon[32207]: Unable to set osd_memory_target on smithi143 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-24T09:32:25.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:24 smithi143 ceph-mon[35965]: pgmap v320: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T09:32:25.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:24 smithi117 ceph-mon[35788]: pgmap v320: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T09:32:25.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:24 smithi116 ceph-mon[32207]: pgmap v320: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-24T09:32:27.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:26 smithi143 ceph-mon[35965]: pgmap v321: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 89 B/s wr, 11 op/s 2024-03-24T09:32:27.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:26 smithi117 ceph-mon[35788]: pgmap v321: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 89 B/s wr, 11 op/s 2024-03-24T09:32:27.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:26 smithi116 ceph-mon[32207]: pgmap v321: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 89 B/s wr, 11 op/s 2024-03-24T09:32:29.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:28 smithi143 ceph-mon[35965]: pgmap v322: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-24T09:32:29.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:28 smithi117 ceph-mon[35788]: pgmap v322: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-24T09:32:29.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:28 smithi116 ceph-mon[32207]: pgmap v322: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 15 op/s 2024-03-24T09:32:29.909 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T09:32:29.909 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T09:32:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 -- 172.21.15.116:0/3100758746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc1005c0 msgr2=0x7fbedc100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/3100758746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc1005c0 0x7fbedc100a40 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009930 tx=0x7fbed002bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 -- 172.21.15.116:0/3100758746 shutdown_connections 2024-03-24T09:32:30.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/3100758746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbedc100f80 0x7fbedc10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/3100758746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc1005c0 0x7fbedc100a40 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/3100758746 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc106660 0x7fbedc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 -- 172.21.15.116:0/3100758746 >> 172.21.15.116:0/3100758746 conn(0x7fbedc076990 msgr2=0x7fbedc076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 -- 172.21.15.116:0/3100758746 shutdown_connections 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.027+0000 7fbee0b7a640 1 -- 172.21.15.116:0/3100758746 wait complete. 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 Processor -- start 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 -- start start 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc1005c0 0x7fbedc19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 0x7fbedc19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbedc106660 0x7fbedc197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc198080 con 0x7fbedc100f80 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc1981f0 con 0x7fbedc106660 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbee0b7a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbedc1983b0 con 0x7fbedc1005c0 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 0x7fbedc19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 0x7fbedc19ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50692/0 (socket says 172.21.15.116:50692) 2024-03-24T09:32:30.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2702976382 learned_addr learned my addr 172.21.15.116:0/2702976382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc1005c0 0x7fbedc19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.028+0000 7fbedad9d640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbedc106660 0x7fbedc197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2702976382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc1005c0 msgr2=0x7fbedc19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc1005c0 0x7fbedc19d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2702976382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbedc106660 msgr2=0x7fbedc197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbedc106660 0x7fbedc197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2702976382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc009490 con 0x7fbedc100f80 2024-03-24T09:32:30.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc1005c0 0x7fbedc19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 0x7fbedc19ddc0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009230 tx=0x7fbed0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbecb7fe640 1 -- 172.21.15.116:0/2702976382 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0004350 con 0x7fbedc100f80 2024-03-24T09:32:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbecb7fe640 1 -- 172.21.15.116:0/2702976382 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbed002de90 con 0x7fbedc100f80 2024-03-24T09:32:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed0009560 con 0x7fbedc100f80 2024-03-24T09:32:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbecb7fe640 1 -- 172.21.15.116:0/2702976382 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0005220 con 0x7fbedc100f80 2024-03-24T09:32:30.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.029+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbedc1a4740 con 0x7fbedc100f80 2024-03-24T09:32:30.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbecb7fe640 1 -- 172.21.15.116:0/2702976382 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fbed0003c30 con 0x7fbedc100f80 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 Processor -- start 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 -- start start 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9b9f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 0x55ede9bac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9bacae0 0x55ede9baeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ede9ac3f30 con 0x55ede9bacae0 2024-03-24T09:32:30.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ede9b0d810 con 0x55ede9b9ee80 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbee3836740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ede9b0ad40 con 0x55ede9b9f7a0 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.032+0000 7fbecb7fe640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbec40a2230 0x7fbec40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 0x55ede9bac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9bacae0 0x55ede9baeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 0x55ede9bac5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:50376/0 (socket says 172.21.15.116:50376) 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 -- 172.21.15.116:0/3994842938 learned_addr learned my addr 172.21.15.116:0/3994842938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbedad9d640 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9b9f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbedad9d640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbec40a2230 0x7fbec40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbecb7fe640 1 -- 172.21.15.116:0/2702976382 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fbed00f02e0 con 0x7fbedc100f80 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 -- 172.21.15.116:0/3994842938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 msgr2=0x55ede9b9f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9b9f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 -- 172.21.15.116:0/3994842938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9bacae0 msgr2=0x55ede9baeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9bacae0 0x55ede9baeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 -- 172.21.15.116:0/3994842938 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ede9ac7ea0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbeda59c640 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 0x55ede9bac5a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbecc011680 tx=0x7fbecc011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b89d4875d98e1eab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbedad9d640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbec40a2230 0x7fbec40a46f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbed4005c40 tx=0x7fbed4009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbec8ff9640 1 -- 172.21.15.116:0/3994842938 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc005020 con 0x55ede9b9f7a0 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbec8ff9640 1 -- 172.21.15.116:0/3994842938 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbecc005630 con 0x55ede9b9f7a0 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.033+0000 7fbec8ff9640 1 -- 172.21.15.116:0/3994842938 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbecc00f920 con 0x55ede9b9f7a0 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 -- 172.21.15.116:0/3994842938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 msgr2=0x55ede9bac5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 0x55ede9bac5a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbecc011680 tx=0x7fbecc011a80 comp rx=0 tx=0).stop 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 -- 172.21.15.116:0/3994842938 shutdown_connections 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9bacae0 0x55ede9baeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9f7a0 0x55ede9bac5a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 --2- 172.21.15.116:0/3994842938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9b9f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 -- 172.21.15.116:0/3994842938 >> 172.21.15.116:0/3994842938 conn(0x55ede9b99900 msgr2=0x55ede9b99ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 -- 172.21.15.116:0/3994842938 shutdown_connections 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 -- 172.21.15.116:0/3994842938 wait complete. 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 -- start start 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.034+0000 7fbee3836740 1 Processor -- start 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9ba9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9ba9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9baa530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ede9b06520 con 0x55ede9b9f7a0 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ede97d9430 con 0x55ede9b9ee80 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ede9795e00 con 0x55ede9bacae0 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 10 client.0 _ll_register_callbacks cb 0x55ede9aa4d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9ba9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbeda59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9ba9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbeda59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9ba9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57342/0 (socket says 172.21.15.116:57342) 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbeda59c640 1 -- 172.21.15.116:0/2665457931 learned_addr learned my addr 172.21.15.116:0/2665457931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbedad9d640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9baa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2665457931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 msgr2=0x55ede9baa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9baa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 msgr2=0x55ede9ba9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9ba9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2665457931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc009e10 con 0x55ede9b9f7a0 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9ba9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbedad9d640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9baa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9ba9ff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fbed00ed730 tx=0x7fbed008c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3520a80f1170fb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbec1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2665457931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00c4ee0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2665457931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fbed00bc9d0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.035+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ede98057e0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2665457931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbed00f0150 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 10 client.18531 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ede9b04ba0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2665457931 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ede98057e0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2665457931 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fbed003c070 con 0x55ede9b9f7a0 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 10 client.18531 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 10 client.18531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 20 client.18531 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9bbce20 0x55ede9bbf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 4 client.18531 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ede9842720 con 0x55ede9bbce20 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.036+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9bbce20 0x55ede9bbf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbec1ffb640 10 client.18531 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9bbce20 0x55ede9bbf240 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42103789e51cb49b server_cookie=37d5903dec8c287e in_seq=0 out_seq=0 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2665457931 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ede9842720 con 0x55ede9bbce20 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbec1ffb640 10 client.18531 handle_command_reply: tid=0 2024-03-24T09:32:30.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 client.18531 shutdown 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 msgr2=0x55ede9ba9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9ba9ff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fbed00ed730 tx=0x7fbed008c830 comp rx=0 tx=0).stop 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 shutdown_connections 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9baa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9ba9ff0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9bbce20 0x55ede9bbf240 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 --2- 172.21.15.116:0/2665457931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9ba9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 >> 172.21.15.116:0/2665457931 conn(0x55ede9b99900 msgr2=0x55ede9b9b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbec1ffb640 0 client.18531 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:30.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 shutdown_connections 2024-03-24T09:32:30.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 1 -- 172.21.15.116:0/2665457931 wait complete. 2024-03-24T09:32:30.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.037+0000 7fbee3836740 20 client.18531 trim_cache size 0 max 16384 2024-03-24T09:32:30.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.054+0000 7fbee3836740 1 Processor -- start 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.054+0000 7fbee3836740 1 -- start start 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbee3836740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9ee80 0x55ede9bb8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbee3836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9baa350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbee3836740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9bacae0 0x55ede9bc4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbee3836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ede9a8b430 con 0x55ede9b9f7a0 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbee3836740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ede9b99730 con 0x55ede9bacae0 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbee3836740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ede9bb8970 con 0x55ede9b9ee80 2024-03-24T09:32:30.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9baa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9baa350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50716/0 (socket says 172.21.15.116:50716) 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 -- 172.21.15.116:0/3138170230 learned_addr learned my addr 172.21.15.116:0/3138170230 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbedad9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9bacae0 0x55ede9bc4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbeda59c640 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9ee80 0x55ede9bb8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 -- 172.21.15.116:0/3138170230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9ee80 msgr2=0x55ede9bb8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9ee80 0x55ede9bb8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 -- 172.21.15.116:0/3138170230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9bacae0 msgr2=0x55ede9bc4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9bacae0 0x55ede9bc4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 -- 172.21.15.116:0/3138170230 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ede9aa0b20 con 0x55ede9b9f7a0 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9baa350 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fbed00bc6d0 tx=0x7fbed002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ca2e94d04a1622 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbec3fff640 1 -- 172.21.15.116:0/3138170230 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00a72e0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbec3fff640 1 -- 172.21.15.116:0/3138170230 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbed00dc040 con 0x55ede9b9f7a0 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.055+0000 7fbec3fff640 1 -- 172.21.15.116:0/3138170230 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00c4f40 con 0x55ede9b9f7a0 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- 172.21.15.116:0/3138170230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 msgr2=0x55ede9baa350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9baa350 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fbed00bc6d0 tx=0x7fbed002bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- 172.21.15.116:0/3138170230 shutdown_connections 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9bacae0 0x55ede9bc4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9baa350 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- 172.21.15.116:0/3138170230 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9b9ee80 0x55ede9bb8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- 172.21.15.116:0/3138170230 >> 172.21.15.116:0/3138170230 conn(0x55ede9b99900 msgr2=0x55ede9b98b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- 172.21.15.116:0/3138170230 shutdown_connections 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- 172.21.15.116:0/3138170230 wait complete. 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- start start 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 Processor -- start 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9bb06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9bb0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9c22100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ede9bb1500 con 0x55ede9b9f7a0 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ede9c22640 con 0x55ede9b9ee80 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ede9c22780 con 0x55ede9bacae0 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbee3836740 10 client.0 _ll_register_callbacks cb 0x7fbed0009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9bb0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbed9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9bb0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50720/0 (socket says 172.21.15.116:50720) 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2643124780 learned_addr learned my addr 172.21.15.116:0/2643124780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.056+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9bb06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbedad9d640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9c22100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2643124780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 msgr2=0x55ede9c22100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9c22100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 msgr2=0x55ede9bb06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9bb06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbed9d9b640 1 -- 172.21.15.116:0/2643124780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ede9b04ba0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbed9d9b640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9bb0c30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fbed0005ec0 tx=0x7fbed00c4f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee93f48eacf5985 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbec1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2643124780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed00a72e0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2643124780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fbed00a70b0 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ede9b9a280 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2643124780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbed00f0150 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbee3836740 10 client.18537 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ede9ac6700 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2643124780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ede9b9a280 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2643124780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fbed00cd070 con 0x55ede9b9f7a0 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.057+0000 7fbee3836740 10 client.18537 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 10 client.18537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:32:30.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 20 client.18537 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9c23980 0x55ede9c25d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 4 client.18537 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fbecc005630 con 0x55ede9c23980 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9c23980 0x55ede9c25d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbec1ffb640 10 client.18537 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbeda59c640 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9c23980 0x55ede9c25d70 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6eada1c4d88715 server_cookie=3a5e6d6b314a62bf in_seq=0 out_seq=0 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbec1ffb640 1 -- 172.21.15.116:0/2643124780 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fbecc005630 con 0x55ede9c23980 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbec1ffb640 10 client.18537 handle_command_reply: tid=0 2024-03-24T09:32:30.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 client.18537 shutdown 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 msgr2=0x55ede9bb0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9bb0c30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fbed0005ec0 tx=0x7fbed00c4f90 comp rx=0 tx=0).stop 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 shutdown_connections 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ede9bacae0 0x55ede9c22100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ede9b9f7a0 0x55ede9bb0c30 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55ede9c23980 0x55ede9c25d70 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 --2- 172.21.15.116:0/2643124780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ede9b9ee80 0x55ede9bb06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbec1ffb640 0 client.18537 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:30.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 >> 172.21.15.116:0/2643124780 conn(0x55ede9b99900 msgr2=0x55ede9b9a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 shutdown_connections 2024-03-24T09:32:30.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 1 -- 172.21.15.116:0/2643124780 wait complete. 2024-03-24T09:32:30.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.058+0000 7fbee3836740 20 client.18537 trim_cache size 0 max 16384 2024-03-24T09:32:30.069 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:32:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: "status": "no active scrubs running", 2024-03-24T09:32:30.069 INFO:teuthology.orchestra.run.smithi116.stdout: "scrubs": {} 2024-03-24T09:32:30.070 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:32:30.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbec40a2230 msgr2=0x7fbec40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbec40a2230 0x7fbec40a46f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbed4005c40 tx=0x7fbed4009500 comp rx=0 tx=0).stop 2024-03-24T09:32:30.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 msgr2=0x7fbedc19ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 0x7fbedc19ddc0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009230 tx=0x7fbed0005dd0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 shutdown_connections 2024-03-24T09:32:30.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbedc106660 0x7fbedc197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbedc100f80 0x7fbedc19ddc0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbec40a2230 0x7fbec40a46f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 --2- 172.21.15.116:0/2702976382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbedc1005c0 0x7fbedc19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 >> 172.21.15.116:0/2702976382 conn(0x7fbedc076990 msgr2=0x7fbedc0ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 shutdown_connections 2024-03-24T09:32:30.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.059+0000 7fbee0b7a640 1 -- 172.21.15.116:0/2702976382 wait complete. 2024-03-24T09:32:30.072 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T09:32:30.073 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:32:30.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 -- 172.21.15.116:0/4180806066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 msgr2=0x7f04d410b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 --2- 172.21.15.116:0/4180806066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 0x7f04d410b780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f04c8009900 tx=0x7f04c802bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 -- 172.21.15.116:0/4180806066 shutdown_connections 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 --2- 172.21.15.116:0/4180806066 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d410e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 --2- 172.21.15.116:0/4180806066 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 0x7f04d410b780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 --2- 172.21.15.116:0/4180806066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d4102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 -- 172.21.15.116:0/4180806066 >> 172.21.15.116:0/4180806066 conn(0x7f04d4077710 msgr2=0x7f04d4077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 -- 172.21.15.116:0/4180806066 shutdown_connections 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 -- 172.21.15.116:0/4180806066 wait complete. 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 Processor -- start 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.174+0000 7f04d9acd640 1 -- start start 2024-03-24T09:32:30.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 0x7f04d419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41982b0 con 0x7f04d410bd50 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f04d4198420 con 0x7f04d4102980 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f04d41985e0 con 0x7f04d41032a0 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d37fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d419da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57386/0 (socket says 172.21.15.116:57386) 2024-03-24T09:32:30.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50732/0 (socket says 172.21.15.116:50732) 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d37fe640 1 -- 172.21.15.116:0/3652937782 learned_addr learned my addr 172.21.15.116:0/3652937782 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d2ffd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 0x7f04d419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 -- 172.21.15.116:0/3652937782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 msgr2=0x7f04d419dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 0x7f04d419dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 -- 172.21.15.116:0/3652937782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 msgr2=0x7f04d419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 -- 172.21.15.116:0/3652937782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c8009560 con 0x7f04d410bd50 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d37fe640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d3fff640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d4197b20 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009fd0 tx=0x7f04cc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc00ce20 con 0x7f04d410bd50 2024-03-24T09:32:30.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d41988c0 con 0x7f04d410bd50 2024-03-24T09:32:30.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f04cc005020 con 0x7f04d410bd50 2024-03-24T09:32:30.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04cc010690 con 0x7f04d410bd50 2024-03-24T09:32:30.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.175+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d41a49e0 con 0x7f04d410bd50 2024-03-24T09:32:30.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.176+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d40690b0 con 0x7f04d410bd50 2024-03-24T09:32:30.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.178+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f04cc005190 con 0x7f04d410bd50 2024-03-24T09:32:30.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.178+0000 7f04d0ff9640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f04bc0a2230 0x7f04bc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.178+0000 7f04d37fe640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f04bc0a2230 0x7f04bc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.179+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f04cc014070 con 0x7f04d410bd50 2024-03-24T09:32:30.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.179+0000 7f04d37fe640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f04bc0a2230 0x7f04bc0a46f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04c4011550 tx=0x7f04c4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.182+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f04cc0973c0 con 0x7f04d410bd50 2024-03-24T09:32:30.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.393+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04d4068500 con 0x7f04d410bd50 2024-03-24T09:32:30.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.394+0000 7f04d0ff9640 1 -- 172.21.15.116:0/3652937782 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f04cc0b2ed0 con 0x7f04d410bd50 2024-03-24T09:32:30.395 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:32:30.395 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:32:30.395 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f04bc0a2230 msgr2=0x7f04bc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f04bc0a2230 0x7f04bc0a46f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04c4011550 tx=0x7f04c4005f90 comp rx=0 tx=0).stop 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 msgr2=0x7f04d4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d4197b20 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f04cc009fd0 tx=0x7f04cc00ef10 comp rx=0 tx=0).stop 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 shutdown_connections 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04d410bd50 0x7f04d4197b20 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f04bc0a2230 0x7f04bc0a46f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f04d41032a0 0x7f04d419dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 --2- 172.21.15.116:0/3652937782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04d4102980 0x7f04d419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 >> 172.21.15.116:0/3652937782 conn(0x7f04d4077710 msgr2=0x7f04d40fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 shutdown_connections 2024-03-24T09:32:30.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.396+0000 7f04d9acd640 1 -- 172.21.15.116:0/3652937782 wait complete. 2024-03-24T09:32:30.408 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T09:32:30.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 -- 172.21.15.116:0/3729852772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 msgr2=0x7f2c14107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/3729852772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14107a90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f2c10009930 tx=0x7f2c1002bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 -- 172.21.15.116:0/3729852772 shutdown_connections 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/3729852772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c14102060 0x7f2c1410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/3729852772 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c141016a0 0x7f2c14101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/3729852772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14107a90 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 -- 172.21.15.116:0/3729852772 >> 172.21.15.116:0/3729852772 conn(0x7f2c14078b10 msgr2=0x7f2c14078f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 -- 172.21.15.116:0/3729852772 shutdown_connections 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.511+0000 7f2c1b33e640 1 -- 172.21.15.116:0/3729852772 wait complete. 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 Processor -- start 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 -- start start 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c141016a0 0x7f2c1419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c14102060 0x7f2c1419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c141980e0 con 0x7f2c141076b0 2024-03-24T09:32:30.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2c14198250 con 0x7f2c141016a0 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c1b33e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2c14198410 con 0x7f2c14102060 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c198db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c198db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50752/0 (socket says 172.21.15.116:50752) 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c190da640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c141016a0 0x7f2c1419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c198db640 1 -- 172.21.15.116:0/646506208 learned_addr learned my addr 172.21.15.116:0/646506208 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.512+0000 7f2c188d9640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c14102060 0x7f2c1419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c198db640 1 -- 172.21.15.116:0/646506208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c14102060 msgr2=0x7f2c1419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c198db640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c14102060 0x7f2c1419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c198db640 1 -- 172.21.15.116:0/646506208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c141016a0 msgr2=0x7f2c1419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c198db640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c141016a0 0x7f2c1419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c198db640 1 -- 172.21.15.116:0/646506208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c10009560 con 0x7f2c141076b0 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c190da640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c141016a0 0x7f2c1419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c198db640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14197a90 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00eb10 tx=0x7f2c0c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c0a7fc640 1 -- 172.21.15.116:0/646506208 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c004b20 con 0x7f2c141076b0 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c141986f0 con 0x7f2c141076b0 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c0a7fc640 1 -- 172.21.15.116:0/646506208 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c0c004c80 con 0x7f2c141076b0 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c0a7fc640 1 -- 172.21.15.116:0/646506208 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c010690 con 0x7f2c141076b0 2024-03-24T09:32:30.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.513+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c141a4980 con 0x7f2c141076b0 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.515+0000 7f2c0a7fc640 1 -- 172.21.15.116:0/646506208 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2c0c005020 con 0x7f2c141076b0 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.515+0000 7f2c1e001740 1 Processor -- start 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.515+0000 7f2c1e001740 1 -- start start 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x5600471027d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c0a7fc640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2c000a2230 0x7f2c000a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x56004710dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c1e001740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560047070840 con 0x560047102d10 2024-03-24T09:32:30.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c1e001740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56004706dd70 con 0x5600471023f0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c1e001740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560047069550 con 0x56004710b8f0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c188d9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x5600471027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c198db640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2c000a2230 0x7f2c000a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50762/0 (socket says 172.21.15.116:50762) 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 -- 172.21.15.116:0/2528775608 learned_addr learned my addr 172.21.15.116:0/2528775608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c188d9640 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x56004710dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c0a7fc640 1 -- 172.21.15.116:0/646506208 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f2c0c014070 con 0x7f2c141076b0 2024-03-24T09:32:30.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 -- 172.21.15.116:0/2528775608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 msgr2=0x56004710dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x56004710dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 -- 172.21.15.116:0/2528775608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 msgr2=0x5600471027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x5600471027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 -- 172.21.15.116:0/2528775608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560046ffede0 con 0x560047102d10 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c188d9640 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x5600471027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c198db640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2c000a2230 0x7f2c000a46f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cbde0 tx=0x7f2c0c00efd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.516+0000 7f2c190da640 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710b3b0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f2c1002c680 tx=0x7f2c10005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9e055cb66f98b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2bff7fe640 1 -- 172.21.15.116:0/2528775608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1002c770 con 0x560047102d10 2024-03-24T09:32:30.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2bff7fe640 1 -- 172.21.15.116:0/2528775608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c1002de90 con 0x560047102d10 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2bff7fe640 1 -- 172.21.15.116:0/2528775608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10003950 con 0x560047102d10 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 -- 172.21.15.116:0/2528775608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 msgr2=0x56004710b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710b3b0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f2c1002c680 tx=0x7f2c10005de0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 -- 172.21.15.116:0/2528775608 shutdown_connections 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x56004710dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710b3b0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2528775608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x5600471027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 -- 172.21.15.116:0/2528775608 >> 172.21.15.116:0/2528775608 conn(0x5600470fcb20 msgr2=0x5600470fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 -- 172.21.15.116:0/2528775608 shutdown_connections 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.517+0000 7f2c1e001740 1 -- 172.21.15.116:0/2528775608 wait complete. 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 -- start start 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 Processor -- start 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 0x560047108920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560047102d10 0x560047108e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x5600471093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560046d3c430 con 0x5600471023f0 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560046cf8e00 con 0x560047102d10 2024-03-24T09:32:30.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560046fee590 con 0x56004710b8f0 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c1e001740 10 client.0 _ll_register_callbacks cb 0x56004705dfc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 0x560047108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 0x560047108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50772/0 (socket says 172.21.15.116:50772) 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 -- 172.21.15.116:0/1139391269 learned_addr learned my addr 172.21.15.116:0/1139391269 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c190da640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560047102d10 0x560047108e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x5600471093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 -- 172.21.15.116:0/1139391269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 msgr2=0x5600471093a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x5600471093a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 -- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560047102d10 msgr2=0x560047108e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.518+0000 7f2c188d9640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560047102d10 0x560047108e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c188d9640 1 -- 172.21.15.116:0/1139391269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1410ee70 con 0x5600471023f0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c188d9640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 0x560047108920 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f2c04003930 tx=0x7f2c04009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6a60505c0b73bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2bfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2bfcff9640 1 -- 172.21.15.116:0/1139391269 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04003eb0 con 0x5600471023f0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2bfcff9640 1 -- 172.21.15.116:0/1139391269 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f2c0400fad0 con 0x5600471023f0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560046d687e0 con 0x5600471023f0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2bfcff9640 1 -- 172.21.15.116:0/1139391269 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2c0400fc90 con 0x5600471023f0 2024-03-24T09:32:30.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 10 client.18555 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560047067bd0 con 0x5600471023f0 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2bfcff9640 1 -- 172.21.15.116:0/1139391269 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x560046d687e0 con 0x5600471023f0 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2bfcff9640 1 -- 172.21.15.116:0/1139391269 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2c04005310 con 0x5600471023f0 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 10 client.18555 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 10 client.18555 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 20 client.18555 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x56004711b9f0 0x56004711de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 4 client.18555 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:32:30.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.519+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560046da5720 con 0x56004711b9f0 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.520+0000 7f2c190da640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x56004711b9f0 0x56004711de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.520+0000 7f2bfcff9640 10 client.18555 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.520+0000 7f2c190da640 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x56004711b9f0 0x56004711de10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=51425a6c7175c7d server_cookie=70ce715abfb01143 in_seq=0 out_seq=0 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2bfcff9640 1 -- 172.21.15.116:0/1139391269 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560046da5720 con 0x56004711b9f0 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2bfcff9640 10 client.18555 handle_command_reply: tid=0 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 client.18555 shutdown 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 msgr2=0x560047108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 0x560047108920 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f2c04003930 tx=0x7f2c04009ac0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 shutdown_connections 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x5600471093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560047102d10 0x560047108e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x56004711b9f0 0x56004711de10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 --2- 172.21.15.116:0/1139391269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5600471023f0 0x560047108920 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2bfcff9640 0 client.18555 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 >> 172.21.15.116:0/1139391269 conn(0x5600470fcb20 msgr2=0x5600470fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 shutdown_connections 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 1 -- 172.21.15.116:0/1139391269 wait complete. 2024-03-24T09:32:30.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.521+0000 7f2c1e001740 20 client.18555 trim_cache size 0 max 16384 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 Processor -- start 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 -- start start 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x560047117150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x560047123700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047113770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560046d68810 con 0x560047102d10 2024-03-24T09:32:30.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560047109db0 con 0x5600471023f0 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c1e001740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x560047117690 con 0x56004710b8f0 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c190da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x560047123700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c188d9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x560047117150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c190da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x560047123700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50778/0 (socket says 172.21.15.116:50778) 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c190da640 1 -- 172.21.15.116:0/2353821579 learned_addr learned my addr 172.21.15.116:0/2353821579 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.534+0000 7f2c188d9640 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c190da640 1 -- 172.21.15.116:0/2353821579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 msgr2=0x560047113770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c190da640 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047113770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c190da640 1 -- 172.21.15.116:0/2353821579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 msgr2=0x560047117150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c190da640 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x560047117150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c190da640 1 -- 172.21.15.116:0/2353821579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560047003c80 con 0x560047102d10 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c188d9640 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x560047117150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c190da640 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x560047123700 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f2c10000c00 tx=0x7f2c10042040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d44e521d73e8f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2bfeffd640 1 -- 172.21.15.116:0/2353821579 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1002c770 con 0x560047102d10 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2bfeffd640 1 -- 172.21.15.116:0/2353821579 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c10004a70 con 0x560047102d10 2024-03-24T09:32:30.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2bfeffd640 1 -- 172.21.15.116:0/2353821579 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c100038a0 con 0x560047102d10 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 -- 172.21.15.116:0/2353821579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 msgr2=0x560047123700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x560047123700 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f2c10000c00 tx=0x7f2c10042040 comp rx=0 tx=0).stop 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 -- 172.21.15.116:0/2353821579 shutdown_connections 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047113770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x560047123700 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 --2- 172.21.15.116:0/2353821579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x560047117150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.535+0000 7f2c1e001740 1 -- 172.21.15.116:0/2353821579 >> 172.21.15.116:0/2353821579 conn(0x5600470fcb20 msgr2=0x5600471088e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 -- 172.21.15.116:0/2353821579 shutdown_connections 2024-03-24T09:32:30.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 -- 172.21.15.116:0/2353821579 wait complete. 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 -- start start 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 Processor -- start 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x56004710f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047180d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560047110330 con 0x560047102d10 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560047181270 con 0x5600471023f0 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x5600471813e0 con 0x56004710b8f0 2024-03-24T09:32:30.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c1e001740 10 client.0 _ll_register_callbacks cb 0x7f2c04009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c188d9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x56004710f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c190da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c188d9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x56004710f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57434/0 (socket says 172.21.15.116:57434) 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c188d9640 1 -- 172.21.15.116:0/693853881 learned_addr learned my addr 172.21.15.116:0/693853881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.536+0000 7f2c188d9640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047180d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c190da640 1 -- 172.21.15.116:0/693853881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 msgr2=0x560047180d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c190da640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047180d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c190da640 1 -- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 msgr2=0x56004710f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c190da640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x56004710f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c190da640 1 -- 172.21.15.116:0/693853881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560047067bd0 con 0x560047102d10 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c188d9640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047180d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c190da640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710fc30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f2c1002c170 tx=0x7f2c10004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60af5bebaef94ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2bfcff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2bfcff9640 1 -- 172.21.15.116:0/693853881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10052810 con 0x560047102d10 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2bfcff9640 1 -- 172.21.15.116:0/693853881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f2c100529c0 con 0x560047102d10 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c1002de90 con 0x560047102d10 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2bfcff9640 1 -- 172.21.15.116:0/693853881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f2c1002cbd0 con 0x560047102d10 2024-03-24T09:32:30.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 10 client.18561 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560046ffede0 con 0x560047102d10 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2bfcff9640 1 -- 172.21.15.116:0/693853881 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f2c1002de90 con 0x560047102d10 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2bfcff9640 1 -- 172.21.15.116:0/693853881 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f2c1003c050 con 0x560047102d10 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 10 client.18561 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 10 client.18561 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.k 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 20 client.18561 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x560047182810 0x560047184c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 4 client.18561 mds_command: new command op to 24422 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:32:30.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.537+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 --> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560046df4890 con 0x560047182810 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c188d9640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x560047182810 0x560047184c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2bfcff9640 10 client.18561 ms_handle_connect on v2:172.21.15.117:6832/3849440293 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c188d9640 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x560047182810 0x560047184c30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c45be60149465931 server_cookie=823c1ebee91eafb2 in_seq=0 out_seq=0 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2bfcff9640 1 -- 172.21.15.116:0/693853881 <== mds.4 v2:172.21.15.117:6832/3849440293 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560046df4890 con 0x560047182810 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2bfcff9640 10 client.18561 handle_command_reply: tid=0 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 client.18561 shutdown 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 msgr2=0x56004710fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710fc30 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f2c1002c170 tx=0x7f2c10004be0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 shutdown_connections 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x56004710b8f0 0x560047180d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:6832/3849440293,v1:172.21.15.117:6833/3849440293] conn(0x560047182810 0x560047184c30 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560047102d10 0x56004710fc30 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 --2- 172.21.15.116:0/693853881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600471023f0 0x56004710f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2bfcff9640 0 client.18561 ms_handle_reset on v2:172.21.15.117:6832/3849440293 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 >> 172.21.15.116:0/693853881 conn(0x5600470fcb20 msgr2=0x560047109170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 shutdown_connections 2024-03-24T09:32:30.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 1 -- 172.21.15.116:0/693853881 wait complete. 2024-03-24T09:32:30.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.538+0000 7f2c1e001740 20 client.18561 trim_cache size 0 max 16384 2024-03-24T09:32:30.550 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:32:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2c000a2230 msgr2=0x7f2c000a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2c000a2230 0x7f2c000a46f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0cbde0 tx=0x7f2c0c00efd0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 msgr2=0x7f2c14197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14197a90 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c00eb10 tx=0x7f2c0c00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 shutdown_connections 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c141076b0 0x7f2c14197a90 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f2c000a2230 0x7f2c000a46f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2c14102060 0x7f2c1419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 --2- 172.21.15.116:0/646506208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c141016a0 0x7f2c1419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 >> 172.21.15.116:0/646506208 conn(0x7f2c14078b10 msgr2=0x7f2c14104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 shutdown_connections 2024-03-24T09:32:30.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.539+0000 7f2c1b33e640 1 -- 172.21.15.116:0/646506208 wait complete. 2024-03-24T09:32:30.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T09:32:30.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 -- 172.21.15.116:0/4024436464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef54106660 msgr2=0x7fef54106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 --2- 172.21.15.116:0/4024436464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef54106660 0x7fef54106a40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fef44009930 tx=0x7fef4402bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 -- 172.21.15.116:0/4024436464 shutdown_connections 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 --2- 172.21.15.116:0/4024436464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5410d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 --2- 172.21.15.116:0/4024436464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef541005a0 0x7fef54100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 --2- 172.21.15.116:0/4024436464 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef54106660 0x7fef54106a40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 -- 172.21.15.116:0/4024436464 >> 172.21.15.116:0/4024436464 conn(0x7fef54077710 msgr2=0x7fef54077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 -- 172.21.15.116:0/4024436464 shutdown_connections 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 -- 172.21.15.116:0/4024436464 wait complete. 2024-03-24T09:32:30.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 Processor -- start 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.657+0000 7fef59c9c640 1 -- start start 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 0x7fef5419d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef54106660 0x7fef54197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef541980c0 con 0x7fef54100f60 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fef54198230 con 0x7fef54106660 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fef541983f0 con 0x7fef541005a0 2024-03-24T09:32:30.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 0x7fef5419d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef53fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef54106660 0x7fef54197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50808/0 (socket says 172.21.15.116:50808) 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef52ffd640 1 -- 172.21.15.116:0/3650894873 learned_addr learned my addr 172.21.15.116:0/3650894873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 0x7fef5419d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:50496/0 (socket says 172.21.15.116:50496) 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 -- 172.21.15.116:0/3650894873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef54106660 msgr2=0x7fef54197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef54106660 0x7fef54197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 -- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 msgr2=0x7fef5419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 -- 172.21.15.116:0/3650894873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef44009560 con 0x7fef541005a0 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef52ffd640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef537fe640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 0x7fef5419d8c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef44009230 tx=0x7fef44005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef50ff9640 1 -- 172.21.15.116:0/3650894873 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44004350 con 0x7fef541005a0 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef54198670 con 0x7fef541005a0 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef50ff9640 1 -- 172.21.15.116:0/3650894873 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef4402de90 con 0x7fef541005a0 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.658+0000 7fef50ff9640 1 -- 172.21.15.116:0/3650894873 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44005220 con 0x7fef541005a0 2024-03-24T09:32:30.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.659+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef541a4960 con 0x7fef541005a0 2024-03-24T09:32:30.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef50ff9640 1 -- 172.21.15.116:0/3650894873 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fef44003c30 con 0x7fef541005a0 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 Processor -- start 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 -- start start 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b077d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b103b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b12ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3a75840 con 0x55bbd3b073f0 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3a72d70 con 0x55bbd3b07d10 2024-03-24T09:32:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef5c87d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3a6e550 con 0x55bbd3b108f0 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef50ff9640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef3c0a2230 0x7fef3c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b077d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b077d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50812/0 (socket says 172.21.15.116:50812) 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef53fff640 1 -- 172.21.15.116:0/241468312 learned_addr learned my addr 172.21.15.116:0/241468312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef537fe640 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.661+0000 7fef52ffd640 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b12ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef53fff640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef3c0a2230 0x7fef3c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef50ff9640 1 -- 172.21.15.116:0/3650894873 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fef440f0240 con 0x7fef541005a0 2024-03-24T09:32:30.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef52ffd640 1 -- 172.21.15.116:0/241468312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 msgr2=0x55bbd3b103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef52ffd640 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b103b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef52ffd640 1 -- 172.21.15.116:0/241468312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 msgr2=0x55bbd3b077d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef52ffd640 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b077d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef52ffd640 1 -- 172.21.15.116:0/241468312 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbd3a03de0 con 0x55bbd3b108f0 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef53fff640 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef52ffd640 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b12ce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef54100060 tx=0x7fef480118e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf67a8145efbe4f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef53fff640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef3c0a2230 0x7fef3c0a46f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009fd0 tx=0x7fef4c004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef39ffb640 1 -- 172.21.15.116:0/241468312 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4800f4e0 con 0x55bbd3b108f0 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef39ffb640 1 -- 172.21.15.116:0/241468312 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef4800faf0 con 0x55bbd3b108f0 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.662+0000 7fef39ffb640 1 -- 172.21.15.116:0/241468312 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef48019ba0 con 0x55bbd3b108f0 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- 172.21.15.116:0/241468312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 msgr2=0x55bbd3b12ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b12ce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fef54100060 tx=0x7fef480118e0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- 172.21.15.116:0/241468312 shutdown_connections 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b12ce0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- 172.21.15.116:0/241468312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- 172.21.15.116:0/241468312 >> 172.21.15.116:0/241468312 conn(0x55bbd3b01b20 msgr2=0x55bbd3b01f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- 172.21.15.116:0/241468312 shutdown_connections 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- 172.21.15.116:0/241468312 wait complete. 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- start start 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 Processor -- start 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b0d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b0dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3741430 con 0x55bbd3b073f0 2024-03-24T09:32:30.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bbd36fde00 con 0x55bbd3b07d10 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.663+0000 7fef5c87d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bbd39f3590 con 0x55bbd3b108f0 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef5c87d740 10 client.0 _ll_register_callbacks cb 0x55bbd3a62fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b0d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b0d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50820/0 (socket says 172.21.15.116:50820) 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 -- 172.21.15.116:0/4151183577 learned_addr learned my addr 172.21.15.116:0/4151183577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef52ffd640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b0dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef53fff640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 -- 172.21.15.116:0/4151183577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 msgr2=0x55bbd3b0e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 -- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 msgr2=0x55bbd3b0dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b0dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 -- 172.21.15.116:0/4151183577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef48009d10 con 0x55bbd3b073f0 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef537fe640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b0d960 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fef440bc860 tx=0x7fef4408c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d76739d819475a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef32ffd640 1 -- 172.21.15.116:0/4151183577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef440a72b0 con 0x55bbd3b073f0 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55bbd376d7e0 con 0x55bbd3b073f0 2024-03-24T09:32:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef32ffd640 1 -- 172.21.15.116:0/4151183577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fef440bc8d0 con 0x55bbd3b073f0 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef32ffd640 1 -- 172.21.15.116:0/4151183577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fef440ed9b0 con 0x55bbd3b073f0 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef5c87d740 10 client.18573 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55bbd3a6cbd0 con 0x55bbd3b073f0 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.664+0000 7fef32ffd640 1 -- 172.21.15.116:0/4151183577 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55bbd376d7e0 con 0x55bbd3b073f0 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef32ffd640 1 -- 172.21.15.116:0/4151183577 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fef4403c070 con 0x55bbd3b073f0 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef5c87d740 10 client.18573 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef5c87d740 10 client.18573 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef5c87d740 20 client.18573 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef5c87d740 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b207d0 0x55bbd3b22c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef5c87d740 4 client.18573 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbd37aa720 con 0x55bbd3b207d0 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef52ffd640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b207d0 0x55bbd3b22c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef52ffd640 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b207d0 0x55bbd3b22c00 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=969c1ce86bedbabd server_cookie=26110f0d3ba2ab1a in_seq=0 out_seq=0 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.665+0000 7fef32ffd640 10 client.18573 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef32ffd640 1 -- 172.21.15.116:0/4151183577 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bbd37aa720 con 0x55bbd3b207d0 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef32ffd640 10 client.18573 handle_command_reply: tid=0 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 client.18573 shutdown 2024-03-24T09:32:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 msgr2=0x55bbd3b0d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b0d960 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fef440bc860 tx=0x7fef4408c730 comp rx=0 tx=0).stop 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 shutdown_connections 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b207d0 0x55bbd3b22c00 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b0dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 --2- 172.21.15.116:0/4151183577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b073f0 0x55bbd3b0d960 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef32ffd640 0 client.18573 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 >> 172.21.15.116:0/4151183577 conn(0x55bbd3b01b20 msgr2=0x55bbd3b03870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 shutdown_connections 2024-03-24T09:32:30.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 1 -- 172.21.15.116:0/4151183577 wait complete. 2024-03-24T09:32:30.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.666+0000 7fef5c87d740 20 client.18573 trim_cache size 0 max 16384 2024-03-24T09:32:30.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.679+0000 7fef5c87d740 1 Processor -- start 2024-03-24T09:32:30.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.679+0000 7fef5c87d740 1 -- start start 2024-03-24T09:32:30.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.679+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b073f0 0x55bbd3b1c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.679+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b284b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.679+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef5c87d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3b01650 con 0x55bbd3b108f0 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef5c87d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3afd3e0 con 0x55bbd3b07d10 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef5c87d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3b06dc0 con 0x55bbd3b073f0 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50830/0 (socket says 172.21.15.116:50830) 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 -- 172.21.15.116:0/1933977115 learned_addr learned my addr 172.21.15.116:0/1933977115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef52ffd640 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b284b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef537fe640 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b073f0 0x55bbd3b1c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 -- 172.21.15.116:0/1933977115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b073f0 msgr2=0x55bbd3b1c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b073f0 0x55bbd3b1c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 -- 172.21.15.116:0/1933977115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 msgr2=0x55bbd3b284b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b284b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 -- 172.21.15.116:0/1933977115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbd3a08c80 con 0x55bbd3b108f0 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef52ffd640 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b284b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef53fff640 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0deb0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0138c0 tx=0x7fef4c009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2872430a2a0303 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef397fa640 1 -- 172.21.15.116:0/1933977115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c013170 con 0x55bbd3b108f0 2024-03-24T09:32:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef397fa640 1 -- 172.21.15.116:0/1933977115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef4c010e60 con 0x55bbd3b108f0 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.680+0000 7fef397fa640 1 -- 172.21.15.116:0/1933977115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c022d30 con 0x55bbd3b108f0 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 -- 172.21.15.116:0/1933977115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 msgr2=0x55bbd3b0deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0deb0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0138c0 tx=0x7fef4c009be0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 -- 172.21.15.116:0/1933977115 shutdown_connections 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b0deb0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b07d10 0x55bbd3b284b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 --2- 172.21.15.116:0/1933977115 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b073f0 0x55bbd3b1c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 -- 172.21.15.116:0/1933977115 >> 172.21.15.116:0/1933977115 conn(0x55bbd3b01b20 msgr2=0x55bbd3b00cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 -- 172.21.15.116:0/1933977115 shutdown_connections 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 -- 172.21.15.116:0/1933977115 wait complete. 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 -- start start 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.681+0000 7fef5c87d740 1 Processor -- start 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 0x55bbd3b14140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b07d10 0x55bbd3b14680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b14bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3b89ef0 con 0x55bbd3b108f0 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3b8a030 con 0x55bbd3b073f0 2024-03-24T09:32:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55bbd3b8a170 con 0x55bbd3b07d10 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 10 client.0 _ll_register_callbacks cb 0x7fef44009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b14bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 0x55bbd3b14140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b14bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50844/0 (socket says 172.21.15.116:50844) 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 -- 172.21.15.116:0/3834841622 learned_addr learned my addr 172.21.15.116:0/3834841622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef537fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 0x55bbd3b14140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57482/0 (socket says 172.21.15.116:57482) 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef52ffd640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b07d10 0x55bbd3b14680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 -- 172.21.15.116:0/3834841622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b07d10 msgr2=0x55bbd3b14680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b07d10 0x55bbd3b14680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 -- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 msgr2=0x55bbd3b14140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 0x55bbd3b14140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 -- 172.21.15.116:0/3834841622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbd3a03de0 con 0x55bbd3b108f0 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef537fe640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 0x55bbd3b14140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef52ffd640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b07d10 0x55bbd3b14680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef53fff640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b14bc0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0133d0 tx=0x7fef4c013400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89a7ee44b9a140a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef32ffd640 1 -- 172.21.15.116:0/3834841622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef4c030070 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef32ffd640 1 -- 172.21.15.116:0/3834841622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fef4c022650 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef4800faf0 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef32ffd640 1 -- 172.21.15.116:0/3834841622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fef4c0109b0 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 10 client.18579 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.682+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x7fef4c013600 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef32ffd640 1 -- 172.21.15.116:0/3834841622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fef4800faf0 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef32ffd640 1 -- 172.21.15.116:0/3834841622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fef4c02b780 con 0x55bbd3b108f0 2024-03-24T09:32:30.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 10 client.18579 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 10 client.18579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 20 client.18579 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b8b8d0 0x55bbd3b8dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 4 client.18579 mds_command: new command op to 24443 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 --> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bbd37f9890 con 0x55bbd3b8b8d0 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef537fe640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b8b8d0 0x55bbd3b8dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef32ffd640 10 client.18579 ms_handle_connect on v2:172.21.15.117:6834/1932277434 2024-03-24T09:32:30.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef537fe640 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b8b8d0 0x55bbd3b8dcf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e04f1acb519757e6 server_cookie=da5895ecf2570df7 in_seq=0 out_seq=0 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef32ffd640 1 -- 172.21.15.116:0/3834841622 <== mds.3 v2:172.21.15.117:6834/1932277434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bbd37f9890 con 0x55bbd3b8b8d0 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef32ffd640 10 client.18579 handle_command_reply: tid=0 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.683+0000 7fef5c87d740 1 client.18579 shutdown 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 msgr2=0x55bbd3b14bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b14bc0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fef4c0133d0 tx=0x7fef4c013400 comp rx=0 tx=0).stop 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 shutdown_connections 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:6834/1932277434,v1:172.21.15.117:6835/1932277434] conn(0x55bbd3b8b8d0 0x55bbd3b8dcf0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bbd3b108f0 0x55bbd3b14bc0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55bbd3b07d10 0x55bbd3b14680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef32ffd640 0 client.18579 ms_handle_reset on v2:172.21.15.117:6834/1932277434 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 --2- 172.21.15.116:0/3834841622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bbd3b073f0 0x55bbd3b14140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 >> 172.21.15.116:0/3834841622 conn(0x55bbd3b01b20 msgr2=0x55bbd3b02e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 shutdown_connections 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 1 -- 172.21.15.116:0/3834841622 wait complete. 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef5c87d740 20 client.18579 trim_cache size 0 max 16384 2024-03-24T09:32:30.696 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:32:30.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef3c0a2230 msgr2=0x7fef3c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef3c0a2230 0x7fef3c0a46f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fef4c009fd0 tx=0x7fef4c004ab0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 msgr2=0x7fef5419d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 0x7fef5419d8c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fef44009230 tx=0x7fef44005dc0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 shutdown_connections 2024-03-24T09:32:30.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef54106660 0x7fef54197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef54100f60 0x7fef5419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fef3c0a2230 0x7fef3c0a46f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 --2- 172.21.15.116:0/3650894873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fef541005a0 0x7fef5419d8c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 >> 172.21.15.116:0/3650894873 conn(0x7fef54077710 msgr2=0x7fef54068ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 shutdown_connections 2024-03-24T09:32:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.684+0000 7fef59c9c640 1 -- 172.21.15.116:0/3650894873 wait complete. 2024-03-24T09:32:30.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T09:32:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 -- 172.21.15.116:0/1683943259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88106660 msgr2=0x7f8f88106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 --2- 172.21.15.116:0/1683943259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88106660 0x7f8f88106a40 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c009930 tx=0x7f8f7c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 -- 172.21.15.116:0/1683943259 shutdown_connections 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 --2- 172.21.15.116:0/1683943259 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88100f80 0x7f8f8810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 --2- 172.21.15.116:0/1683943259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 0x7f8f88100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 --2- 172.21.15.116:0/1683943259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88106660 0x7f8f88106a40 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 -- 172.21.15.116:0/1683943259 >> 172.21.15.116:0/1683943259 conn(0x7f8f88076990 msgr2=0x7f8f88076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 -- 172.21.15.116:0/1683943259 shutdown_connections 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 -- 172.21.15.116:0/1683943259 wait complete. 2024-03-24T09:32:30.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 Processor -- start 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 -- start start 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 0x7f8f8819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.805+0000 7f8f8fece640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 0x7f8f8819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8fece640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88106660 0x7f8f88197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8fece640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88198100 con 0x7f8f88100f80 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8fece640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88198270 con 0x7f8f881005c0 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8fece640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88198430 con 0x7f8f88106660 2024-03-24T09:32:30.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 0x7f8f8819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8dc6a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 0x7f8f8819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 0x7f8f8819de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50862/0 (socket says 172.21.15.116:50862) 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8e46b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88106660 0x7f8f88197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 -- 172.21.15.116:0/946545203 learned_addr learned my addr 172.21.15.116:0/946545203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 -- 172.21.15.116:0/946545203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88106660 msgr2=0x7f8f88197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88106660 0x7f8f88197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 -- 172.21.15.116:0/946545203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 msgr2=0x7f8f8819d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 0x7f8f8819d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 -- 172.21.15.116:0/946545203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f7c009560 con 0x7f8f88100f80 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 0x7f8f8819d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88106660 0x7f8f88197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8d469640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 0x7f8f8819de40 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400e9f0 tx=0x7f8f8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f7affd640 1 -- 172.21.15.116:0/946545203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84009700 con 0x7f8f88100f80 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f7affd640 1 -- 172.21.15.116:0/946545203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f8400cd80 con 0x7f8f88100f80 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f88198710 con 0x7f8f88100f80 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f7affd640 1 -- 172.21.15.116:0/946545203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84018840 con 0x7f8f88100f80 2024-03-24T09:32:30.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.806+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f881a49a0 con 0x7f8f88100f80 2024-03-24T09:32:30.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f7affd640 1 -- 172.21.15.116:0/946545203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f8f840189a0 con 0x7f8f88100f80 2024-03-24T09:32:30.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 Processor -- start 2024-03-24T09:32:30.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 -- start start 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbb97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbc23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55604bb27840 con 0x55604bbc28f0 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55604bb24d70 con 0x55604bbb9d10 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f92b83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55604bb20550 con 0x55604bbb93f0 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f7affd640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8f740a22b0 0x7f8f740a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f8dc6a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbc23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f8e46b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbb97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f8dc6a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbc23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57512/0 (socket says 172.21.15.116:57512) 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f8dc6a640 1 -- 172.21.15.116:0/243773830 learned_addr learned my addr 172.21.15.116:0/243773830 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f7affd640 1 -- 172.21.15.116:0/946545203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f8f84014070 con 0x7f8f88100f80 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.809+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8f740a22b0 0x7f8f740a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 -- 172.21.15.116:0/243773830 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 msgr2=0x55604bbc23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbc23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 -- 172.21.15.116:0/243773830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 msgr2=0x55604bbc4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 -- 172.21.15.116:0/243773830 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55604bab5de0 con 0x55604bbb93f0 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbb97d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000f440 tx=0x7f8f8000f900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a7d886ea51c02fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8f740a22b0 0x7f8f740a4770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000fe00 tx=0x7f8f8001e000 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f73fff640 1 -- 172.21.15.116:0/243773830 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8000ee20 con 0x55604bbb93f0 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f73fff640 1 -- 172.21.15.116:0/243773830 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f800114e0 con 0x55604bbb93f0 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.810+0000 7f8f73fff640 1 -- 172.21.15.116:0/243773830 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800054b0 con 0x55604bbb93f0 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 -- 172.21.15.116:0/243773830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 msgr2=0x55604bbb97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbb97d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000f440 tx=0x7f8f8000f900 comp rx=0 tx=0).stop 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 -- 172.21.15.116:0/243773830 shutdown_connections 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbc23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 --2- 172.21.15.116:0/243773830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbb97d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 -- 172.21.15.116:0/243773830 >> 172.21.15.116:0/243773830 conn(0x55604bbb3b20 msgr2=0x55604bbb3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 -- 172.21.15.116:0/243773830 shutdown_connections 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 -- 172.21.15.116:0/243773830 wait complete. 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 -- start start 2024-03-24T09:32:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.811+0000 7f8f92b83740 1 Processor -- start 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbbfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55604b7f3430 con 0x55604bbc28f0 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55604b7afe00 con 0x55604bbb9d10 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55604baa5590 con 0x55604bbb93f0 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f92b83740 10 client.0 _ll_register_callbacks cb 0x55604bb14fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8e46b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8dc6a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbbfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8dc6a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbbfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:50554/0 (socket says 172.21.15.116:50554) 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8e46b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50886/0 (socket says 172.21.15.116:50886) 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8dc6a640 1 -- 172.21.15.116:0/2225887623 learned_addr learned my addr 172.21.15.116:0/2225887623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 -- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 msgr2=0x55604bbbfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbbfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 -- 172.21.15.116:0/2225887623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 msgr2=0x55604bbc04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 -- 172.21.15.116:0/2225887623 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f80019040 con 0x55604bbb9d10 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbbfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f8d469640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbff70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8f840cba70 tx=0x7f8f8406a020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbbc921b6349afec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f717fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.812+0000 7f8f717fa640 1 -- 172.21.15.116:0/2225887623 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8409adc0 con 0x55604bbb9d10 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f717fa640 1 -- 172.21.15.116:0/2225887623 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f8f8409ee60 con 0x55604bbb9d10 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55604b81f7e0 con 0x55604bbb9d10 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f717fa640 1 -- 172.21.15.116:0/2225887623 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8f840a34b0 con 0x55604bbb9d10 2024-03-24T09:32:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 10 client.28406 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55604bb1ebd0 con 0x55604bbb9d10 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f717fa640 1 -- 172.21.15.116:0/2225887623 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f840b2030 con 0x55604bbb9d10 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f717fa640 1 -- 172.21.15.116:0/2225887623 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8f840a3020 con 0x55604bbb9d10 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 10 client.28406 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 10 client.28406 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 20 client.28406 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bbd2a40 0x55604bbd4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 4 client.28406 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:32:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55604b85c720 con 0x55604bbd2a40 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.813+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bbd2a40 0x55604bbd4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.814+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bbd2a40 0x55604bbd4e30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94d53a44a2764982 server_cookie=a20cb38dd79002d0 in_seq=0 out_seq=0 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.814+0000 7f8f717fa640 10 client.28406 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f717fa640 1 -- 172.21.15.116:0/2225887623 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55604b85c720 con 0x55604bbd2a40 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f717fa640 10 client.28406 handle_command_reply: tid=0 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 client.28406 shutdown 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 msgr2=0x55604bbbff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbff70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8f840cba70 tx=0x7f8f8406a020 comp rx=0 tx=0).stop 2024-03-24T09:32:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 shutdown_connections 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbc04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bbd2a40 0x55604bbd4e30 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbff70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f717fa640 0 client.28406 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 --2- 172.21.15.116:0/2225887623 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbbfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 >> 172.21.15.116:0/2225887623 conn(0x55604bbb3b20 msgr2=0x55604bbb5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 shutdown_connections 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 1 -- 172.21.15.116:0/2225887623 wait complete. 2024-03-24T09:32:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.815+0000 7f8f92b83740 20 client.28406 trim_cache size 0 max 16384 2024-03-24T09:32:30.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 Processor -- start 2024-03-24T09:32:30.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 -- start start 2024-03-24T09:32:30.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 0x55604bbc02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbc28f0 0x55604bbda700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55604bbb46f0 con 0x55604bbb9d10 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55604bbce600 con 0x55604bbc28f0 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f92b83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55604bbb9a10 con 0x55604bbb93f0 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 0x55604bbc02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 0x55604bbc02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50900/0 (socket says 172.21.15.116:50900) 2024-03-24T09:32:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f8d469640 1 -- 172.21.15.116:0/553222715 learned_addr learned my addr 172.21.15.116:0/553222715 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbc28f0 0x55604bbda700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.828+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f8d469640 1 -- 172.21.15.116:0/553222715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 msgr2=0x55604bbce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f8d469640 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f8d469640 1 -- 172.21.15.116:0/553222715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbc28f0 msgr2=0x55604bbda700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f8d469640 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbc28f0 0x55604bbda700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f8d469640 1 -- 172.21.15.116:0/553222715 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55604babac80 con 0x55604bbb9d10 2024-03-24T09:32:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f8d469640 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 0x55604bbc02d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f8f840cc070 tx=0x7f8f84000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2720ff6cb23f064c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f737fe640 1 -- 172.21.15.116:0/553222715 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8409a900 con 0x55604bbb9d10 2024-03-24T09:32:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f737fe640 1 -- 172.21.15.116:0/553222715 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f840b2040 con 0x55604bbb9d10 2024-03-24T09:32:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f737fe640 1 -- 172.21.15.116:0/553222715 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f840ce300 con 0x55604bbb9d10 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 -- 172.21.15.116:0/553222715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 msgr2=0x55604bbc02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 0x55604bbc02d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f8f840cc070 tx=0x7f8f84000bc0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 -- 172.21.15.116:0/553222715 shutdown_connections 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbc28f0 0x55604bbda700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbb9d10 0x55604bbc02d0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 --2- 172.21.15.116:0/553222715 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 -- 172.21.15.116:0/553222715 >> 172.21.15.116:0/553222715 conn(0x55604bbb3b20 msgr2=0x55604bbb57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 -- 172.21.15.116:0/553222715 shutdown_connections 2024-03-24T09:32:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.829+0000 7f8f92b83740 1 -- 172.21.15.116:0/553222715 wait complete. 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 -- start start 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 Processor -- start 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbbc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55604bbbcf80 con 0x55604bbc28f0 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55604bbbd0f0 con 0x55604bbb9d10 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55604bc37d30 con 0x55604bbb93f0 2024-03-24T09:32:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f92b83740 10 client.0 _ll_register_callbacks cb 0x7f8f84097130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f8e46b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbbc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f8d469640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f8e46b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbbc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50912/0 (socket says 172.21.15.116:50912) 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f8e46b640 1 -- 172.21.15.116:0/552188534 learned_addr learned my addr 172.21.15.116:0/552188534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.830+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8e46b640 1 -- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 msgr2=0x55604bbca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8e46b640 1 -- 172.21.15.116:0/552188534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 msgr2=0x55604bbbc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8e46b640 1 -- 172.21.15.116:0/552188534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55604bb1ebd0 con 0x55604bbc28f0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8d469640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f8e46b640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbbc880 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f8f80005200 tx=0x7f8f80005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a9a54bc0a00f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f717fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f717fa640 1 -- 172.21.15.116:0/552188534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800056e0 con 0x55604bbc28f0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f717fa640 1 -- 172.21.15.116:0/552188534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f8f80005840 con 0x55604bbc28f0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f800114e0 con 0x55604bbc28f0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f717fa640 1 -- 172.21.15.116:0/552188534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8f80005a00 con 0x55604bbc28f0 2024-03-24T09:32:30.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 10 client.18597 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55604bae0920 con 0x55604bbc28f0 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f717fa640 1 -- 172.21.15.116:0/552188534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f8f800114e0 con 0x55604bbc28f0 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f717fa640 1 -- 172.21.15.116:0/552188534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f8f8002e040 con 0x55604bbc28f0 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 10 client.18597 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 10 client.18597 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.f 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 20 client.18597 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bc39420 0x55604bc3b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 4 client.18597 mds_command: new command op to 24451 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.831+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 --> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55604b8ab890 con 0x55604bc39420 2024-03-24T09:32:30.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bc39420 0x55604bc3b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f8dc6a640 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bc39420 0x55604bc3b840 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc8548c132c664f server_cookie=b9ff62f2f377bf38 in_seq=0 out_seq=0 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f717fa640 10 client.18597 ms_handle_connect on v2:172.21.15.143:6834/1036797529 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f717fa640 1 -- 172.21.15.116:0/552188534 <== mds.2 v2:172.21.15.143:6834/1036797529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55604b8ab890 con 0x55604bc39420 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f717fa640 10 client.18597 handle_command_reply: tid=0 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 client.18597 shutdown 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 msgr2=0x55604bbbc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbbc880 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f8f80005200 tx=0x7f8f80005540 comp rx=0 tx=0).stop 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 shutdown_connections 2024-03-24T09:32:30.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55604bbc28f0 0x55604bbbc880 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:6834/1036797529,v1:172.21.15.143:6835/1036797529] conn(0x55604bc39420 0x55604bc3b840 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55604bbb9d10 0x55604bbbc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f717fa640 0 client.18597 ms_handle_reset on v2:172.21.15.143:6834/1036797529 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 --2- 172.21.15.116:0/552188534 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55604bbb93f0 0x55604bbca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 >> 172.21.15.116:0/552188534 conn(0x55604bbb3b20 msgr2=0x55604bbb2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 shutdown_connections 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 1 -- 172.21.15.116:0/552188534 wait complete. 2024-03-24T09:32:30.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.832+0000 7f8f92b83740 20 client.18597 trim_cache size 0 max 16384 2024-03-24T09:32:30.843 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:32:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8f740a22b0 msgr2=0x7f8f740a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8f740a22b0 0x7f8f740a4770 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000fe00 tx=0x7f8f8001e000 comp rx=0 tx=0).stop 2024-03-24T09:32:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 msgr2=0x7f8f8819de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 0x7f8f8819de40 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400e9f0 tx=0x7f8f8400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 shutdown_connections 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f8f740a22b0 0x7f8f740a4770 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8f88106660 0x7f8f88197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f88100f80 0x7f8f8819de40 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 --2- 172.21.15.116:0/946545203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f881005c0 0x7f8f8819d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 >> 172.21.15.116:0/946545203 conn(0x7f8f88076990 msgr2=0x7f8f880fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 shutdown_connections 2024-03-24T09:32:30.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.833+0000 7f8f8fece640 1 -- 172.21.15.116:0/946545203 wait complete. 2024-03-24T09:32:30.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T09:32:30.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 -- 172.21.15.116:0/3215865405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 msgr2=0x7f83dc106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 --2- 172.21.15.116:0/3215865405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 0x7f83dc106a40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009930 tx=0x7f83d802bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 -- 172.21.15.116:0/3215865405 shutdown_connections 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 --2- 172.21.15.116:0/3215865405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc100f60 0x7f83dc10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 --2- 172.21.15.116:0/3215865405 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83dc1005a0 0x7f83dc100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 --2- 172.21.15.116:0/3215865405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 0x7f83dc106a40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 -- 172.21.15.116:0/3215865405 >> 172.21.15.116:0/3215865405 conn(0x7f83dc077710 msgr2=0x7f83dc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 -- 172.21.15.116:0/3215865405 shutdown_connections 2024-03-24T09:32:30.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.948+0000 7f83e4015640 1 -- 172.21.15.116:0/3215865405 wait complete. 2024-03-24T09:32:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 Processor -- start 2024-03-24T09:32:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 -- start start 2024-03-24T09:32:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 0x7f83dc198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83dc100f60 0x7f83dc198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 0x7f83dc1a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc199830 con 0x7f83dc1005a0 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc1a4030 con 0x7f83dc106660 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e4015640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f83dc1a4170 con 0x7f83dc100f60 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 0x7f83dc198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 0x7f83dc198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50926/0 (socket says 172.21.15.116:50926) 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 -- 172.21.15.116:0/411562598 learned_addr learned my addr 172.21.15.116:0/411562598 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e25b2640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 0x7f83dc1a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e15b0640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83dc100f60 0x7f83dc198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 -- 172.21.15.116:0/411562598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83dc100f60 msgr2=0x7f83dc198fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83dc100f60 0x7f83dc198fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 -- 172.21.15.116:0/411562598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 msgr2=0x7f83dc1a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 0x7f83dc1a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 -- 172.21.15.116:0/411562598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d4009840 con 0x7f83dc1005a0 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.949+0000 7f83e1db1640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 0x7f83dc198a70 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009430 tx=0x7f83d8005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.950+0000 7f83d2ffd640 1 -- 172.21.15.116:0/411562598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8004350 con 0x7f83dc1005a0 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.950+0000 7f83d2ffd640 1 -- 172.21.15.116:0/411562598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83d802de90 con 0x7f83dc1005a0 2024-03-24T09:32:30.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.950+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d8009560 con 0x7f83dc1005a0 2024-03-24T09:32:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.950+0000 7f83d2ffd640 1 -- 172.21.15.116:0/411562598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8003a10 con 0x7f83dc1005a0 2024-03-24T09:32:30.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.950+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc1a4690 con 0x7f83dc1005a0 2024-03-24T09:32:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83d2ffd640 1 -- 172.21.15.116:0/411562598 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f83d8005390 con 0x7f83dc1005a0 2024-03-24T09:32:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 Processor -- start 2024-03-24T09:32:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 -- start start 2024-03-24T09:32:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec3341b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec334243b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec334248f0 0x55ec33426ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ec33389840 con 0x55ec3341bd10 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ec33386d70 con 0x55ec334248f0 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.952+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ec33382550 con 0x55ec3341b3f0 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec334243b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83d2ffd640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83c80a2230 0x7f83c80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e25b2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec3341b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec334243b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50940/0 (socket says 172.21.15.116:50940) 2024-03-24T09:32:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e25b2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec3341b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:50606/0 (socket says 172.21.15.116:50606) 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e1db1640 1 -- 172.21.15.116:0/3420310019 learned_addr learned my addr 172.21.15.116:0/3420310019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec334248f0 0x55ec33426ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e25b2640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83c80a2230 0x7f83c80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 -- 172.21.15.116:0/3420310019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 msgr2=0x55ec3341b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec3341b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 -- 172.21.15.116:0/3420310019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 msgr2=0x55ec334243b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec334243b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83d2ffd640 1 -- 172.21.15.116:0/411562598 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f83d80f0350 con 0x7f83dc1005a0 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 -- 172.21.15.116:0/3420310019 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec33317de0 con 0x55ec334248f0 2024-03-24T09:32:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e15b0640 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec334248f0 0x55ec33426ce0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009bf0 tx=0x7f83cc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68da9bf82d02b17c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83e25b2640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83c80a2230 0x7f83c80a46f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f83d4005fd0 tx=0x7f83d4004ab0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83c7fff640 1 -- 172.21.15.116:0/3420310019 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc004050 con 0x55ec334248f0 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83c7fff640 1 -- 172.21.15.116:0/3420310019 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83cc005020 con 0x55ec334248f0 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.953+0000 7f83c7fff640 1 -- 172.21.15.116:0/3420310019 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc004c40 con 0x55ec334248f0 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 -- 172.21.15.116:0/3420310019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec334248f0 msgr2=0x55ec33426ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec334248f0 0x55ec33426ce0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f83cc009bf0 tx=0x7f83cc011810 comp rx=0 tx=0).stop 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 -- 172.21.15.116:0/3420310019 shutdown_connections 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec334248f0 0x55ec33426ce0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec334243b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/3420310019 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec3341b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 -- 172.21.15.116:0/3420310019 >> 172.21.15.116:0/3420310019 conn(0x55ec33415b20 msgr2=0x55ec33415f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 -- 172.21.15.116:0/3420310019 shutdown_connections 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 -- 172.21.15.116:0/3420310019 wait complete. 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 -- start start 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.954+0000 7f83e6bf2740 1 Processor -- start 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec33421940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec33421e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 0x55ec334223c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ec33055430 con 0x55ec334248f0 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ec33011e00 con 0x55ec3341bd10 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ec33307590 con 0x55ec3341b3f0 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e6bf2740 10 client.0 _ll_register_callbacks cb 0x55ec33376fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 0x55ec334223c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec33421940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 0x55ec334223c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50950/0 (socket says 172.21.15.116:50950) 2024-03-24T09:32:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 -- 172.21.15.116:0/1865202605 learned_addr learned my addr 172.21.15.116:0/1865202605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e15b0640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec33421e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 -- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 msgr2=0x55ec33421940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec33421940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 -- 172.21.15.116:0/1865202605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 msgr2=0x55ec33421e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec33421e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e25b2640 1 -- 172.21.15.116:0/1865202605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83cc009850 con 0x55ec334248f0 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e15b0640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec33421e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.955+0000 7f83e1db1640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec33421940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e25b2640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 0x55ec334223c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f83d4013770 tx=0x7f83d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0368ca219bef2e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83c57fa640 1 -- 172.21.15.116:0/1865202605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4022650 con 0x55ec334248f0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83c57fa640 1 -- 172.21.15.116:0/1865202605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f83d40227b0 con 0x55ec334248f0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ec330817e0 con 0x55ec334248f0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83c57fa640 1 -- 172.21.15.116:0/1865202605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83d4022970 con 0x55ec334248f0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 10 client.18609 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ec33380bd0 con 0x55ec334248f0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83c57fa640 1 -- 172.21.15.116:0/1865202605 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55ec330817e0 con 0x55ec334248f0 2024-03-24T09:32:30.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83c57fa640 1 -- 172.21.15.116:0/1865202605 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f83d4028020 con 0x55ec334248f0 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 10 client.18609 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 10 client.18609 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 20 client.18609 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec33434a80 0x55ec33436ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 4 client.18609 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec330be720 con 0x55ec33434a80 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.956+0000 7f83e1db1640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec33434a80 0x55ec33436ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.957+0000 7f83c57fa640 10 client.18609 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:32:30.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.957+0000 7f83e1db1640 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec33434a80 0x55ec33436ea0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2cbfbd910daa2e9 server_cookie=c69d16924bf8986b in_seq=0 out_seq=0 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83c57fa640 1 -- 172.21.15.116:0/1865202605 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ec330be720 con 0x55ec33434a80 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83c57fa640 10 client.18609 handle_command_reply: tid=0 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 client.18609 shutdown 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 msgr2=0x55ec334223c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 0x55ec334223c0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f83d4013770 tx=0x7f83d4004a40 comp rx=0 tx=0).stop 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 shutdown_connections 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec33434a80 0x55ec33436ea0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec334248f0 0x55ec334223c0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83c57fa640 0 client.18609 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec33421e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/1865202605 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec3341b3f0 0x55ec33421940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 >> 172.21.15.116:0/1865202605 conn(0x55ec33415b20 msgr2=0x55ec33417870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 shutdown_connections 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 1 -- 172.21.15.116:0/1865202605 wait complete. 2024-03-24T09:32:30.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.958+0000 7f83e6bf2740 20 client.18609 trim_cache size 0 max 16384 2024-03-24T09:32:30.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 Processor -- start 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 -- start start 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 0x55ec33430180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec334221e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3343c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ec33433250 con 0x55ec3341b3f0 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ec33416b20 con 0x55ec3341bd10 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.971+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ec33415650 con 0x55ec334248f0 2024-03-24T09:32:30.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 0x55ec33430180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 0x55ec33430180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50964/0 (socket says 172.21.15.116:50964) 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 -- 172.21.15.116:0/2144008486 learned_addr learned my addr 172.21.15.116:0/2144008486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e25b2640 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3343c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e15b0640 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec334221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 -- 172.21.15.116:0/2144008486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 msgr2=0x55ec3343c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3343c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 -- 172.21.15.116:0/2144008486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 msgr2=0x55ec334221e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec334221e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 -- 172.21.15.116:0/2144008486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec3331cc80 con 0x55ec3341b3f0 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 0x55ec33430180 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f83d802cd70 tx=0x7f83d80f2820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d638de750649990c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83c77fe640 1 -- 172.21.15.116:0/2144008486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d80bd140 con 0x55ec3341b3f0 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83c77fe640 1 -- 172.21.15.116:0/2144008486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83d80f0170 con 0x55ec3341b3f0 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.972+0000 7f83c77fe640 1 -- 172.21.15.116:0/2144008486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d80b90b0 con 0x55ec3341b3f0 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2144008486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 msgr2=0x55ec33430180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 0x55ec33430180 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f83d802cd70 tx=0x7f83d80f2820 comp rx=0 tx=0).stop 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2144008486 shutdown_connections 2024-03-24T09:32:30.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3343c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341bd10 0x55ec334221e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2144008486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341b3f0 0x55ec33430180 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2144008486 >> 172.21.15.116:0/2144008486 conn(0x55ec33415b20 msgr2=0x55ec33414430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2144008486 shutdown_connections 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2144008486 wait complete. 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- start start 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 Processor -- start 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341b3f0 0x55ec3342c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec3341e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3341e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ec3341ec30 con 0x55ec3341bd10 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ec3341eda0 con 0x55ec3341b3f0 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.973+0000 7f83e6bf2740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55ec3341ef60 con 0x55ec334248f0 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e6bf2740 10 client.0 _ll_register_callbacks cb 0x7f83d4013530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec3341e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec3341e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50980/0 (socket says 172.21.15.116:50980) 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 -- 172.21.15.116:0/2833139651 learned_addr learned my addr 172.21.15.116:0/2833139651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e25b2640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3341e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341b3f0 0x55ec3342c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 -- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 msgr2=0x55ec3341e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3341e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 -- 172.21.15.116:0/2833139651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341b3f0 msgr2=0x55ec3342c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341b3f0 0x55ec3342c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 -- 172.21.15.116:0/2833139651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83dc1a4690 con 0x55ec3341bd10 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341b3f0 0x55ec3342c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e25b2640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3341e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e15b0640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec3341e1b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x55ec33416cb0 tx=0x7f83cc005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f0fdca49896b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83c57fa640 1 -- 172.21.15.116:0/2833139651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83cc029070 con 0x55ec3341bd10 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83c57fa640 1 -- 172.21.15.116:0/2833139651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f83cc004c00 con 0x55ec3341bd10 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83cc005020 con 0x55ec3341bd10 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83c57fa640 1 -- 172.21.15.116:0/2833139651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f83cc004e00 con 0x55ec3341bd10 2024-03-24T09:32:30.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e6bf2740 10 client.18615 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.974+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ec33342920 con 0x55ec3341bd10 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83c57fa640 1 -- 172.21.15.116:0/2833139651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f83cc037080 con 0x55ec3341bd10 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83c57fa640 1 -- 172.21.15.116:0/2833139651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7f83cc021040 con 0x55ec3341bd10 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 10 client.18615 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 10 client.18615 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 20 client.18615 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec3349b2f0 0x55ec3349d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 4 client.18615 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:32:30.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 --> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec3310d890 con 0x55ec3349b2f0 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec3349b2f0 0x55ec3349d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83c57fa640 10 client.18615 ms_handle_connect on v2:172.21.15.143:6838/3139487666 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e1db1640 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec3349b2f0 0x55ec3349d710 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87a59cc8751c4224 server_cookie=fb5dce37a9a8182a in_seq=0 out_seq=0 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83c57fa640 1 -- 172.21.15.116:0/2833139651 <== mds.1 v2:172.21.15.143:6838/3139487666 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec3310d890 con 0x55ec3349b2f0 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83c57fa640 10 client.18615 handle_command_reply: tid=0 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 1 client.18615 shutdown 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 msgr2=0x55ec3341e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.975+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec3341e1b0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x55ec33416cb0 tx=0x7f83cc005db0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 shutdown_connections 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:6838/3139487666,v1:172.21.15.143:6839/3139487666] conn(0x55ec3349b2f0 0x55ec3349d710 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55ec334248f0 0x55ec3341e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ec3341bd10 0x55ec3341e1b0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 --2- 172.21.15.116:0/2833139651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec3341b3f0 0x55ec3342c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83c57fa640 0 client.18615 ms_handle_reset on v2:172.21.15.143:6838/3139487666 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 >> 172.21.15.116:0/2833139651 conn(0x55ec33415b20 msgr2=0x55ec33417760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 shutdown_connections 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 1 -- 172.21.15.116:0/2833139651 wait complete. 2024-03-24T09:32:30.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e6bf2740 20 client.18615 trim_cache size 0 max 16384 2024-03-24T09:32:30.988 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:32:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83c80a2230 msgr2=0x7f83c80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83c80a2230 0x7f83c80a46f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f83d4005fd0 tx=0x7f83d4004ab0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 msgr2=0x7f83dc198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:30.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 0x7f83dc198a70 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009430 tx=0x7f83d8005dd0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 shutdown_connections 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83dc106660 0x7f83dc1a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f83c80a2230 0x7f83c80a46f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f83dc100f60 0x7f83dc198fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 --2- 172.21.15.116:0/411562598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83dc1005a0 0x7f83dc198a70 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 >> 172.21.15.116:0/411562598 conn(0x7f83dc077710 msgr2=0x7f83dc0ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 shutdown_connections 2024-03-24T09:32:30.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:30.976+0000 7f83e4015640 1 -- 172.21.15.116:0/411562598 wait complete. 2024-03-24T09:32:30.991 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T09:32:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 -- 172.21.15.116:0/3629946709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc940107760 msgr2=0x7fc940107b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 --2- 172.21.15.116:0/3629946709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc940107760 0x7fc940107b40 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fc930009930 tx=0x7fc93002bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 -- 172.21.15.116:0/3629946709 shutdown_connections 2024-03-24T09:32:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 --2- 172.21.15.116:0/3629946709 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc9400692d0 0x7fc94010e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 --2- 172.21.15.116:0/3629946709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc940068910 0x7fc940068d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 --2- 172.21.15.116:0/3629946709 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc940107760 0x7fc940107b40 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.095+0000 7fc946a07640 1 -- 172.21.15.116:0/3629946709 >> 172.21.15.116:0/3629946709 conn(0x7fc940077710 msgr2=0x7fc940077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 -- 172.21.15.116:0/3629946709 shutdown_connections 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 -- 172.21.15.116:0/3629946709 wait complete. 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 Processor -- start 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 -- start start 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc940068910 0x7fc94019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 0x7fc94019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc940107760 0x7fc940197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc9401980b0 con 0x7fc9400692d0 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc940198220 con 0x7fc940107760 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.096+0000 7fc946a07640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc9401983e0 con 0x7fc940068910 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 0x7fc94019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 0x7fc94019ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51002/0 (socket says 172.21.15.116:51002) 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc940068910 0x7fc94019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 -- 172.21.15.116:0/3349866061 learned_addr learned my addr 172.21.15.116:0/3349866061 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc944fa4640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc940107760 0x7fc940197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 -- 172.21.15.116:0/3349866061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc940068910 msgr2=0x7fc94019d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc940068910 0x7fc94019d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 -- 172.21.15.116:0/3349866061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc940107760 msgr2=0x7fc940197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc940107760 0x7fc940197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 -- 172.21.15.116:0/3349866061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc930009560 con 0x7fc9400692d0 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 0x7fc94019ddf0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc93400e9f0 tx=0x7fc93400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93d7fa640 1 -- 172.21.15.116:0/3349866061 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934009700 con 0x7fc9400692d0 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93d7fa640 1 -- 172.21.15.116:0/3349866061 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc93400cd80 con 0x7fc9400692d0 2024-03-24T09:32:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9401986c0 con 0x7fc9400692d0 2024-03-24T09:32:31.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc93d7fa640 1 -- 172.21.15.116:0/3349866061 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934018840 con 0x7fc9400692d0 2024-03-24T09:32:31.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.097+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9401a4950 con 0x7fc9400692d0 2024-03-24T09:32:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.099+0000 7fc93d7fa640 1 -- 172.21.15.116:0/3349866061 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc9340189a0 con 0x7fc9400692d0 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.099+0000 7fc9495e9740 1 Processor -- start 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 -- start start 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cd3f0 0x55682d7cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7d63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7d68f0 0x55682d7d8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682d73b840 con 0x55682d7d68f0 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682d738d70 con 0x55682d7cdd10 2024-03-24T09:32:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc9495e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55682d734550 con 0x55682d7cd3f0 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7d68f0 0x55682d7d8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7d63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93d7fa640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9280a2230 0x7fc9280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7d63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57630/0 (socket says 172.21.15.116:57630) 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 -- 172.21.15.116:0/3656635227 learned_addr learned my addr 172.21.15.116:0/3656635227 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc944fa4640 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cd3f0 0x55682d7cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc944fa4640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9280a2230 0x7fc9280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 -- 172.21.15.116:0/3656635227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cd3f0 msgr2=0x55682d7cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cd3f0 0x55682d7cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 -- 172.21.15.116:0/3656635227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7d68f0 msgr2=0x55682d7d8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7d68f0 0x55682d7d8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 -- 172.21.15.116:0/3656635227 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682d6c9de0 con 0x55682d7cdd10 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc944fa4640 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cd3f0 0x55682d7cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93d7fa640 1 -- 172.21.15.116:0/3349866061 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fc934014070 con 0x7fc9400692d0 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.100+0000 7fc93ffff640 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7d63b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc930036040 tx=0x7fc930005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4452987983e78c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc944fa4640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9280a2230 0x7fc9280a46f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc938005c40 tx=0x7fc938009500 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9267fc640 1 -- 172.21.15.116:0/3656635227 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930004800 con 0x55682d7cdd10 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9267fc640 1 -- 172.21.15.116:0/3656635227 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc930004960 con 0x55682d7cdd10 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9267fc640 1 -- 172.21.15.116:0/3656635227 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930003830 con 0x55682d7cdd10 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 -- 172.21.15.116:0/3656635227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 msgr2=0x55682d7d63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7d63b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc930036040 tx=0x7fc930005e50 comp rx=0 tx=0).stop 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 -- 172.21.15.116:0/3656635227 shutdown_connections 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7d68f0 0x55682d7d8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7d63b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3656635227 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cd3f0 0x55682d7cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 -- 172.21.15.116:0/3656635227 >> 172.21.15.116:0/3656635227 conn(0x55682d7c7b20 msgr2=0x55682d7c7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 -- 172.21.15.116:0/3656635227 shutdown_connections 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.101+0000 7fc9495e9740 1 -- 172.21.15.116:0/3656635227 wait complete. 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 -- start start 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 Processor -- start 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7d3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cdd10 0x55682d7d3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7d68f0 0x55682d7d4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682d407430 con 0x55682d7cd3f0 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682d3c3e00 con 0x55682d7d68f0 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55682d6b9590 con 0x55682d7cdd10 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc9495e9740 10 client.0 _ll_register_callbacks cb 0x55682d728fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7d3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc944fa4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7d68f0 0x55682d7d4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7d3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51024/0 (socket says 172.21.15.116:51024) 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 -- 172.21.15.116:0/3345363363 learned_addr learned my addr 172.21.15.116:0/3345363363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cdd10 0x55682d7d3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 -- 172.21.15.116:0/3345363363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cdd10 msgr2=0x55682d7d3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cdd10 0x55682d7d3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 -- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7d68f0 msgr2=0x55682d7d4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7d68f0 0x55682d7d4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc93ffff640 1 -- 172.21.15.116:0/3345363363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc94010ee70 con 0x55682d7cd3f0 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.102+0000 7fc944fa4640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7d68f0 0x55682d7d4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc93ffff640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7d3980 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc930004ff0 tx=0x7fc930005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3b0578c8dee9b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc91f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc91f7fe640 1 -- 172.21.15.116:0/3345363363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc930004800 con 0x55682d7cd3f0 2024-03-24T09:32:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc91f7fe640 1 -- 172.21.15.116:0/3345363363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fc93004d070 con 0x55682d7cd3f0 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55682d4337e0 con 0x55682d7cd3f0 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc91f7fe640 1 -- 172.21.15.116:0/3345363363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc930005650 con 0x55682d7cd3f0 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 10 client.18627 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55682d732bd0 con 0x55682d7cd3f0 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc91f7fe640 1 -- 172.21.15.116:0/3345363363 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x55682d4337e0 con 0x55682d7cd3f0 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc91f7fe640 1 -- 172.21.15.116:0/3345363363 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc93003c050 con 0x55682d7cd3f0 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 10 client.18627 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 10 client.18627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:32:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 20 client.18627 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7e66c0 0x55682d7e8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 4 client.18627 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.103+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55682d470720 con 0x55682d7e66c0 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.104+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7e66c0 0x55682d7e8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.104+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7e66c0 0x55682d7e8ad0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=831cc6c9a00ce4c7 server_cookie=8039bad81b4ed982 in_seq=0 out_seq=0 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.104+0000 7fc91f7fe640 10 client.18627 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.104+0000 7fc91f7fe640 1 -- 172.21.15.116:0/3345363363 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55682d470720 con 0x55682d7e66c0 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.104+0000 7fc91f7fe640 10 client.18627 handle_command_reply: tid=0 2024-03-24T09:32:31.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.104+0000 7fc9495e9740 1 client.18627 shutdown 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 msgr2=0x55682d7d3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7d3980 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fc930004ff0 tx=0x7fc930005020 comp rx=0 tx=0).stop 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 shutdown_connections 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7e66c0 0x55682d7e8ad0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7d68f0 0x55682d7d4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7cdd10 0x55682d7d3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 --2- 172.21.15.116:0/3345363363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7d3980 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc91f7fe640 0 client.18627 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 >> 172.21.15.116:0/3345363363 conn(0x55682d7c7b20 msgr2=0x55682d7c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 shutdown_connections 2024-03-24T09:32:31.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 1 -- 172.21.15.116:0/3345363363 wait complete. 2024-03-24T09:32:31.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.105+0000 7fc9495e9740 20 client.18627 trim_cache size 0 max 16384 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 Processor -- start 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 -- start start 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7e2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682d7d3770 con 0x55682d7cd3f0 2024-03-24T09:32:31.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682d7c8b20 con 0x55682d7cdd10 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc9495e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55682d7d91a0 con 0x55682d7c7b20 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7e2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51034/0 (socket says 172.21.15.116:51034) 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 -- 172.21.15.116:0/1008968671 learned_addr learned my addr 172.21.15.116:0/1008968671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc944fa4640 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93ffff640 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 -- 172.21.15.116:0/1008968671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 msgr2=0x55682d7d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7d60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 -- 172.21.15.116:0/1008968671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 msgr2=0x55682d7ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 -- 172.21.15.116:0/1008968671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55682d6cec80 con 0x55682d7cd3f0 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7e2120 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc93409add0 tx=0x7fc93408e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1493311144e2f23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc925ffb640 1 -- 172.21.15.116:0/1008968671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc934099140 con 0x55682d7cd3f0 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.118+0000 7fc925ffb640 1 -- 172.21.15.116:0/1008968671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc934010b50 con 0x55682d7cd3f0 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc925ffb640 1 -- 172.21.15.116:0/1008968671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93409a4f0 con 0x55682d7cd3f0 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 -- 172.21.15.116:0/1008968671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 msgr2=0x55682d7e2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7e2120 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7fc93409add0 tx=0x7fc93408e040 comp rx=0 tx=0).stop 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 -- 172.21.15.116:0/1008968671 shutdown_connections 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cdd10 0x55682d7ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cd3f0 0x55682d7e2120 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 --2- 172.21.15.116:0/1008968671 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 -- 172.21.15.116:0/1008968671 >> 172.21.15.116:0/1008968671 conn(0x55682d7cf350 msgr2=0x55682d7c6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 -- 172.21.15.116:0/1008968671 shutdown_connections 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 -- 172.21.15.116:0/1008968671 wait complete. 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.119+0000 7fc9495e9740 1 -- start start 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 Processor -- start 2024-03-24T09:32:31.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7da3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 0x55682d7da8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 0x55682d84b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55682d7db080 con 0x55682d7cdd10 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55682d7db1f0 con 0x55682d7cd3f0 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x55682d84bf10 con 0x55682d7c7b20 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc9495e9740 10 client.0 _ll_register_callbacks cb 0x7fc93409ac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 0x55682d84b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc93ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7da3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 0x55682d7da8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 0x55682d84b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51042/0 (socket says 172.21.15.116:51042) 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc93f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 0x55682d7da8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.116:57650/0 (socket says 172.21.15.116:57650) 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 -- 172.21.15.116:0/862364253 learned_addr learned my addr 172.21.15.116:0/862364253 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 -- 172.21.15.116:0/862364253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 msgr2=0x55682d7da3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7da3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 -- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 msgr2=0x55682d7da8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 0x55682d7da8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc944fa4640 1 -- 172.21.15.116:0/862364253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9401a4950 con 0x55682d7cdd10 2024-03-24T09:32:31.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc93f7fe640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 0x55682d7da8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.120+0000 7fc93ffff640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7da3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc944fa4640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 0x55682d84b9d0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc938004840 tx=0x7fc938004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df85af420567d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc91f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc91f7fe640 1 -- 172.21.15.116:0/862364253 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc938025070 con 0x55682d7cdd10 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc91f7fe640 1 -- 172.21.15.116:0/862364253 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7fc938020480 con 0x55682d7cdd10 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc930004960 con 0x55682d7cdd10 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc91f7fe640 1 -- 172.21.15.116:0/862364253 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc938020640 con 0x55682d7cdd10 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 10 client.18633 fetch_fsmap learned FSMap version 35 2024-03-24T09:32:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55682d6f4920 con 0x55682d7cdd10 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc91f7fe640 1 -- 172.21.15.116:0/862364253 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc930004960 con 0x55682d7cdd10 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc91f7fe640 1 -- 172.21.15.116:0/862364253 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8362+0+0 (secure 0 0 0) 0x7fc93801f780 con 0x55682d7cdd10 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 10 client.18633 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 10 client.18633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 20 client.18633 populate_metadata read hostname 'smithi116' 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7d68f0 0x55682d84f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 4 client.18633 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T09:32:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 --> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55682d4bf890 con 0x55682d7d68f0 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.121+0000 7fc93ffff640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7d68f0 0x55682d84f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc93ffff640 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7d68f0 0x55682d84f820 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a600071e0041f7 server_cookie=3e46a9e3295d4971 in_seq=0 out_seq=0 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc91f7fe640 10 client.18633 ms_handle_connect on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc91f7fe640 1 -- 172.21.15.116:0/862364253 <== mds.0 v2:172.21.15.117:6838/3950675281 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55682d4bf890 con 0x55682d7d68f0 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc91f7fe640 10 client.18633 handle_command_reply: tid=0 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 client.18633 shutdown 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 msgr2=0x55682d84b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 0x55682d84b9d0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fc938004840 tx=0x7fc938004a40 comp rx=0 tx=0).stop 2024-03-24T09:32:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 shutdown_connections 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55682d7cdd10 0x55682d84b9d0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:6838/3950675281,v1:172.21.15.117:6839/3950675281] conn(0x55682d7d68f0 0x55682d84f820 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55682d7cd3f0 0x55682d7da8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 --2- 172.21.15.116:0/862364253 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x55682d7c7b20 0x55682d7da3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc91f7fe640 0 client.18633 ms_handle_reset on v2:172.21.15.117:6838/3950675281 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 >> 172.21.15.116:0/862364253 conn(0x55682d7cf350 msgr2=0x55682d7d41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 shutdown_connections 2024-03-24T09:32:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 1 -- 172.21.15.116:0/862364253 wait complete. 2024-03-24T09:32:31.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.122+0000 7fc9495e9740 20 client.18633 trim_cache size 0 max 16384 2024-03-24T09:32:31.134 INFO:teuthology.orchestra.run.smithi116.stdout:[] 2024-03-24T09:32:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9280a2230 msgr2=0x7fc9280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9280a2230 0x7fc9280a46f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc938005c40 tx=0x7fc938009500 comp rx=0 tx=0).stop 2024-03-24T09:32:31.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 msgr2=0x7fc94019ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 0x7fc94019ddf0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fc93400e9f0 tx=0x7fc93400eeb0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 shutdown_connections 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc940107760 0x7fc940197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9400692d0 0x7fc94019ddf0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc9280a2230 0x7fc9280a46f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 --2- 172.21.15.116:0/3349866061 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc940068910 0x7fc94019d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 >> 172.21.15.116:0/3349866061 conn(0x7fc940077710 msgr2=0x7fc9400fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 shutdown_connections 2024-03-24T09:32:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:31.123+0000 7fc946a07640 1 -- 172.21.15.116:0/3349866061 wait complete. 2024-03-24T09:32:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:31 smithi143 ceph-mon[35965]: pgmap v323: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:31 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3652937782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:32:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:31 smithi117 ceph-mon[35788]: pgmap v323: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:31.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:31 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3652937782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:32:31.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:31 smithi116 ceph-mon[32207]: pgmap v323: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:31.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:31 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3652937782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:32:32.138 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-24T09:32:32.138 INFO:tasks.fwd_scrub:done joining 2024-03-24T09:32:32.138 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-24T09:32:32.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:32:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 -- 172.21.15.116:0/397097462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928100f90 msgr2=0x7f992810d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 --2- 172.21.15.116:0/397097462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928100f90 0x7f992810d450 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f9920009ca0 tx=0x7f992002c0f0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 -- 172.21.15.116:0/397097462 shutdown_connections 2024-03-24T09:32:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 --2- 172.21.15.116:0/397097462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928100f90 0x7f992810d450 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 --2- 172.21.15.116:0/397097462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99281005d0 0x7f9928100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 --2- 172.21.15.116:0/397097462 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9928106690 0x7f9928106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 -- 172.21.15.116:0/397097462 >> 172.21.15.116:0/397097462 conn(0x7f9928076990 msgr2=0x7f9928076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 -- 172.21.15.116:0/397097462 shutdown_connections 2024-03-24T09:32:32.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.263+0000 7f992c9d7640 1 -- 172.21.15.116:0/397097462 wait complete. 2024-03-24T09:32:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 Processor -- start 2024-03-24T09:32:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 -- start start 2024-03-24T09:32:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99281005d0 0x7f992819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9928100f90 0x7f992819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 0x7f9928196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9928196eb0 con 0x7f9928106690 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9928197020 con 0x7f9928100f90 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992c9d7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f99281971e0 con 0x7f99281005d0 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 0x7f9928196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 0x7f9928196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51070/0 (socket says 172.21.15.116:51070) 2024-03-24T09:32:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 -- 172.21.15.116:0/2186608036 learned_addr learned my addr 172.21.15.116:0/2186608036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f992659c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99281005d0 0x7f992819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9925d9b640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9928100f90 0x7f992819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 -- 172.21.15.116:0/2186608036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99281005d0 msgr2=0x7f992819c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99281005d0 0x7f992819c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 -- 172.21.15.116:0/2186608036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9928100f90 msgr2=0x7f992819cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9928100f90 0x7f992819cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.264+0000 7f9926d9d640 1 -- 172.21.15.116:0/2186608036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9920009900 con 0x7f9928106690 2024-03-24T09:32:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.265+0000 7f9926d9d640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 0x7f9928196860 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f9920005e30 tx=0x7f9920034a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.265+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920032820 con 0x7f9928106690 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.265+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9920003e20 con 0x7f9928106690 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.265+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9928197460 con 0x7f9928106690 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.265+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920004790 con 0x7f9928106690 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.265+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99281a3750 con 0x7f9928106690 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.266+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928101cf0 con 0x7f9928106690 2024-03-24T09:32:32.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.267+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f992000a040 con 0x7f9928106690 2024-03-24T09:32:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.267+0000 7f991f7fe640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99140a2230 0x7f99140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.268+0000 7f992659c640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99140a2230 0x7f99140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.268+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f99200f1380 con 0x7f9928106690 2024-03-24T09:32:32.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.268+0000 7f992659c640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99140a2230 0x7f99140a46f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f99281979a0 tx=0x7f9918005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.271+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f99200b9d10 con 0x7f9928106690 2024-03-24T09:32:32.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.443+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9928068950 con 0x7f9928106690 2024-03-24T09:32:32.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.445+0000 7f991f7fe640 1 -- 172.21.15.116:0/2186608036 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+25479 (secure 0 0 0) 0x7f9920039090 con 0x7f9928106690 2024-03-24T09:32:32.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:32:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":102,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:32:14.039978+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:32:32.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99140a2230 msgr2=0x7f99140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99140a2230 0x7f99140a46f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f99281979a0 tx=0x7f9918005f90 comp rx=0 tx=0).stop 2024-03-24T09:32:32.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 msgr2=0x7f9928196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 0x7f9928196860 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f9920005e30 tx=0x7f9920034a60 comp rx=0 tx=0).stop 2024-03-24T09:32:32.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 shutdown_connections 2024-03-24T09:32:32.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9928106690 0x7f9928196860 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9928100f90 0x7f992819cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f99140a2230 0x7f99140a46f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 --2- 172.21.15.116:0/2186608036 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f99281005d0 0x7f992819c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 >> 172.21.15.116:0/2186608036 conn(0x7f9928076990 msgr2=0x7f99280fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:32.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 shutdown_connections 2024-03-24T09:32:32.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.447+0000 7f992c9d7640 1 -- 172.21.15.116:0/2186608036 wait complete. 2024-03-24T09:32:32.459 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:32:32.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:32:32.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 -- 172.21.15.116:0/905642838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c102930 msgr2=0x7f6b1c102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 --2- 172.21.15.116:0/905642838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c102930 0x7f6b1c102d10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c009860 tx=0x7f6b0c02bc20 comp rx=0 tx=0).stop 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 -- 172.21.15.116:0/905642838 shutdown_connections 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 --2- 172.21.15.116:0/905642838 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c10bc70 0x7f6b1c10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 --2- 172.21.15.116:0/905642838 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c103250 0x7f6b1c10b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 --2- 172.21.15.116:0/905642838 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c102930 0x7f6b1c102d10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 -- 172.21.15.116:0/905642838 >> 172.21.15.116:0/905642838 conn(0x7f6b1c0777c0 msgr2=0x7f6b1c077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 -- 172.21.15.116:0/905642838 shutdown_connections 2024-03-24T09:32:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 -- 172.21.15.116:0/905642838 wait complete. 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.563+0000 7f6b21b80640 1 Processor -- start 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 -- start start 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 0x7f6b1c19d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c103250 0x7f6b1c19dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c10bc70 0x7f6b1c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c198050 con 0x7f6b1c102930 2024-03-24T09:32:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1981c0 con 0x7f6b1c103250 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c198380 con 0x7f6b1c10bc70 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 0x7f6b1c19d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 0x7f6b1c19d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51082/0 (socket says 172.21.15.116:51082) 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/3903786478 learned_addr learned my addr 172.21.15.116:0/3903786478 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b13fff640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c103250 0x7f6b1c19dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1bfff640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c10bc70 0x7f6b1c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/3903786478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c10bc70 msgr2=0x7f6b1c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c10bc70 0x7f6b1c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/3903786478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c103250 msgr2=0x7f6b1c19dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c103250 0x7f6b1c19dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 -- 172.21.15.116:0/3903786478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b14009840 con 0x7f6b1c102930 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b1b7fe640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 0x7f6b1c19d850 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c037040 tx=0x7f6b0c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c02c7c0 con 0x7f6b1c102930 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0c009490 con 0x7f6b1c102930 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f6b0c0044f0 con 0x7f6b1c102930 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c03d3f0 con 0x7f6b1c102930 2024-03-24T09:32:32.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.564+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1c1a4710 con 0x7f6b1c102930 2024-03-24T09:32:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.566+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b1c068500 con 0x7f6b1c102930 2024-03-24T09:32:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.566+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f6b0c03b040 con 0x7f6b1c102930 2024-03-24T09:32:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.567+0000 7f6b197fa640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b080a2230 0x7f6b080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.567+0000 7f6b13fff640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b080a2230 0x7f6b080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.567+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f6b0c0efcf0 con 0x7f6b1c102930 2024-03-24T09:32:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.567+0000 7f6b13fff640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b080a2230 0x7f6b080a46f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c10dcb0 tx=0x7f6b04009260 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.571+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f6b0c0b8740 con 0x7f6b1c102930 2024-03-24T09:32:32.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.743+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6b1c106df0 con 0x7f6b1c102930 2024-03-24T09:32:32.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.744+0000 7f6b197fa640 1 -- 172.21.15.116:0/3903786478 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) ==== 93+0+10 (secure 0 0 0) 0x7f6b0c0bbe00 con 0x7f6b1c102930 2024-03-24T09:32:32.745 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:32:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.745+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b080a2230 msgr2=0x7f6b080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.745+0000 7f6b21b80640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b080a2230 0x7f6b080a46f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c10dcb0 tx=0x7f6b04009260 comp rx=0 tx=0).stop 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.745+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 msgr2=0x7f6b1c19d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.745+0000 7f6b21b80640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 0x7f6b1c19d850 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c037040 tx=0x7f6b0c034670 comp rx=0 tx=0).stop 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 shutdown_connections 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f6b080a2230 0x7f6b080a46f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f6b1c10bc70 0x7f6b1c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b1c103250 0x7f6b1c19dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 --2- 172.21.15.116:0/3903786478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b1c102930 0x7f6b1c19d850 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 >> 172.21.15.116:0/3903786478 conn(0x7f6b1c0777c0 msgr2=0x7f6b1c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:32.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 shutdown_connections 2024-03-24T09:32:32.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.746+0000 7f6b21b80640 1 -- 172.21.15.116:0/3903786478 wait complete. 2024-03-24T09:32:32.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:32:32.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3337516632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa881005c0 msgr2=0x7faa88100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3337516632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa881005c0 0x7faa88100a40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009890 tx=0x7faa7c02bb70 comp rx=0 tx=0).stop 2024-03-24T09:32:32.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3337516632 shutdown_connections 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3337516632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa88100f80 0x7faa8810d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3337516632 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa881005c0 0x7faa88100a40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3337516632 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa88106660 0x7faa88106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3337516632 >> 172.21.15.116:0/3337516632 conn(0x7faa88076990 msgr2=0x7faa88076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3337516632 shutdown_connections 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3337516632 wait complete. 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 Processor -- start 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.861+0000 7faa8d5dd640 1 -- start start 2024-03-24T09:32:32.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 0x7faa8819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa88100f80 0x7faa8819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa88106660 0x7faa881969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa88196ff0 con 0x7faa881005c0 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faa88197160 con 0x7faa88106660 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7faa88197320 con 0x7faa88100f80 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 0x7faa8819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 0x7faa8819c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51112/0 (socket says 172.21.15.116:51112) 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 -- 172.21.15.116:0/3692731946 learned_addr learned my addr 172.21.15.116:0/3692731946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:32.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa877fe640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa88106660 0x7faa881969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa867fc640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa88100f80 0x7faa8819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 -- 172.21.15.116:0/3692731946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa88100f80 msgr2=0x7faa8819cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa88100f80 0x7faa8819cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 -- 172.21.15.116:0/3692731946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa88106660 msgr2=0x7faa881969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa88106660 0x7faa881969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 -- 172.21.15.116:0/3692731946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa80009560 con 0x7faa881005c0 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa86ffd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 0x7faa8819c7f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7faa7800eb70 tx=0x7faa7800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:32.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78003b70 con 0x7faa881005c0 2024-03-24T09:32:32.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7c009490 con 0x7faa881005c0 2024-03-24T09:32:32.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7faa7800ee70 con 0x7faa881005c0 2024-03-24T09:32:32.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78010690 con 0x7faa881005c0 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.862+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa881a36b0 con 0x7faa881005c0 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.864+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa88101d00 con 0x7faa881005c0 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.864+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7faa78004020 con 0x7faa881005c0 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.865+0000 7faa6ffff640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faa700a2230 0x7faa700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.865+0000 7faa867fc640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faa700a2230 0x7faa700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.865+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7faa78014070 con 0x7faa881005c0 2024-03-24T09:32:32.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.865+0000 7faa867fc640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faa700a2230 0x7faa700a46f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0097b0 tx=0x7faa7c0058d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:32.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:32.869+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7faa78097370 con 0x7faa881005c0 2024-03-24T09:32:33.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.045+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faa88197ed0 con 0x7faa881005c0 2024-03-24T09:32:33.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.045+0000 7faa6ffff640 1 -- 172.21.15.116:0/3692731946 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) ==== 104+0+11 (secure 0 0 0) 0x7faa88197ed0 con 0x7faa881005c0 2024-03-24T09:32:33.047 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faa700a2230 msgr2=0x7faa700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faa700a2230 0x7faa700a46f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0097b0 tx=0x7faa7c0058d0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 msgr2=0x7faa8819c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 0x7faa8819c7f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7faa7800eb70 tx=0x7faa7800c6a0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 shutdown_connections 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa88106660 0x7faa881969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7faa700a2230 0x7faa700a46f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7faa88100f80 0x7faa8819cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 --2- 172.21.15.116:0/3692731946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa881005c0 0x7faa8819c7f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 >> 172.21.15.116:0/3692731946 conn(0x7faa88076990 msgr2=0x7faa880ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:33.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 shutdown_connections 2024-03-24T09:32:33.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.047+0000 7faa8d5dd640 1 -- 172.21.15.116:0/3692731946 wait complete. 2024-03-24T09:32:33.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:33 smithi116 ceph-mon[32207]: pgmap v324: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:33.058 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:33 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2186608036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:32:33.059 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:33 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3903786478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:32:33.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:32:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 -- 172.21.15.116:0/2048044685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0718103250 msgr2=0x7f071810b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 --2- 172.21.15.116:0/2048044685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0718103250 0x7f071810b730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0708009860 tx=0x7f070802bc00 comp rx=0 tx=0).stop 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 -- 172.21.15.116:0/2048044685 shutdown_connections 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 --2- 172.21.15.116:0/2048044685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f071810bc70 0x7f071810e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 --2- 172.21.15.116:0/2048044685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0718103250 0x7f071810b730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 --2- 172.21.15.116:0/2048044685 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0718102930 0x7f0718102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.165+0000 7f071e78a640 1 -- 172.21.15.116:0/2048044685 >> 172.21.15.116:0/2048044685 conn(0x7f07180777c0 msgr2=0x7f0718077bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 -- 172.21.15.116:0/2048044685 shutdown_connections 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 -- 172.21.15.116:0/2048044685 wait complete. 2024-03-24T09:32:33.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 Processor -- start 2024-03-24T09:32:33.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 -- start start 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 0x7f071819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0718103250 0x7f071819ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071810bc70 0x7f0718197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0718198090 con 0x7f0718102930 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0718198200 con 0x7f071810bc70 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.166+0000 7f071e78a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f07181983c0 con 0x7f0718103250 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 0x7f071819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 0x7f071819d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51126/0 (socket says 172.21.15.116:51126) 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 -- 172.21.15.116:0/4150265059 learned_addr learned my addr 172.21.15.116:0/4150265059 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f071cd27640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071810bc70 0x7f0718197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f07177fe640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0718103250 0x7f071819ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 -- 172.21.15.116:0/4150265059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0718103250 msgr2=0x7f071819ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0718103250 0x7f071819ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 -- 172.21.15.116:0/4150265059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071810bc70 msgr2=0x7f0718197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071810bc70 0x7f0718197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 -- 172.21.15.116:0/4150265059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0708009490 con 0x7f0718102930 2024-03-24T09:32:33.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f071cd27640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071810bc70 0x7f0718197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f0717fff640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 0x7f071819d890 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f070c00e9f0 tx=0x7f070c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c009700 con 0x7f0718102930 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07181986a0 con 0x7f0718102930 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f070c00cd80 con 0x7f0718102930 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070c018840 con 0x7f0718102930 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.167+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07181a4930 con 0x7f0718102930 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.168+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0718068500 con 0x7f0718102930 2024-03-24T09:32:33.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.169+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f070c0189a0 con 0x7f0718102930 2024-03-24T09:32:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.170+0000 7f07157fa640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f07000a2230 0x7f07000a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.170+0000 7f07177fe640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f07000a2230 0x7f07000a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.170+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f070c014070 con 0x7f0718102930 2024-03-24T09:32:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.170+0000 7f07177fe640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f07000a2230 0x7f07000a46f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f070802c210 tx=0x7f07080058d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.174+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f070c053280 con 0x7f0718102930 2024-03-24T09:32:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:33 smithi143 ceph-mon[35965]: pgmap v324: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:33 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2186608036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:32:33.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:33 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3903786478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:32:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:33 smithi117 ceph-mon[35788]: pgmap v324: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:33 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2186608036' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:32:33.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:33 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3903786478' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:32:33.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.355+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0718106df0 con 0x7f0718102930 2024-03-24T09:32:33.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.356+0000 7f07157fa640 1 -- 172.21.15.116:0/4150265059 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) ==== 100+0+11 (secure 0 0 0) 0x7f070c097120 con 0x7f0718102930 2024-03-24T09:32:33.357 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:32:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f07000a2230 msgr2=0x7f07000a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f07000a2230 0x7f07000a46f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f070802c210 tx=0x7f07080058d0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 msgr2=0x7f071819d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 0x7f071819d890 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f070c00e9f0 tx=0x7f070c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 shutdown_connections 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f071810bc70 0x7f0718197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f07000a2230 0x7f07000a46f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0718103250 0x7f071819ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 --2- 172.21.15.116:0/4150265059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718102930 0x7f071819d890 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 >> 172.21.15.116:0/4150265059 conn(0x7f07180777c0 msgr2=0x7f07180fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 shutdown_connections 2024-03-24T09:32:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.358+0000 7f071e78a640 1 -- 172.21.15.116:0/4150265059 wait complete. 2024-03-24T09:32:33.370 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:32:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 -- 172.21.15.116:0/4125771390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b41005c0 msgr2=0x7f36b4100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 --2- 172.21.15.116:0/4125771390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b41005c0 0x7f36b4100a40 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f369c009930 tx=0x7f369c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:32:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 -- 172.21.15.116:0/4125771390 shutdown_connections 2024-03-24T09:32:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 --2- 172.21.15.116:0/4125771390 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b4100f80 0x7f36b410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 --2- 172.21.15.116:0/4125771390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b41005c0 0x7f36b4100a40 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 --2- 172.21.15.116:0/4125771390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b4106660 0x7f36b4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 -- 172.21.15.116:0/4125771390 >> 172.21.15.116:0/4125771390 conn(0x7f36b4076990 msgr2=0x7f36b4076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 -- 172.21.15.116:0/4125771390 shutdown_connections 2024-03-24T09:32:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.487+0000 7f36b8981640 1 -- 172.21.15.116:0/4125771390 wait complete. 2024-03-24T09:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 Processor -- start 2024-03-24T09:32:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 -- start start 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b41005c0 0x7f36b4198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 0x7f36b4199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b4106660 0x7f36b41a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36b4199810 con 0x7f36b4100f80 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36b4199980 con 0x7f36b4106660 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b8981640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f36b41a40f0 con 0x7f36b41005c0 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 0x7f36b4199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 0x7f36b4199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51140/0 (socket says 172.21.15.116:51140) 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36abfff640 1 -- 172.21.15.116:0/905333581 learned_addr learned my addr 172.21.15.116:0/905333581 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.488+0000 7f36b259c640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b41005c0 0x7f36b4198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36b2d9d640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b4106660 0x7f36b41a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36abfff640 1 -- 172.21.15.116:0/905333581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b41005c0 msgr2=0x7f36b4198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36abfff640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b41005c0 0x7f36b4198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36abfff640 1 -- 172.21.15.116:0/905333581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b4106660 msgr2=0x7f36b41a3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36abfff640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b4106660 0x7f36b41a3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36abfff640 1 -- 172.21.15.116:0/905333581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a4009490 con 0x7f36b4100f80 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36b259c640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b41005c0 0x7f36b4198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36abfff640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 0x7f36b4199020 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f369c009230 tx=0x7f369c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c004350 con 0x7f36b4100f80 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f369c02de90 con 0x7f36b4100f80 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f369c009560 con 0x7f36b4100f80 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c005220 con 0x7f36b4100f80 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.489+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b41a46f0 con 0x7f36b4100f80 2024-03-24T09:32:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.490+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b4101d00 con 0x7f36b4100f80 2024-03-24T09:32:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.491+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f369c003c30 con 0x7f36b4100f80 2024-03-24T09:32:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.492+0000 7f36ab7fe640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.492+0000 7f36b259c640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.492+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f369c0f0a80 con 0x7f36b4100f80 2024-03-24T09:32:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.492+0000 7f36b259c640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f36b4101700 tx=0x7f36a4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.495+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f369c0b9410 con 0x7f36b4100f80 2024-03-24T09:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.674+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f36b4108b50 con 0x7f36b4100f80 2024-03-24T09:32:33.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.675+0000 7f36ab7fe640 1 -- 172.21.15.116:0/905333581 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) ==== 103+0+11 (secure 0 0 0) 0x7f369c0bcad0 con 0x7f36b4100f80 2024-03-24T09:32:33.676 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 msgr2=0x7f36a00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f36b4101700 tx=0x7f36a4005f90 comp rx=0 tx=0).stop 2024-03-24T09:32:33.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 msgr2=0x7f36b4199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 0x7f36b4199020 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f369c009230 tx=0x7f369c005dd0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 shutdown_connections 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36b4106660 0x7f36b41a3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36b4100f80 0x7f36b4199020 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f36a00a2230 0x7f36a00a46f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 --2- 172.21.15.116:0/905333581 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f36b41005c0 0x7f36b4198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.677+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 >> 172.21.15.116:0/905333581 conn(0x7f36b4076990 msgr2=0x7f36b40fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:33.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.678+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 shutdown_connections 2024-03-24T09:32:33.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.678+0000 7f36b8981640 1 -- 172.21.15.116:0/905333581 wait complete. 2024-03-24T09:32:33.693 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3366334713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b84100d40 msgr2=0x7f4b8410d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3366334713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b84100d40 0x7f4b8410d4e0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009ca0 tx=0x7f4b7c02c180 comp rx=0 tx=0).stop 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3366334713 shutdown_connections 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3366334713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b84100d40 0x7f4b8410d4e0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3366334713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84106fd0 0x7f4b84100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3366334713 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b841066b0 0x7f4b84106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3366334713 >> 172.21.15.116:0/3366334713 conn(0x7f4b84077710 msgr2=0x7f4b84077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:33.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3366334713 shutdown_connections 2024-03-24T09:32:33.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3366334713 wait complete. 2024-03-24T09:32:33.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.795+0000 7f4b89c2c640 1 Processor -- start 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 -- start start 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84100d40 0x7f4b84196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 0x7f4b84196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b84106fd0 0x7f4b84197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841a5fd0 con 0x7f4b841066b0 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841a6140 con 0x7f4b84100d40 2024-03-24T09:32:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b89c2c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f4b841a6300 con 0x7f4b84106fd0 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 0x7f4b84196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 0x7f4b84196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51164/0 (socket says 172.21.15.116:51164) 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b837fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84100d40 0x7f4b84196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3981896832 learned_addr learned my addr 172.21.15.116:0/3981896832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b83fff640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b84106fd0 0x7f4b84197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3981896832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b84106fd0 msgr2=0x7f4b84197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b84106fd0 0x7f4b84197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3981896832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84100d40 msgr2=0x7f4b84196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84100d40 0x7f4b84196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3981896832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b78009490 con 0x7f4b841066b0 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.796+0000 7f4b837fe640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84100d40 0x7f4b84196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.797+0000 7f4b82ffd640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 0x7f4b84196dd0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e970 tx=0x7f4b7800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.797+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7800cd80 con 0x7f4b841066b0 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.797+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7c009900 con 0x7f4b841066b0 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.797+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f4b780044d0 con 0x7f4b841066b0 2024-03-24T09:32:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.797+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78010430 con 0x7f4b841066b0 2024-03-24T09:32:33.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.797+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b841a6980 con 0x7f4b841066b0 2024-03-24T09:32:33.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.798+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b84068500 con 0x7f4b841066b0 2024-03-24T09:32:33.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.799+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f4b780036f0 con 0x7f4b841066b0 2024-03-24T09:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.799+0000 7f4b80ff9640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4b6c0a2230 0x7f4b6c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.800+0000 7f4b837fe640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4b6c0a2230 0x7f4b6c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.800+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9554+0+0 (secure 0 0 0) 0x7f4b78014070 con 0x7f4b841066b0 2024-03-24T09:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.800+0000 7f4b837fe640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4b6c0a2230 0x7f4b6c0a46f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b84101750 tx=0x7f4b74005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:32:33.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:33.803+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f4b78097340 con 0x7f4b841066b0 2024-03-24T09:32:34.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.034+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f4b84068e60 con 0x7f4b841066b0 2024-03-24T09:32:34.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.042+0000 7f4b80ff9640 1 -- 172.21.15.116:0/3981896832 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) ==== 81+0+736939 (secure 0 0 0) 0x7f4b7809aa00 con 0x7f4b841066b0 2024-03-24T09:32:34.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:32:34.054 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:32:34.054 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 26, 2024-03-24T09:32:34.054 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "active_gid": 15543, 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "active_name": "x", 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addrs": { 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6800", 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2615204634 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.055 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6801", 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2615204634 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addr": "172.21.15.116:6801/2615204634", 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: "active_change": "2024-03-24T09:22:05.256085+0000", 2024-03-24T09:32:34.056 INFO:teuthology.orchestra.run.smithi116.stdout: "active_mgr_features": 4540701547738038271, 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "available": true, 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "standbys": [ 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 25273, 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-24T09:32:34.057 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.058 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.059 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.060 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.062 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.064 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.065 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.066 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.067 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.068 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.069 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.070 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.071 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.072 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.073 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-24T09:32:34.074 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.075 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.076 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-24T09:32:34.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T09:32:34.078 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.079 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.080 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-24T09:32:34.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T09:32:34.082 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.083 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-24T09:32:34.085 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-24T09:32:34.086 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-24T09:32:34.089 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-24T09:32:34.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-24T09:32:34.093 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.095 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.098 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.099 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.102 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.103 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.104 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-24T09:32:34.105 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-24T09:32:34.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T09:32:34.107 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-24T09:32:34.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T09:32:34.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.111 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.112 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.116 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.117 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-24T09:32:34.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.120 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-24T09:32:34.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-24T09:32:34.122 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-24T09:32:34.123 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-24T09:32:34.124 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.125 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.126 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-24T09:32:34.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.128 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.129 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-24T09:32:34.130 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.131 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T09:32:34.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-24T09:32:34.133 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.134 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.135 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.136 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.138 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.141 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.142 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.144 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.146 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.149 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.155 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-24T09:32:34.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.157 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.158 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.159 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.161 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-24T09:32:34.162 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.163 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.165 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.167 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.168 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.169 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.171 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.172 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.173 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-24T09:32:34.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.183 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-24T09:32:34.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T09:32:34.186 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T09:32:34.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-24T09:32:34.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.189 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.190 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-24T09:32:34.191 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T09:32:34.192 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.193 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.194 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.195 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.196 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.198 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.199 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-24T09:32:34.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.203 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.204 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-24T09:32:34.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.207 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.208 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.209 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.210 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.211 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.212 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.214 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-24T09:32:34.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-24T09:32:34.216 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.218 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.219 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.220 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.221 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.222 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T09:32:34.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.225 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.227 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T09:32:34.228 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.229 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.230 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.231 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.232 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T09:32:34.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.234 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.235 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.236 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.237 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.238 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.239 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.240 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T09:32:34.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.242 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.243 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-24T09:32:34.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.245 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:34.246 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T09:32:34.247 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.248 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.249 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T09:32:34.250 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.251 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.252 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T09:32:34.253 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.254 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.255 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.257 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T09:32:34.258 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.259 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.260 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.261 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.262 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.263 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T09:32:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.265 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.266 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T09:32:34.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.268 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.269 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.270 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.271 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T09:32:34.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.273 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.274 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.276 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.277 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.278 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.279 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.280 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.282 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.285 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.286 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.288 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.289 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.291 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.292 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.294 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.295 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.296 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.297 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.298 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.299 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.300 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-24T09:32:34.301 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.302 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.303 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.304 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.305 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.306 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.307 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.308 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.309 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.310 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-24T09:32:34.311 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.312 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-24T09:32:34.313 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-24T09:32:34.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.315 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.316 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-24T09:32:34.317 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.319 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.320 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.321 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.322 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.323 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.324 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-24T09:32:34.325 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T09:32:34.326 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-24T09:32:34.327 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-24T09:32:34.328 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.329 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-24T09:32:34.330 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-24T09:32:34.331 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T09:32:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.333 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.335 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.336 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.337 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.338 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.339 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.340 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.341 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.342 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.344 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.345 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.346 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.347 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.349 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-24T09:32:34.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.351 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.352 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-24T09:32:34.353 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.354 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.355 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.356 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.357 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.358 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.359 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.360 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.361 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.362 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.364 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.365 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-24T09:32:34.366 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-24T09:32:34.367 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.368 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-24T09:32:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-24T09:32:34.372 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.373 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.374 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.376 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.377 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.380 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.381 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.382 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.383 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.384 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.385 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.386 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.387 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.388 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.389 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.390 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.394 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.395 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.396 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.397 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.398 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.399 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.400 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.401 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.402 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.403 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.404 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.405 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.406 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.408 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.409 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.410 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.412 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.413 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T09:32:34.414 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.415 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.416 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.418 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.419 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.420 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.421 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.422 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.423 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.424 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.425 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.426 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.427 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.428 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.429 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.430 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.431 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.432 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.433 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.435 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.436 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.437 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.438 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.440 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.441 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.442 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.444 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.446 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.447 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.448 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.451 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.452 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.454 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-24T09:32:34.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-24T09:32:34.458 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.460 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.461 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.463 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.465 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.466 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.469 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.471 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.472 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.473 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.474 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.475 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.476 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.477 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.478 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.482 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.483 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.484 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.487 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-24T09:32:34.488 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.489 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-24T09:32:34.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.493 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.494 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.495 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.496 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.497 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.498 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.499 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.500 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.501 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.505 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.506 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.508 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.509 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.511 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.512 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.513 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.514 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-24T09:32:34.515 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.518 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.519 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.520 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-24T09:32:34.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.522 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T09:32:34.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-24T09:32:34.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-24T09:32:34.526 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-24T09:32:34.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.528 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.529 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.530 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.531 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.532 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.534 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.536 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.537 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.539 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.541 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.542 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.543 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.544 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.545 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.547 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.548 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.549 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.550 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.551 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.552 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.553 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.554 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.555 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:34.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.558 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.559 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.560 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.562 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.563 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.565 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.567 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.568 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.569 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.572 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.573 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.574 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.575 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.578 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.579 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.581 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.591 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.592 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.599 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.603 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-24T09:32:34.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.607 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.608 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.612 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.613 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-24T09:32:34.615 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.616 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.617 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.619 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.620 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.621 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.622 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.625 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.626 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.627 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.628 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.629 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.632 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.633 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.634 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.635 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.637 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.638 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.639 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.640 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.641 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.644 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.646 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.647 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.648 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.649 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.650 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-24T09:32:34.651 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.653 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.654 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-24T09:32:34.660 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-24T09:32:34.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.664 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.665 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.666 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.667 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.669 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.671 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-24T09:32:34.675 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.676 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.677 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.678 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.680 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.684 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.686 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.687 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.691 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.693 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.695 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.698 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-24T09:32:34.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T09:32:34.700 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.703 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.709 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.713 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.719 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.720 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 25276, 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "y", 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-24T09:32:34.721 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.724 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.725 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.727 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.728 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.733 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.734 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.737 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.739 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.742 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.746 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-24T09:32:34.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T09:32:34.748 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T09:32:34.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-24T09:32:34.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-24T09:32:34.753 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.757 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.758 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.760 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.761 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.762 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.766 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-24T09:32:34.767 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-24T09:32:34.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.769 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.770 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-24T09:32:34.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T09:32:34.772 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.774 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.775 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.776 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.777 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.778 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.779 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T09:32:34.780 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.781 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.782 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.783 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.784 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T09:32:34.785 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-24T09:32:34.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T09:32:34.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-24T09:32:34.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T09:32:34.792 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-24T09:32:34.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-24T09:32:34.796 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-24T09:32:34.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T09:32:34.801 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-24T09:32:34.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.804 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.805 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.806 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.807 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.813 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.814 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.816 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.817 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.818 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.819 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.820 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-24T09:32:34.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-24T09:32:34.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-24T09:32:34.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.827 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.828 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.829 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T09:32:34.830 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.833 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.837 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.838 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T09:32:34.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.843 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.844 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-24T09:32:34.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T09:32:34.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-24T09:32:34.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-24T09:32:34.851 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.853 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-24T09:32:34.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-24T09:32:34.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-24T09:32:34.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T09:32:34.858 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-24T09:32:34.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T09:32:34.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-24T09:32:34.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T09:32:34.866 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.867 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.870 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.872 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.880 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.882 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.883 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.884 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.893 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.894 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T09:32:34.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.899 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-24T09:32:34.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T09:32:34.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T09:32:34.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T09:32:34.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T09:32:34.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T09:32:34.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.928 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T09:32:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.932 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.933 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.934 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.935 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.939 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.945 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.946 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.947 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.948 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.949 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.950 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.953 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.954 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.959 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.960 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.961 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T09:32:34.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:34.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-24T09:32:34.972 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.973 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-24T09:32:34.974 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-24T09:32:34.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.977 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.978 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.980 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.984 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:34.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.986 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.988 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T09:32:34.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.991 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.992 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.994 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T09:32:34.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:34.998 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:34.999 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.000 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-24T09:32:35.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-24T09:32:35.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:35.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.013 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.014 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.015 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.018 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.021 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.022 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.024 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-24T09:32:35.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-24T09:32:35.027 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T09:32:35.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:35.029 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-24T09:32:35.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-24T09:32:35.031 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.035 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.036 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.037 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.038 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.039 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.040 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.042 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.043 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.044 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.045 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.046 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.047 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.050 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.051 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.052 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.054 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.055 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.058 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.060 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-24T09:32:35.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T09:32:35.063 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.064 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.065 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.066 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.069 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.070 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-24T09:32:35.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.072 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.073 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.074 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T09:32:35.075 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.076 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-24T09:32:35.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T09:32:35.078 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.080 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.081 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.082 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.082 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.082 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.083 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.084 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.086 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.087 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.089 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.090 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.091 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.093 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.096 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.097 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.098 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.099 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.100 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.101 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.104 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.105 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.106 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.107 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.108 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.109 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.110 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.112 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.113 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-24T09:32:35.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-24T09:32:35.115 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.117 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.118 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.122 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.124 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.126 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.127 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.129 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.130 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.132 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.133 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.134 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.135 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.137 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.138 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.139 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.140 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.141 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.143 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.144 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.145 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.147 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.148 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.149 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.151 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.153 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.155 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.156 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-24T09:32:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.160 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-24T09:32:35.161 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T09:32:35.162 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.164 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.167 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.168 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.169 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.170 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T09:32:35.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.173 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.174 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.176 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.179 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.180 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.181 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-24T09:32:35.182 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.184 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.188 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.190 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.192 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.194 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.197 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.198 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.199 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.201 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.202 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.205 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.206 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.207 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:35.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.210 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.211 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.216 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.218 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-24T09:32:35.219 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.220 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.223 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.224 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.226 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.228 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.230 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.231 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.232 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.234 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.235 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.236 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.237 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.238 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.239 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.240 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.241 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.242 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.243 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.244 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.245 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.246 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.247 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.248 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.249 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.250 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.251 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.252 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.253 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.254 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.255 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.256 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.257 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.258 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.259 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.260 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.261 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.262 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.263 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.264 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.265 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.266 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.267 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.268 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.269 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.270 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.271 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.274 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.275 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.276 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.277 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.279 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.281 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.282 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.283 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-24T09:32:35.284 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T09:32:35.285 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.287 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.288 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.289 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.290 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.291 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.292 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.293 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.294 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.295 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.296 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T09:32:35.297 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.299 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.300 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.301 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.303 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.304 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.305 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.306 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.308 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.309 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.310 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-24T09:32:35.311 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.312 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.313 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.314 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.315 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.316 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.318 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.319 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.320 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.321 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.322 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.323 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.324 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.325 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.326 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.327 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T09:32:35.328 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.329 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.332 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.334 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.335 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.336 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.337 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.339 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.340 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.341 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.342 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.343 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.344 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.346 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.347 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.349 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.350 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.352 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.353 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.354 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.355 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.356 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.358 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.359 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.360 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.362 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.363 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.364 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.365 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.366 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.367 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-24T09:32:35.368 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T09:32:35.370 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: "modules": [ 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard", 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: "iostat", 2024-03-24T09:32:35.372 INFO:teuthology.orchestra.run.smithi116.stdout: "nfs", 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "restful", 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "snap_schedule" 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.373 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.374 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.375 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.376 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.377 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.378 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.379 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.380 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.381 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.382 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.383 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.384 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.386 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.387 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.388 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.389 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.390 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.392 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.393 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.394 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-24T09:32:35.395 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.396 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.397 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:35.398 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.399 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-24T09:32:35.400 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.401 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T09:32:35.402 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-24T09:32:35.403 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-24T09:32:35.404 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T09:32:35.405 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:35.406 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.407 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.408 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.409 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.410 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.411 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.413 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.414 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-24T09:32:35.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.416 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-24T09:32:35.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-24T09:32:35.418 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.419 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-24T09:32:35.422 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T09:32:35.423 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.424 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.426 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.427 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.428 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-24T09:32:35.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T09:32:35.430 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.431 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.432 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T09:32:35.433 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-24T09:32:35.434 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.435 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-24T09:32:35.436 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-24T09:32:35.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.438 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.440 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.441 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T09:32:35.442 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-24T09:32:35.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.444 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.445 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-24T09:32:35.446 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:35.447 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.448 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.449 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.450 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.451 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.453 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.454 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.456 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.457 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.459 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.460 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.461 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.462 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.464 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.465 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-24T09:32:35.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.468 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-24T09:32:35.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-24T09:32:35.471 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.472 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.473 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.474 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.475 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.476 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-24T09:32:35.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.478 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.479 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.480 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.481 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.482 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.483 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.484 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.485 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.486 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.487 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.488 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.489 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T09:32:35.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.491 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.492 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-24T09:32:35.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-24T09:32:35.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.497 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-24T09:32:35.498 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.499 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.500 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-24T09:32:35.501 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.505 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.506 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-24T09:32:35.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T09:32:35.510 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.511 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.514 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.515 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.516 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-24T09:32:35.517 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.518 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.519 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.521 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.523 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.524 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.526 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.527 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.528 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.529 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.531 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.534 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.535 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T09:32:35.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.537 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.538 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.540 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.542 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.543 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.544 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.545 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.546 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.547 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.548 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.550 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T09:32:35.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.552 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.553 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.555 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.556 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.557 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:35.558 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T09:32:35.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.560 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.561 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.563 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.564 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.565 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T09:32:35.567 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.569 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T09:32:35.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.571 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.572 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T09:32:35.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T09:32:35.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.577 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T09:32:35.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.580 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.581 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.582 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T09:32:35.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.588 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.590 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.591 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T09:32:35.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.594 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.597 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T09:32:35.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.599 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.600 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.601 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.603 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.605 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-24T09:32:35.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-24T09:32:35.607 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.609 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.612 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.613 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.616 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-24T09:32:35.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.624 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.629 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.630 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.631 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.632 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.633 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.636 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.640 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.641 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.643 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.644 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.650 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-24T09:32:35.653 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.655 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.659 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.661 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.664 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.666 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-24T09:32:35.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T09:32:35.669 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.670 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.671 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T09:32:35.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.674 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.674 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.674 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.674 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.674 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.677 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.678 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-24T09:32:35.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.682 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-24T09:32:35.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-24T09:32:35.684 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T09:32:35.685 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-24T09:32:35.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-24T09:32:35.688 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.690 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.693 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.696 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.699 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.700 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.702 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.704 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.708 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-24T09:32:35.710 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.712 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.713 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.716 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-24T09:32:35.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-24T09:32:35.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.723 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.727 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-24T09:32:35.732 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-24T09:32:35.733 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.734 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.735 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-24T09:32:35.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T09:32:35.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-24T09:32:35.739 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.740 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.741 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.745 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.750 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.752 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.753 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.754 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.755 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.757 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.758 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.759 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.760 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.761 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.762 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.763 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.770 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.772 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.774 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.777 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-24T09:32:35.778 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.779 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.780 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.781 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.782 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.786 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.790 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.790 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.790 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.790 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.790 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.791 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.791 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.791 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.792 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-24T09:32:35.792 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.792 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.792 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.792 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.793 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.793 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.793 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.793 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.794 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.796 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.797 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.798 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.800 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.801 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.804 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.805 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.807 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.809 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-24T09:32:35.811 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.813 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.814 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.815 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.817 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.818 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.819 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.820 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.821 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-24T09:32:35.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.827 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.830 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.831 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.834 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.835 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.836 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-24T09:32:35.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-24T09:32:35.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.841 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.842 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.844 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.845 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.846 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-24T09:32:35.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.850 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.851 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.853 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.855 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.856 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-24T09:32:35.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.872 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-24T09:32:35.878 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.882 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.885 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.888 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.892 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-24T09:32:35.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.898 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.901 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.902 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-24T09:32:35.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-24T09:32:35.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-24T09:32:35.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-24T09:32:35.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.919 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.922 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-24T09:32:35.924 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-24T09:32:35.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-24T09:32:35.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.929 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.930 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.932 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.933 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.935 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.938 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.942 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.945 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.947 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.948 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.950 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.951 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.953 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.957 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.958 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.962 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.963 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.965 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-24T09:32:35.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.968 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-24T09:32:35.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T09:32:35.970 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.972 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T09:32:35.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.974 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-24T09:32:35.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T09:32:35.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-24T09:32:35.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-24T09:32:35.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-24T09:32:35.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-24T09:32:35.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.987 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:35.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:35.991 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:35.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:35.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:35.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:36.000 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.002 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.006 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:36.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:36.008 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:36.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.011 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:36.012 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.017 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.020 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-24T09:32:36.021 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.022 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.024 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.025 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:36.027 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-24T09:32:36.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.030 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-24T09:32:36.031 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.034 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.035 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.036 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-24T09:32:36.037 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: "services": { 2024-03-24T09:32:36.038 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard": "https://172.21.15.116:8443/" 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "always_on_modules": { 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus": [ 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-24T09:32:36.039 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific": [ 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-24T09:32:36.040 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy": [ 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-24T09:32:36.041 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-24T09:32:36.042 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "reef": [ 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-24T09:32:36.043 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "squid": [ 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-24T09:32:36.044 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "last_failure_osd_epoch": 94, 2024-03-24T09:32:36.045 INFO:teuthology.orchestra.run.smithi116.stdout: "active_clients": [ 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3085526057 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.046 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "libcephsqlite", 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2800690092 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.047 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1265798029 2024-03-24T09:32:36.048 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-24T09:32:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 368906330 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-24T09:32:36.050 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1310109461 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-24T09:32:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-24T09:32:36.052 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 4031791706 2024-03-24T09:32:36.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.052 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.052 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:36.052 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-24T09:32:36.052 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:32:36.054 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:34 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3692731946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.054 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:34 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4150265059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.054 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:34 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/905333581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.054 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:35 smithi117 ceph-mon[35788]: pgmap v325: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:36.054 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:35 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3981896832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T09:32:36.054 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:34 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3692731946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.055 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:34 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4150265059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.055 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:34 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/905333581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.055 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:35 smithi143 ceph-mon[35965]: pgmap v325: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:36.055 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:35 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3981896832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T09:32:36.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4b6c0a2230 msgr2=0x7f4b6c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4b6c0a2230 0x7f4b6c0a46f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4b84101750 tx=0x7f4b74005f90 comp rx=0 tx=0).stop 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 msgr2=0x7f4b84196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 0x7f4b84196dd0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e970 tx=0x7f4b7800ee30 comp rx=0 tx=0).stop 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 shutdown_connections 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f4b6c0a2230 0x7f4b6c0a46f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4b84106fd0 0x7f4b84197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b841066b0 0x7f4b84196dd0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 --2- 172.21.15.116:0/3981896832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b84100d40 0x7f4b84196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:32:36.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 >> 172.21.15.116:0/3981896832 conn(0x7f4b84077710 msgr2=0x7f4b840ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 shutdown_connections 2024-03-24T09:32:36.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:32:34.047+0000 7f4b89c2c640 1 -- 172.21.15.116:0/3981896832 wait complete. 2024-03-24T09:32:36.063 DEBUG:tasks.check_counter:Getting stats from a 2024-03-24T09:32:36.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-24T09:32:36.064 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:34 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3692731946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.064 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:34 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4150265059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.064 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:34 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/905333581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:32:36.064 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:35 smithi116 ceph-mon[32207]: pgmap v325: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:36.064 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:35 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3981896832' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T09:32:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:32:37.068 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:37 smithi116 ceph-mon[32207]: pgmap v326: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:37.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:37 smithi143 ceph-mon[35965]: pgmap v326: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:37.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:37 smithi117 ceph-mon[35788]: pgmap v326: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:38.342 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:32:38.342 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1514, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1496, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 5192815, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 527159, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.167275377, 2024-03-24T09:32:38.343 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.059280674, 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.065513432, 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.040524595, 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1495, 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.057288670, 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000038320 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.344 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1642896, 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 133744 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.345 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 671, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 831, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 8555050, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 651263, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 8, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 6, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.073951363, 2024-03-24T09:32:38.346 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.026295085, 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.019581859, 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.020538371, 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 828, 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.038694054, 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000046731 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:38.347 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 443136 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.348 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 294, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 294, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 8466789, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 90639, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 5, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.023674119, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.005969292, 2024-03-24T09:32:38.349 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.010227436, 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.005821527, 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 293, 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.008994682, 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000030698 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.350 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T09:32:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3746, 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.099420680, 2024-03-24T09:32:38.352 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000026540 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.353 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-24T09:32:38.354 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-24T09:32:38.355 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 1928, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 1, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 1927, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 1927, 2024-03-24T09:32:38.356 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 47, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-24T09:32:38.357 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-24T09:32:38.358 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T09:32:38.359 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T09:32:38.360 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-24T09:32:38.361 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-24T09:32:38.362 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:32:38.363 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:32:38.364 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 29490, 2024-03-24T09:32:38.365 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 33, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 4, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-24T09:32:38.366 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 24134845, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 24134271, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 29490 2024-03-24T09:32:38.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1071, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 1622, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 551, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 68, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 69, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 1, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 1928, 2024-03-24T09:32:38.368 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 1941, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 13, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 59664, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-24T09:32:38.369 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.371 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.372 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-24T09:32:38.373 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.374 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.375 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.377 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.378 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.379 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.381 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-24T09:32:38.382 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.383 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.384 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.385 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-24T09:32:38.386 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 1, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 2, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 1, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 2, 2024-03-24T09:32:38.387 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 2773, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:38.388 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:38.389 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:38.390 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 214417, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 65, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T09:32:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 9704, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 186, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:38.393 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 3717122, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 13957, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T09:32:38.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 2224, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 2224, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2224, 2024-03-24T09:32:38.395 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.002983535, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000450981 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2224, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2225 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 2224, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 2224, 2024-03-24T09:32:38.396 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 1387, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 835, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-24T09:32:38.397 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:32:38.398 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-24T09:32:38.399 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-24T09:32:38.400 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 102, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-24T09:32:38.401 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 13, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 8, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 8, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-24T09:32:38.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-24T09:32:38.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-24T09:32:38.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 2475, 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 21976165, 2024-03-24T09:32:38.405 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 2475, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2475, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 21976165, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.406 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:38.407 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 2224, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 19940546, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2224, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 19940546, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.408 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:38.409 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 2224, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 2224, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2224, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 2224, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.410 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:38.411 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:38.412 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x55ac9d59a6e0": { 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:38.413 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-24T09:32:38.414 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:38.415 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:38.415 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:38.415 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:38.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.415 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:38.415 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:32:39.183 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-24T09:32:39.183 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-24T09:32:39.183 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-24T09:32:39.183 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-24T09:32:39.183 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-03-24T09:32:39.184 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:32:39.184 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:32:39.184 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-24T09:32:39.184 DEBUG:tasks.check_counter:Getting stats from d 2024-03-24T09:32:39.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-24T09:32:39.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:39 smithi143 ceph-mon[35965]: pgmap v327: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:39.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:39 smithi117 ceph-mon[35788]: pgmap v327: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:39.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:39 smithi116 ceph-mon[32207]: pgmap v327: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:32:41.202 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:41 smithi116 ceph-mon[32207]: pgmap v328: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:41.202 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-24T09:32:41.203 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.001112808, 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000278007, 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:41.204 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.205 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 233, 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 220, 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1573680, 2024-03-24T09:32:41.206 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 134272, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.041559730, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.013590158, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.032695235, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.004357994, 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:41.207 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 220, 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.010753706, 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000048880 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:41.208 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1573680, 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 134272 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2, 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 166, 2024-03-24T09:32:41.209 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 320, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 184496, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 5, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.013690405, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.009424679, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.001631350, 2024-03-24T09:32:41.210 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 163, 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.014187281, 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000087038 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:41.211 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 184496 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.212 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:41.213 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:41.214 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:41.215 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:41.216 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:41.217 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 149057, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 32, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:41.218 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-24T09:32:41.219 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T09:32:41.220 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:41.221 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 231, 2024-03-24T09:32:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 1538391, 2024-03-24T09:32:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 231, 2024-03-24T09:32:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:41.222 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 231, 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1538391, 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:41.223 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:32:41.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:41 smithi143 ceph-mon[35965]: pgmap v328: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:41.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:41 smithi117 ceph-mon[35788]: pgmap v328: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:41.974 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-24T09:32:41.974 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-24T09:32:41.974 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-24T09:32:41.974 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-24T09:32:41.974 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-24T09:32:41.975 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-24T09:32:41.975 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-24T09:32:41.975 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-24T09:32:41.975 DEBUG:tasks.check_counter:Getting stats from g 2024-03-24T09:32:41.975 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-24T09:32:42.219 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:32:43.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:43 smithi143 ceph-mon[35965]: pgmap v329: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:43.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:43 smithi117 ceph-mon[35788]: pgmap v329: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:43.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:43 smithi116 ceph-mon[32207]: pgmap v329: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:44.062 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:32:44.063 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:44.063 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1510, 2024-03-24T09:32:44.063 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1492, 2024-03-24T09:32:44.063 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 6215536, 2024-03-24T09:32:44.063 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 526573, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 6, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.164626588, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.064154474, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.062042332, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.038635915, 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:44.064 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1491, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.052315315, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000035087 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1643440, 2024-03-24T09:32:44.065 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 135024 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 929, 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1091, 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 18748384, 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 735074, 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 7, 2024-03-24T09:32:44.066 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 5, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.083733832, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.028018017, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.028976440, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.019226844, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1088, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.036211331, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000033282 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.067 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 446720 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.068 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 48, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 48, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 4209117, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 14144, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 5, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.005910073, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000786598, 2024-03-24T09:32:44.069 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.003996521, 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000869486, 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 47, 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.003154996, 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000067127 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.070 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T09:32:44.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3785, 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.089017335, 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000023518 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.072 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.073 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-24T09:32:44.074 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 1874, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 0, 2024-03-24T09:32:44.075 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 1874, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 1874, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 50, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-24T09:32:44.076 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-24T09:32:44.077 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-24T09:32:44.078 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-24T09:32:44.079 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-24T09:32:44.080 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-24T09:32:44.081 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:32:44.082 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-24T09:32:44.083 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 41614, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 46, 2024-03-24T09:32:44.084 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 5, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 31090662, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 31090088, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.085 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 41614 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 1153, 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 1659, 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 506, 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 68, 2024-03-24T09:32:44.086 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 68, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 0, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 1874, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 1874, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 0, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 56856, 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-24T09:32:44.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-24T09:32:44.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-24T09:32:44.089 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.090 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.091 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.092 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.093 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.094 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.095 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-24T09:32:44.095 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.095 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.096 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.098 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-24T09:32:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.100 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-24T09:32:44.101 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.102 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.103 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-24T09:32:44.104 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:32:44.106 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 2, 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 2, 2024-03-24T09:32:44.107 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 0, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 2, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 1708, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T09:32:44.108 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:44.109 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:44.110 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:44.111 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 206377, 2024-03-24T09:32:44.112 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 62, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:44.113 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 9704, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 186, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 3847101, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 13570, 2024-03-24T09:32:44.114 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 2230, 2024-03-24T09:32:44.115 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 2230, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2230, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.987983748, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000443042 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-24T09:32:44.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2230, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2231 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 2230, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 2230, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 1396, 2024-03-24T09:32:44.117 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 832, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-24T09:32:44.118 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-24T09:32:44.119 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-24T09:32:44.120 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-24T09:32:44.121 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 102, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 13, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 7, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 7, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 0, 2024-03-24T09:32:44.122 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:32:44.123 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T09:32:44.124 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T09:32:44.125 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 2483, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 28933724, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 2483, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2483, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 28933724, 2024-03-24T09:32:44.126 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T09:32:44.127 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 2230, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 26896363, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2230, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 26896363, 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:44.128 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:44.129 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 2230, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 2230, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 2230, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 2230, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.130 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:44.131 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.132 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x564299b446e0": { 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:44.133 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:44.134 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.135 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:44.135 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:32:44.857 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-24T09:32:44.857 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-24T09:32:44.857 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-24T09:32:44.858 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-24T09:32:44.858 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-24T09:32:44.858 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:32:44.858 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:32:44.858 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-24T09:32:44.858 DEBUG:tasks.check_counter:Getting stats from j 2024-03-24T09:32:44.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-24T09:32:45.098 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T09:32:45.220 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:45 smithi116 ceph-mon[32207]: pgmap v330: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:45.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:45 smithi143 ceph-mon[35965]: pgmap v330: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:45.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:45 smithi117 ceph-mon[35788]: pgmap v330: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:46.853 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-24T09:32:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 231, 2024-03-24T09:32:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 220, 2024-03-24T09:32:46.853 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1571232, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 135104, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.045192709, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.014405119, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.036310863, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.004485191, 2024-03-24T09:32:46.854 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 219, 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.011703469, 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000053440 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:46.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1571232, 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 135104 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 6, 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 171, 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 3312, 2024-03-24T09:32:46.856 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 185472, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 5, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.014295214, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.009763195, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002330927, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 169, 2024-03-24T09:32:46.857 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.019534385, 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000115588 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.858 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 185472 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-24T09:32:46.859 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.000321075, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000222102, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:46.860 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:32:46.861 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:46.862 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:46.863 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:46.864 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:46.865 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:46.866 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 148977, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 30, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:46.867 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-24T09:32:46.868 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-24T09:32:46.869 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 233, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 1538651, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 233, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 233, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1538651, 2024-03-24T09:32:46.870 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-24T09:32:46.871 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-24T09:32:46.871 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-24T09:32:46.871 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-24T09:32:46.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:46.871 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-24T09:32:46.871 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-24T09:32:47.259 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:47 smithi116 ceph-mon[32207]: pgmap v331: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:47.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:47 smithi143 ceph-mon[35965]: pgmap v331: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:47.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:47 smithi117 ceph-mon[35788]: pgmap v331: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:47.627 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-24T09:32:47.627 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-24T09:32:47.627 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-24T09:32:47.627 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-24T09:32:47.627 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-24T09:32:47.628 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-24T09:32:47.628 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-24T09:32:47.628 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-24T09:32:47.628 DEBUG:tasks.check_counter:Getting stats from b 2024-03-24T09:32:47.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-24T09:32:47.961 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:32:49.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:49 smithi143 ceph-mon[35965]: pgmap v332: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:49.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:49 smithi117 ceph-mon[35788]: pgmap v332: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:49.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:49 smithi116 ceph-mon[32207]: pgmap v332: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:50.167 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T09:32:50.168 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:50.168 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 17749, 2024-03-24T09:32:50.168 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 23686, 2024-03-24T09:32:50.168 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 7946201, 2024-03-24T09:32:50.168 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 58156391, 2024-03-24T09:32:50.168 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 15, 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 1.419268275, 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.519105137, 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.806285390, 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.158907355, 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:50.169 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 23683, 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.206104353, 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000050927 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 32350, 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.007896898, 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000244 2024-03-24T09:32:50.170 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1701616, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 488992 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 46169, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 67964, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 19658873, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 79711289, 2024-03-24T09:32:50.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 28, 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 3.501191135, 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 1.546193856, 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 1.954969794, 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.224746990, 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:50.172 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 67964, 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2.574671358, 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000037882 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 103653, 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.020761882, 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000200 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 409952, 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 256848 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 31342, 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 43474, 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 13428665, 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 114795480, 2024-03-24T09:32:50.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 152, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 6, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 2.642924126, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 1.258458082, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 1.443634032, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.155612880, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 43472, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.337645781, 2024-03-24T09:32:50.175 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000030770 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 64558, 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.013571278, 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000210 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 42224, 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 976 2024-03-24T09:32:50.176 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 11670, 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 18.218397104, 2024-03-24T09:32:50.177 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001561130 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2695, 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.479268281, 2024-03-24T09:32:50.178 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000177836 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 17688, 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 16072, 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 16072, 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 68.170558918, 2024-03-24T09:32:50.179 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004241572 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 1616, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 194, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 22, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T09:32:50.180 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 348, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 159, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 157, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 32, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 120, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 4503, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 62, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 62, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 120, 2024-03-24T09:32:50.181 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 27585, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 22118, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 1593, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 5, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 2, 2024-03-24T09:32:50.182 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 179, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 4714, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 4372, 2024-03-24T09:32:50.183 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 3452, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 26526152, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 4372, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 494, 2024-03-24T09:32:50.184 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 19, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 475, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 3878, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 4001, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 6779, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 8, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:32:50.185 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 9470, 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 5238, 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 66, 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 8523 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 438, 2024-03-24T09:32:50.186 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 9, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 6, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 8, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 1522, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 2, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 1440, 2024-03-24T09:32:50.187 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 1436, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 2, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:32:50.188 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 19, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 200, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 22, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T09:32:50.189 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 17786, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 17786, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 27, 2024-03-24T09:32:50.190 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 27, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 3, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 57478682, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-24T09:32:50.191 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4508, 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 11.390199368, 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002526663 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T09:32:50.192 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 311, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 5661, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 5350, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 177, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 274, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 97, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 348, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 6449, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 6101, 2024-03-24T09:32:50.193 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 62, 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 5447, 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 5385, 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 89656, 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 231804 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 22690, 2024-03-24T09:32:50.194 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 3122, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 17688, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 217, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 6211, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 5346, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 17.943562743, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.003356446 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.195 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1123, 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.345445812, 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000307609 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.004153404, 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002076702 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T09:32:50.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 7788, 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 42.998067906, 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.005521066 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.198 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.199 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.200 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.201 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.201 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T09:32:50.201 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10, 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002216384, 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000221638 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 83, 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.727971125, 2024-03-24T09:32:50.202 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.020818929 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10, 2024-03-24T09:32:50.203 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.055892585, 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.005589258 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 118, 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.132692464, 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001124512 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T09:32:50.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 77, 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.619571781, 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.008046386 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 76, 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.194698457, 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002561821 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.205 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 21, 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.132095735, 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.006290273 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4, 2024-03-24T09:32:50.207 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.018590339, 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004647584 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8, 2024-03-24T09:32:50.208 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.015613049, 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001951631 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T09:32:50.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8, 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.004087396, 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000510924 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.210 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8, 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.018443899, 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002305487 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 6, 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.007524101, 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001254016 2024-03-24T09:32:50.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1384, 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 3.949931738, 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002853996 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:32:50.212 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-24T09:32:50.213 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 6, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 4, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 84, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 42, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 616, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.214 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:50.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:50.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:50.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:50.218 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 322002, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 121, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:50.219 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9704, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 186, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 1278403, 2024-03-24T09:32:50.220 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 11154, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T09:32:50.221 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 12983, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 56042838, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 33, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 12983, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 12983, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 40.410212374, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.003112548 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 12983, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 13380 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 12983, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 4527, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 8456, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T09:32:50.223 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 11, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 136, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 4625, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 92, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T09:32:50.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 6, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 3400, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 4391, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 22, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T09:32:50.225 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 8, 2024-03-24T09:32:50.226 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T09:32:50.227 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 102, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T09:32:50.228 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 18, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 12, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 12, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 587, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 102 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.229 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T09:32:50.230 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 6, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 2860, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 1436, 2024-03-24T09:32:50.231 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 95256, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 32393463, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:50.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 95256, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 95256, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 32393463, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:50.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 12983, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 56200202, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 12983, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 56200202, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.235 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:50.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 12983, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 12983, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 12983, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 12983, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.237 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 1436, 2024-03-24T09:32:50.238 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 157364, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 1436, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 117, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 157364, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.239 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x563787c46500": { 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 17786, 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 53284378, 2024-03-24T09:32:50.240 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 17786, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 4508, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 53284378, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:50.241 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:50.242 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:50.242 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.242 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:50.242 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds.exported=179 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=438 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=9 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-24T09:32:50.947 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-24T09:32:50.948 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=22 2024-03-24T09:32:50.948 DEBUG:tasks.check_counter:Getting stats from e 2024-03-24T09:32:50.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-24T09:32:51.272 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:32:51.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:51 smithi117 ceph-mon[35788]: pgmap v333: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:51.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:51 smithi116 ceph-mon[32207]: pgmap v333: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:51.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:51 smithi143 ceph-mon[35965]: pgmap v333: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:53.169 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T09:32:53.170 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:53.170 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 24011, 2024-03-24T09:32:53.170 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 18042, 2024-03-24T09:32:53.170 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 45269578, 2024-03-24T09:32:53.170 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 16443417, 2024-03-24T09:32:53.170 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 11, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 1.570452971, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.679603594, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.959610397, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.114622056, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 18044, 2024-03-24T09:32:53.171 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.688410906, 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000038151 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 33377, 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.006100559, 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000182 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:32:53.172 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 5947, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 6148, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1566079, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 10050537, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 45, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-24T09:32:53.173 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.502241951, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.200647904, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.116881568, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.148350796, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 6149, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.536410417, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000087235 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.174 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 918, 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000147295, 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000160 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 453056 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.175 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 10849, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 13385, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 4662929, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 9464695, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 19, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 10, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.755290997, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.273673218, 2024-03-24T09:32:53.176 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.334553592, 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.156818239, 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 13377, 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.660177951, 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000049351 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 11725, 2024-03-24T09:32:53.177 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002003159, 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000170 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1677488, 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 159520 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T09:32:53.178 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10009, 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 4.815949014, 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000481161 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.179 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1300, 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.094450406, 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000072654 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 4187, 2024-03-24T09:32:53.180 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 3234, 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3234, 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 4.891602251, 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001512554 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 953, 2024-03-24T09:32:53.181 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 11, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 1413, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 23, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 1390, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-24T09:32:53.182 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 1380, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 8, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 812, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 812, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 135, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 9389, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 3535, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 953, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 39, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:32:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 45, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 1136, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T09:32:53.184 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 12717, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 3452, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 26526152, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 12717, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 46, 2024-03-24T09:32:53.185 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 46, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 12671, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 21, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 911, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:32:53.186 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 714, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 461, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 60, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 1459 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 231, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 419, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 220, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 333, 2024-03-24T09:32:53.187 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 298, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 551, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 551, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T09:32:53.188 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T09:32:53.189 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 29490, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T09:32:53.190 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 29490, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 33, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 33, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 4, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T09:32:53.191 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 24134845, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 966, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.691378491, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001750909 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-24T09:32:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 1090, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 1646, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 556, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 195, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 260, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 65, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 1413, 2024-03-24T09:32:53.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 2003, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 590, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 812, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 1363, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 551, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 68276, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223612 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T09:32:53.194 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 6855, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 4187, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 13, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 886, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 356, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.546457329, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004343981 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.195 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 12, 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.004110259, 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000342521 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.196 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2302, 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.946789540, 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000411289 2024-03-24T09:32:53.197 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.198 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.199 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.200 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T09:32:53.201 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.201 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.201 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.201 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T09:32:53.201 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1, 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002827739, 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002827739 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.202 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000486049, 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000243024 2024-03-24T09:32:53.203 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10, 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.013427668, 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001342766 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.205 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1, 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.004381984, 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004381984 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.206 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T09:32:53.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T09:32:53.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.209 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.210 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 550, 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2.373121683, 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004314766 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.211 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 1, 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 2, 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 1, 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 1, 2024-03-24T09:32:53.212 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 2069, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 2069, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 649, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:53.213 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:53.214 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:53.215 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:53.216 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 292914, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 115, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T09:32:53.217 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9704, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 186, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:53.218 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 3974554, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 25154, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.219 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 15201, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 20133101, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 33, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 15201, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 15201, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 15.476703838, 2024-03-24T09:32:53.220 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001018137 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 15201, 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 15342 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 15201, 2024-03-24T09:32:53.221 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 12833, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 2368, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 116, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 1082, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 75, 2024-03-24T09:32:53.222 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 1117, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 12717, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-03-24T09:32:53.223 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:32:53.224 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T09:32:53.225 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 102, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T09:32:53.226 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 12, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 12, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 146, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 59 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T09:32:53.227 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T09:32:53.228 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 5, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 1101, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 551, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:53.229 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 40803, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 47806901, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 40803, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 40803, 2024-03-24T09:32:53.230 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 47806901, 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:53.231 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 15201, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 20194600, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 15201, 2024-03-24T09:32:53.232 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 20194600, 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T09:32:53.233 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 15201, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 15201, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 15201, 2024-03-24T09:32:53.234 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 15201, 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:53.235 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 551, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 61499, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 551, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 116, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 61499, 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:53.236 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x558d7b87c500": { 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:53.237 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 29490, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 19940541, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 29490, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 966, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 19940541, 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:53.238 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:53.239 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:53.239 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:53.239 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.239 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:53.239 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T09:32:53.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:53 smithi117 ceph-mon[35788]: pgmap v334: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:53.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:53 smithi116 ceph-mon[32207]: pgmap v334: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:53.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:53 smithi143 ceph-mon[35965]: pgmap v334: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:53.940 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-24T09:32:53.940 INFO:tasks.check_counter:Daemon mds.e mds.imported=45 2024-03-24T09:32:53.940 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=231 2024-03-24T09:32:53.940 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=419 2024-03-24T09:32:53.940 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-03-24T09:32:53.941 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:32:53.941 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:32:53.941 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-24T09:32:53.941 DEBUG:tasks.check_counter:Getting stats from h 2024-03-24T09:32:53.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-24T09:32:54.270 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:32:55.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:55 smithi117 ceph-mon[35788]: pgmap v335: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:55.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:55 smithi116 ceph-mon[32207]: pgmap v335: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:55.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:55 smithi143 ceph-mon[35965]: pgmap v335: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:56.179 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1285, 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1260, 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 14581550, 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 455590, 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-03-24T09:32:56.180 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.115905408, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.034360321, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.058947052, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.027486130, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1259, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.029822318, 2024-03-24T09:32:56.181 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000023687 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1674128, 2024-03-24T09:32:56.182 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 137744 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 169, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 332, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 25214458, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 501687, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 8, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 6, 2024-03-24T09:32:56.183 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.029206497, 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.007501313, 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.016213196, 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003479522, 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 330, 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.014913560, 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000045192 2024-03-24T09:32:56.184 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T09:32:56.185 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 450928 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1310, 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1312, 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 15872194, 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 407417, 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 5, 2024-03-24T09:32:56.186 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.076667144, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.017940155, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.026917522, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.026594538, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1310, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.026417686, 2024-03-24T09:32:56.187 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000020166 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:32:56.188 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:56.189 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4161, 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.083721013, 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000020120 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:56.190 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 0, 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 0, 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T09:32:56.191 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 0, 2024-03-24T09:32:56.192 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 1555, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 1555, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 1555, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-24T09:32:56.193 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 22, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 0, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 0, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:32:56.194 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T09:32:56.195 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 3452, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 26526152, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:32:56.196 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:32:56.197 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-24T09:32:56.198 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T09:32:56.199 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-24T09:32:56.200 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 0, 2024-03-24T09:32:56.201 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 17786, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T09:32:56.202 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 27, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 3, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 57478682, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 57478546, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.203 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 17786 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 77, 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 5639, 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 5562, 2024-03-24T09:32:56.204 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 46, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 271, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 225, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 1555, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 6297, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 4742, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 0, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-03-24T09:32:56.205 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 62776, 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 215420 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 0, 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 0, 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 0, 2024-03-24T09:32:56.206 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.207 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.208 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.209 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T09:32:56.210 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T09:32:56.211 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.212 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T09:32:56.213 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.214 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.215 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.216 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.217 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.218 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.219 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T09:32:56.220 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.221 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.222 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.223 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 6, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 4, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 4, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 0, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 0, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 2775, 2024-03-24T09:32:56.224 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:56.225 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:56.226 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:56.227 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:56.228 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 214451, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 67, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:56.229 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9704, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 186, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 1359819, 2024-03-24T09:32:56.230 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 7325, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T09:32:56.231 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 2495, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 26, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 4, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 2495, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2495, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.105978803, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000443278 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.232 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2495, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2496 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 2495, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 2495, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 0, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T09:32:56.233 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 1552, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 940, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T09:32:56.234 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 2, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:32:56.235 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-24T09:32:56.236 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T09:32:56.237 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 102, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-24T09:32:56.238 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 8, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 8, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 0, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 2, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 0 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T09:32:56.239 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 0, 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T09:32:56.240 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:56.241 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 2760, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 55403388, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 2760, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:56.242 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2760, 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 55403388, 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.243 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2495, 2024-03-24T09:32:56.244 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 53293110, 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2495, 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 53293110, 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:56.246 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2495, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2495, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2495, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2495, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.247 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:56.248 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x55cc0a7f06e0": { 2024-03-24T09:32:56.249 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:56.250 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:56.251 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T09:32:57.014 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:32:57.015 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-24T09:32:57.016 DEBUG:tasks.check_counter:Getting stats from k 2024-03-24T09:32:57.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-24T09:32:57.273 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T09:32:57.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:57 smithi117 ceph-mon[35788]: pgmap v336: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:57.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:57 smithi116 ceph-mon[32207]: pgmap v336: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:57.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:57 smithi143 ceph-mon[35965]: pgmap v336: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:32:59.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:32:59 smithi117 ceph-mon[35788]: pgmap v337: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:59.230 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-24T09:32:59.231 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:32:59.231 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 21834, 2024-03-24T09:32:59.231 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 16099, 2024-03-24T09:32:59.232 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 45637080, 2024-03-24T09:32:59.232 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 20451872, 2024-03-24T09:32:59.232 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 46, 2024-03-24T09:32:59.232 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 8, 2024-03-24T09:32:59.232 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 1.556668401, 2024-03-24T09:32:59.233 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.726975918, 2024-03-24T09:32:59.233 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.964901939, 2024-03-24T09:32:59.233 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.067103859, 2024-03-24T09:32:59.233 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:59.233 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 16098, 2024-03-24T09:32:59.234 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.481604608, 2024-03-24T09:32:59.234 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000029917 2024-03-24T09:32:59.234 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.234 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:59.235 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 32733, 2024-03-24T09:32:59.235 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.005668036, 2024-03-24T09:32:59.235 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000173 2024-03-24T09:32:59.235 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.235 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1666544, 2024-03-24T09:32:59.236 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 165184 2024-03-24T09:32:59.236 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.236 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:32:59.236 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 10277, 2024-03-24T09:32:59.236 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 13103, 2024-03-24T09:32:59.237 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 2880608, 2024-03-24T09:32:59.237 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 5695932, 2024-03-24T09:32:59.237 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 20, 2024-03-24T09:32:59.237 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-24T09:32:59.237 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.774504709, 2024-03-24T09:32:59.238 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.303656042, 2024-03-24T09:32:59.238 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.283868032, 2024-03-24T09:32:59.238 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.167834815, 2024-03-24T09:32:59.238 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:59.238 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 13101, 2024-03-24T09:32:59.239 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.673572199, 2024-03-24T09:32:59.239 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000051413 2024-03-24T09:32:59.239 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.239 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:59.239 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10218, 2024-03-24T09:32:59.240 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.001911321, 2024-03-24T09:32:59.240 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000187 2024-03-24T09:32:59.240 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.240 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 4880, 2024-03-24T09:32:59.240 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 458032 2024-03-24T09:32:59.241 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.241 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:32:59.241 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 8679, 2024-03-24T09:32:59.241 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 8627, 2024-03-24T09:32:59.241 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 2856353, 2024-03-24T09:32:59.241 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 19673017, 2024-03-24T09:32:59.242 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 9, 2024-03-24T09:32:59.242 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 7, 2024-03-24T09:32:59.242 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.613472359, 2024-03-24T09:32:59.242 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.202911831, 2024-03-24T09:32:59.242 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.165436625, 2024-03-24T09:32:59.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.204382892, 2024-03-24T09:32:59.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:32:59.243 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8625, 2024-03-24T09:32:59.243 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.683403266, 2024-03-24T09:32:59.243 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000079235 2024-03-24T09:32:59.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.244 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:32:59.244 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1884, 2024-03-24T09:32:59.244 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000344641, 2024-03-24T09:32:59.244 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000182 2024-03-24T09:32:59.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:32:59.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:32:59.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.245 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-24T09:32:59.245 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-24T09:32:59.246 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-24T09:32:59.246 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.246 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-24T09:32:59.246 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:59.247 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:59.247 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10604, 2024-03-24T09:32:59.247 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 5.384162082, 2024-03-24T09:32:59.247 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000507748 2024-03-24T09:32:59.247 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.248 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.248 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-24T09:32:59.248 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-24T09:32:59.248 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-24T09:32:59.248 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1149, 2024-03-24T09:32:59.248 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.090364197, 2024-03-24T09:32:59.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000078645 2024-03-24T09:32:59.249 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.249 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-24T09:32:59.249 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 3441, 2024-03-24T09:32:59.250 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 2620, 2024-03-24T09:32:59.250 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-24T09:32:59.250 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2620, 2024-03-24T09:32:59.250 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 5.944783793, 2024-03-24T09:32:59.251 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002269001 2024-03-24T09:32:59.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.251 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-24T09:32:59.251 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 821, 2024-03-24T09:32:59.251 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-24T09:32:59.252 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-24T09:32:59.252 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 11, 2024-03-24T09:32:59.252 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-24T09:32:59.252 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-24T09:32:59.252 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 1399, 2024-03-24T09:32:59.253 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 19, 2024-03-24T09:32:59.253 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 1380, 2024-03-24T09:32:59.253 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-24T09:32:59.253 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 1372, 2024-03-24T09:32:59.253 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 8, 2024-03-24T09:32:59.253 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 854, 2024-03-24T09:32:59.254 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 854, 2024-03-24T09:32:59.254 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 117, 2024-03-24T09:32:59.254 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 8170, 2024-03-24T09:32:59.254 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 3392, 2024-03-24T09:32:59.254 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 821, 2024-03-24T09:32:59.255 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 72, 2024-03-24T09:32:59.255 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:32:59.255 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-24T09:32:59.255 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-24T09:32:59.255 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-24T09:32:59.255 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-24T09:32:59.256 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-24T09:32:59.256 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-24T09:32:59.256 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 48, 2024-03-24T09:32:59.256 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 1059, 2024-03-24T09:32:59.256 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-24T09:32:59.256 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 13569, 2024-03-24T09:32:59.257 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-24T09:32:59.257 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 3452, 2024-03-24T09:32:59.257 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 26526152, 2024-03-24T09:32:59.257 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-24T09:32:59.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 13569, 2024-03-24T09:32:59.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-24T09:32:59.258 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:32:59.258 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:32:59.258 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-24T09:32:59.258 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:32:59.258 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T09:32:59.259 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 13569, 2024-03-24T09:32:59.259 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:32:59.259 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 11, 2024-03-24T09:32:59.259 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 908, 2024-03-24T09:32:59.259 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:32:59.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:32:59.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:32:59.260 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 617, 2024-03-24T09:32:59.260 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 507, 2024-03-24T09:32:59.260 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 51, 2024-03-24T09:32:59.260 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 1415 2024-03-24T09:32:59.261 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.261 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-24T09:32:59.261 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 335, 2024-03-24T09:32:59.261 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 492, 2024-03-24T09:32:59.261 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 258, 2024-03-24T09:32:59.262 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 447, 2024-03-24T09:32:59.262 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 412, 2024-03-24T09:32:59.262 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-24T09:32:59.262 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-24T09:32:59.262 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:32:59.262 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 506, 2024-03-24T09:32:59.263 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 506, 2024-03-24T09:32:59.263 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-24T09:32:59.263 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-24T09:32:59.263 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-24T09:32:59.264 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:32:59.264 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:32:59.264 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-24T09:32:59.264 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-24T09:32:59.264 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:32:59.265 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:32:59.265 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:32:59.265 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-24T09:32:59.265 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-24T09:32:59.265 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:32:59.265 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-24T09:32:59.266 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-24T09:32:59.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.266 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-24T09:32:59.266 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-24T09:32:59.267 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 41614, 2024-03-24T09:32:59.267 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-24T09:32:59.267 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-24T09:32:59.267 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 41614, 2024-03-24T09:32:59.267 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-24T09:32:59.267 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-24T09:32:59.268 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 46, 2024-03-24T09:32:59.268 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-24T09:32:59.268 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-24T09:32:59.268 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 46, 2024-03-24T09:32:59.268 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 5, 2024-03-24T09:32:59.268 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-24T09:32:59.269 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-24T09:32:59.269 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-24T09:32:59.269 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 31090662, 2024-03-24T09:32:59.269 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-24T09:32:59.269 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-24T09:32:59.269 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 992, 2024-03-24T09:32:59.270 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.932753793, 2024-03-24T09:32:59.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001948340 2024-03-24T09:32:59.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.270 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-24T09:32:59.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.270 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-24T09:32:59.271 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 1175, 2024-03-24T09:32:59.271 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 1686, 2024-03-24T09:32:59.271 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 511, 2024-03-24T09:32:59.271 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 175, 2024-03-24T09:32:59.271 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 306, 2024-03-24T09:32:59.271 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 131, 2024-03-24T09:32:59.272 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 1399, 2024-03-24T09:32:59.272 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 1925, 2024-03-24T09:32:59.272 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 526, 2024-03-24T09:32:59.272 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 854, 2024-03-24T09:32:59.272 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 1360, 2024-03-24T09:32:59.272 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 506, 2024-03-24T09:32:59.273 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 68208, 2024-03-24T09:32:59.273 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223612 2024-03-24T09:32:59.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.273 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-24T09:32:59.273 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 6180, 2024-03-24T09:32:59.273 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-24T09:32:59.274 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 3441, 2024-03-24T09:32:59.274 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 17, 2024-03-24T09:32:59.274 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 910, 2024-03-24T09:32:59.274 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-24T09:32:59.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 403, 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2.306124572, 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.005722393 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.276 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-24T09:32:59.276 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.276 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.276 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.277 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-24T09:32:59.277 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.277 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.277 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.277 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.277 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-24T09:32:59.278 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1706, 2024-03-24T09:32:59.278 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.938880559, 2024-03-24T09:32:59.278 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000550340 2024-03-24T09:32:59.278 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.278 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-24T09:32:59.279 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.279 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.279 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.279 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.279 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-24T09:32:59.279 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.280 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.280 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.280 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.280 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-24T09:32:59.280 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.280 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.281 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.281 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.281 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-24T09:32:59.281 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.281 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.281 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.282 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-24T09:32:59.282 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.282 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.282 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.283 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-24T09:32:59.283 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.284 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.284 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.284 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.284 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-24T09:32:59.284 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.284 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.285 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.286 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-24T09:32:59.286 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.286 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.286 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.286 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.287 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-24T09:32:59.287 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1, 2024-03-24T09:32:59.287 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000338375, 2024-03-24T09:32:59.287 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000338375 2024-03-24T09:32:59.287 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.287 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4, 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002551267, 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000637816 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.288 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.289 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.289 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.289 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-24T09:32:59.289 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.289 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.289 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.290 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-24T09:32:59.290 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.290 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.290 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.291 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-24T09:32:59.291 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.291 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.291 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.291 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.291 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-24T09:32:59.292 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.292 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.292 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.292 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-24T09:32:59.292 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.293 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.294 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.294 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-24T09:32:59.294 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.294 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.294 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.295 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.329 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-24T09:32:59.329 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.329 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.329 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.329 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-24T09:32:59.330 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-24T09:32:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.331 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 506, 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2.696889020, 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.005329820 2024-03-24T09:32:59.332 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.333 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.334 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.334 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-24T09:32:59.334 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-24T09:32:59.334 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 2, 2024-03-24T09:32:59.334 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 0, 2024-03-24T09:32:59.334 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 1572, 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 786, 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 627, 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.335 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:32:59.336 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:32:59.337 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:32:59.337 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:32:59.337 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:32:59.337 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:32:59.337 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:32:59.337 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:32:59.338 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:32:59.339 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:32:59.339 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:32:59.339 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:32:59.339 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:32:59.339 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:32:59.339 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-24T09:32:59.340 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:32:59.341 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:32:59.341 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:32:59.341 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:32:59.341 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 325597, 2024-03-24T09:32:59.341 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 140, 2024-03-24T09:32:59.341 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:32:59.342 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-24T09:32:59.342 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-24T09:32:59.342 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-24T09:32:59.342 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:32:59.342 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-24T09:32:59.342 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9704, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 186, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-24T09:32:59.343 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 4127309, 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 25107, 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-24T09:32:59.344 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.345 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-24T09:32:59.345 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-24T09:32:59.345 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-24T09:32:59.345 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 15954, 2024-03-24T09:32:59.345 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 27090256, 2024-03-24T09:32:59.345 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 33, 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 15954, 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 15954, 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 15.261708165, 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000956607 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.346 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 15954, 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 16093 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 15954, 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 13669, 2024-03-24T09:32:59.347 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 2285, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 100, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 1092, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 66, 2024-03-24T09:32:59.348 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 1030, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-24T09:32:59.349 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-24T09:32:59.350 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 13569, 2024-03-24T09:32:59.350 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-03-24T09:32:59.350 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:32:59.350 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-24T09:32:59.350 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:32:59.350 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-24T09:32:59.351 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-24T09:32:59.352 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-24T09:32:59.353 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-24T09:32:59.353 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-24T09:32:59.353 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-24T09:32:59.353 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-24T09:32:59.353 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-24T09:32:59.353 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 102, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 12, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 12, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-24T09:32:59.354 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 148, 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 58 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-24T09:32:59.355 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.356 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 5, 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 1012, 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 506, 2024-03-24T09:32:59.357 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 40786, 2024-03-24T09:32:59.358 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 47683660, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 40786, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 40786, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 47683660, 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:59.359 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-24T09:32:59.360 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 15954, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 27147066, 2024-03-24T09:32:59.361 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 15954, 2024-03-24T09:32:59.362 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 27147066, 2024-03-24T09:32:59.362 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:59.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.362 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-24T09:32:59.363 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 15954, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 15954, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 15954, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 15954, 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:59.364 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:59.365 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 506, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 56810, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 506, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 100, 2024-03-24T09:32:59.366 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 56810, 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x56235ecda500": { 2024-03-24T09:32:59.367 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 41614, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 26896358, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 41614, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-24T09:32:59.368 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 992, 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 26896358, 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-24T09:32:59.369 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-24T09:32:59.370 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:32:59 smithi116 ceph-mon[32207]: pgmap v337: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:32:59.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:32:59 smithi143 ceph-mon[35965]: pgmap v337: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-24T09:33:00.070 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-24T09:33:00.070 INFO:tasks.check_counter:Daemon mds.k mds.imported=48 2024-03-24T09:33:00.070 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=335 2024-03-24T09:33:00.070 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=492 2024-03-24T09:33:00.071 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=6 2024-03-24T09:33:00.071 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:33:00.071 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:33:00.071 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-24T09:33:00.071 DEBUG:tasks.check_counter:Getting stats from c 2024-03-24T09:33:00.071 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-24T09:33:00.327 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:33:01.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:01 smithi117 ceph-mon[35788]: pgmap v338: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:33:01.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:01 smithi116 ceph-mon[32207]: pgmap v338: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:33:01.485 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:01 smithi143 ceph-mon[35965]: pgmap v338: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T09:33:02.233 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T09:33:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:33:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 1781, 2024-03-24T09:33:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 1783, 2024-03-24T09:33:02.234 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 13096215, 2024-03-24T09:33:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 553898, 2024-03-24T09:33:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 6, 2024-03-24T09:33:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 2, 2024-03-24T09:33:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.114798436, 2024-03-24T09:33:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.033835773, 2024-03-24T09:33:02.235 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.031713999, 2024-03-24T09:33:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.040279819, 2024-03-24T09:33:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1781, 2024-03-24T09:33:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.036262289, 2024-03-24T09:33:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000020360 2024-03-24T09:33:02.236 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.237 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.238 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:33:02.238 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:33:02.238 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.238 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:33:02.238 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 799, 2024-03-24T09:33:02.239 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 774, 2024-03-24T09:33:02.239 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 2833340, 2024-03-24T09:33:02.239 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 305311, 2024-03-24T09:33:02.239 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 4, 2024-03-24T09:33:02.239 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 3, 2024-03-24T09:33:02.239 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.066295304, 2024-03-24T09:33:02.240 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.018330473, 2024-03-24T09:33:02.240 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.043720609, 2024-03-24T09:33:02.240 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.013220886, 2024-03-24T09:33:02.240 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:02.240 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 773, 2024-03-24T09:33:02.240 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.018936699, 2024-03-24T09:33:02.241 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000024497 2024-03-24T09:33:02.241 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.241 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:02.241 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.241 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.241 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.242 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.242 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1674512, 2024-03-24T09:33:02.242 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 138864 2024-03-24T09:33:02.242 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.242 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:33:02.242 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 7, 2024-03-24T09:33:02.243 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 172, 2024-03-24T09:33:02.243 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 3580, 2024-03-24T09:33:02.243 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 454823, 2024-03-24T09:33:02.243 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 6, 2024-03-24T09:33:02.243 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 4, 2024-03-24T09:33:02.243 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.009050786, 2024-03-24T09:33:02.244 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.005182041, 2024-03-24T09:33:02.244 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.002365851, 2024-03-24T09:33:02.244 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.000038260, 2024-03-24T09:33:02.244 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:02.244 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 170, 2024-03-24T09:33:02.244 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.011604061, 2024-03-24T09:33:02.245 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000068259 2024-03-24T09:33:02.245 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.245 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:02.245 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.245 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.245 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.246 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.246 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T09:33:02.246 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 454512 2024-03-24T09:33:02.246 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.246 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T09:33:02.246 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T09:33:02.247 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T09:33:02.247 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.247 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T09:33:02.247 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:02.247 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:02.247 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3927, 2024-03-24T09:33:02.248 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.081819239, 2024-03-24T09:33:02.248 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000020835 2024-03-24T09:33:02.248 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.248 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.248 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T09:33:02.248 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.249 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T09:33:02.250 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 0, 2024-03-24T09:33:02.250 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 0, 2024-03-24T09:33:02.250 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T09:33:02.250 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.250 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.250 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.251 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.251 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T09:33:02.251 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 0, 2024-03-24T09:33:02.251 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T09:33:02.251 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T09:33:02.251 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 0, 2024-03-24T09:33:02.252 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T09:33:02.252 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T09:33:02.252 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 1649, 2024-03-24T09:33:02.252 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 0, 2024-03-24T09:33:02.252 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 1649, 2024-03-24T09:33:02.252 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T09:33:02.253 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 1649, 2024-03-24T09:33:02.253 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 0, 2024-03-24T09:33:02.253 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 0, 2024-03-24T09:33:02.253 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 0, 2024-03-24T09:33:02.253 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 143, 2024-03-24T09:33:02.253 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 0, 2024-03-24T09:33:02.254 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 0, 2024-03-24T09:33:02.254 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 0, 2024-03-24T09:33:02.254 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 0, 2024-03-24T09:33:02.254 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:33:02.254 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T09:33:02.255 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T09:33:02.255 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T09:33:02.255 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T09:33:02.255 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T09:33:02.255 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T09:33:02.255 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 0, 2024-03-24T09:33:02.256 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 0, 2024-03-24T09:33:02.256 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T09:33:02.256 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 0, 2024-03-24T09:33:02.256 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T09:33:02.256 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 0, 2024-03-24T09:33:02.256 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 0, 2024-03-24T09:33:02.257 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:33:02.258 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T09:33:02.258 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 0, 2024-03-24T09:33:02.258 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:33:02.258 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T09:33:02.258 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 0, 2024-03-24T09:33:02.258 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:33:02.259 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:33:02.259 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:33:02.259 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 0, 2024-03-24T09:33:02.259 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 0, 2024-03-24T09:33:02.259 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 0, 2024-03-24T09:33:02.259 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 0 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 0, 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 0, 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 0, 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 0, 2024-03-24T09:33:02.260 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T09:33:02.261 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T09:33:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T09:33:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:33:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:33:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T09:33:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T09:33:02.262 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:33:02.263 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:33:02.263 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:33:02.263 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T09:33:02.263 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T09:33:02.263 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:33:02.263 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T09:33:02.264 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T09:33:02.264 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.264 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T09:33:02.264 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T09:33:02.264 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 0, 2024-03-24T09:33:02.264 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T09:33:02.265 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T09:33:02.265 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 20208, 2024-03-24T09:33:02.265 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T09:33:02.265 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T09:33:02.265 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 0, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 24, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 3, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T09:33:02.266 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T09:33:02.267 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 17796162, 2024-03-24T09:33:02.267 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 17795588, 2024-03-24T09:33:02.267 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T09:33:02.267 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.267 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.267 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 20208 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 1191, 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 1426, 2024-03-24T09:33:02.268 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 235, 2024-03-24T09:33:02.269 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 206, 2024-03-24T09:33:02.269 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 206, 2024-03-24T09:33:02.269 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 0, 2024-03-24T09:33:02.269 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 1649, 2024-03-24T09:33:02.269 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 1649, 2024-03-24T09:33:02.269 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 0, 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 0, 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 0, 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 0, 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 56424, 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 215420 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.270 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T09:33:02.271 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 0, 2024-03-24T09:33:02.271 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 0, 2024-03-24T09:33:02.271 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 0, 2024-03-24T09:33:02.271 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 0, 2024-03-24T09:33:02.271 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 0, 2024-03-24T09:33:02.271 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.272 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.273 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.274 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T09:33:02.274 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.274 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.274 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.274 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.274 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T09:33:02.275 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.275 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.275 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.275 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.275 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T09:33:02.275 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.276 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.277 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.278 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.279 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.280 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T09:33:02.280 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.281 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.281 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.281 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.281 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T09:33:02.281 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.281 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.282 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.282 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.282 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T09:33:02.282 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.282 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.282 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T09:33:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.285 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.286 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T09:33:02.286 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.286 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.286 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.286 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.286 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.287 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.288 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.289 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.290 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T09:33:02.291 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.292 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.293 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.294 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.295 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.296 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 2, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 1, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 1, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 0, 2024-03-24T09:33:02.297 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 0, 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 1722, 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T09:33:02.298 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:33:02.299 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:33:02.300 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:33:02.301 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T09:33:02.302 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 190247, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 54, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:33:02.303 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T09:33:02.304 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9704, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 186, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 3964216, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 14888, 2024-03-24T09:33:02.305 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T09:33:02.306 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 2317, 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 21, 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 3, 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 2317, 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2317, 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.067971745, 2024-03-24T09:33:02.307 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000460928 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2317, 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 2318 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 2317, 2024-03-24T09:33:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 2317, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 0, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 1470, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 0, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 845, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 0, 2024-03-24T09:33:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T09:33:02.310 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 1, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T09:33:02.311 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T09:33:02.312 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T09:33:02.313 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T09:33:02.314 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 102, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 13, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 5, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 5, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 0, 2024-03-24T09:33:02.315 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 2, 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 0 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 0, 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T09:33:02.316 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T09:33:02.317 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 0, 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 0, 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 0, 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:02.318 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 2583, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 15684193, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 2583, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:02.319 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2583, 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 15684193, 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.320 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:02.321 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2317, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 13601863, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2317, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 13601863, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.322 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:02.323 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2317, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 2317, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2317, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 2317, 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:02.324 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:02.325 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:02.326 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x55d8cf8706e0": { 2024-03-24T09:33:02.327 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:02.328 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.329 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:02.330 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T09:33:03.189 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-24T09:33:03.189 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-24T09:33:03.189 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-24T09:33:03.190 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-24T09:33:03.190 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=0 2024-03-24T09:33:03.190 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:33:03.190 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:33:03.190 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-24T09:33:03.190 DEBUG:tasks.check_counter:Getting stats from f 2024-03-24T09:33:03.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-24T09:33:03.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:03 smithi117 ceph-mon[35788]: pgmap v339: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:33:03.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:03 smithi116 ceph-mon[32207]: pgmap v339: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:33:03.453 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:33:03.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:03 smithi143 ceph-mon[35965]: pgmap v339: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:33:05.139 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:05 smithi143 ceph-mon[35965]: pgmap v340: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:33:05.140 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T09:33:05.140 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:33:05.140 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 28849, 2024-03-24T09:33:05.140 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 22840, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 46555068, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 12610018, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 16, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 4, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 1.599819555, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.596685692, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.832098911, 2024-03-24T09:33:05.141 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.272246138, 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 22842, 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.112481052, 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000048703 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:05.142 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 33533, 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.004460747, 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000133 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 5481, 2024-03-24T09:33:05.143 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 5427, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 3033551, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 8826606, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 45, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 7, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.626035815, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.275920813, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.150889232, 2024-03-24T09:33:05.144 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.176753673, 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 5426, 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.446006814, 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000082198 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 534, 2024-03-24T09:33:05.145 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000078723, 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000147 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1684912, 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 168864 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 6404, 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 9162, 2024-03-24T09:33:05.146 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 1958162, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 8526751, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 16, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 7, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.548278730, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.267027691, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.205659690, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.076939116, 2024-03-24T09:33:05.147 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 9156, 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.326567382, 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000035667 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 9506, 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.002054449, 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000216 2024-03-24T09:33:05.148 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 6384, 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 459760 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.149 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 10545, 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 4.375074068, 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000414895 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T09:33:05.150 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1113, 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.077788710, 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000069891 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 2886, 2024-03-24T09:33:05.151 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 2048, 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2048, 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 3.400528847, 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001660414 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 838, 2024-03-24T09:33:05.152 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 11, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 1604, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 32, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 1572, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T09:33:05.153 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 1557, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 9, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 910, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 910, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 128, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 6981, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 3305, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 838, 2024-03-24T09:33:05.154 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 71, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 44, 2024-03-24T09:33:05.155 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 1347, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 14128, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 3452, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 26526152, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 6, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 14128, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T09:33:05.156 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 312, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 312, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 13816, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 14, 2024-03-24T09:33:05.157 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 819, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 450, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 373, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 43, 2024-03-24T09:33:05.158 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 1323 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 117, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 706, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 363, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 686, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 159, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T09:33:05.159 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 501, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 501, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:33:05.160 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:33:05.161 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 21450, 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T09:33:05.162 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 21450, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 25, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 25, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 3, 2024-03-24T09:33:05.163 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 19258965, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 0, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 899, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.702895617, 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001894210 2024-03-24T09:33:05.164 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 0 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 1222, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 1728, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 506, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 201, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 435, 2024-03-24T09:33:05.165 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 234, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 1604, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 2128, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 524, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 910, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 1415, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 505, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 66888, 2024-03-24T09:33:05.166 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 223612 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 5377, 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 3, 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 2886, 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 21, 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 822, 2024-03-24T09:33:05.167 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 313, 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.071952177, 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.003424767 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 6, 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.001661345, 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000276890 2024-03-24T09:33:05.168 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.169 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1218, 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.516036823, 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000423675 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.170 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T09:33:05.171 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.172 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T09:33:05.173 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.173 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.173 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.173 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.173 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T09:33:05.173 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1, 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.002096474, 2024-03-24T09:33:05.174 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.002096474 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.175 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 4, 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.002078760, 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000519690 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T09:33:05.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 4, 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.004236367, 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001059091 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3, 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.009949272, 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.003316424 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.177 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.179 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T09:33:05.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T09:33:05.181 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.182 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 499, 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.792517629, 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.003592219 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:33:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 4, 2024-03-24T09:33:05.185 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 3, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 1, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 1070, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 1070, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 657, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T09:33:05.186 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:33:05.187 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:33:05.188 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:33:05.189 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:33:05.190 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 333038, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 133, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T09:33:05.191 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9704, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 186, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 4444067, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 27652, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T09:33:05.192 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 16352, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 15255216, 2024-03-24T09:33:05.193 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 33, 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 16352, 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 16352, 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 15.328596943, 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000937414 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T09:33:05.194 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 16352, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 16477 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 16352, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 14208, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 2144, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T09:33:05.195 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 0, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 10, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 80, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 979, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 58, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T09:33:05.196 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 1017, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 14128, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 20, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T09:33:05.197 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T09:33:05.198 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T09:33:05.199 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 102, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 13, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 12, 2024-03-24T09:33:05.200 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 12, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 132, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 0, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 53 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 1, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:33:05.201 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T09:33:05.202 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 5, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 1002, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 501, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:05.203 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 40730, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 47860697, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 40730, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 40730, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 47860697, 2024-03-24T09:33:05.204 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:05.205 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 16352, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 15311105, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 16352, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 15311105, 2024-03-24T09:33:05.206 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T09:33:05.207 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 16352, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 16352, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 16352, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 16352, 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:05.208 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 501, 2024-03-24T09:33:05.209 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 55889, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 501, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 80, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 55889, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x55d9bde28500": { 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 21450, 2024-03-24T09:33:05.211 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 15064661, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 21450, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 899, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 15064661, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:05.212 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:05.213 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.213 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:05.213 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T09:33:05.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:05 smithi117 ceph-mon[35788]: pgmap v340: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:33:05.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:05 smithi116 ceph-mon[32207]: pgmap v340: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds.imported=44 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=117 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=706 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:33:06.055 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-24T09:33:06.056 DEBUG:tasks.check_counter:Getting stats from i 2024-03-24T09:33:06.056 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-24T09:33:06.314 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:33:07.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:07 smithi117 ceph-mon[35788]: pgmap v341: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:33:07.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:07 smithi116 ceph-mon[32207]: pgmap v341: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:33:07.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:07 smithi143 ceph-mon[35965]: pgmap v341: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-24T09:33:08.148 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 25025, 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 18600, 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 44769790, 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 20090822, 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 16, 2024-03-24T09:33:08.149 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 4, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 1.380959950, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.541152838, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.821719307, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.162711813, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 18600, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.720126688, 2024-03-24T09:33:08.150 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000038716 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 32311, 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.004621085, 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000143 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:33:08.151 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 6895, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 7948, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 3731972, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 3950568, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 46, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 8, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.505738670, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.219499897, 2024-03-24T09:33:08.152 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.243580869, 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.069089229, 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 7942, 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.306067092, 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000038537 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 9244, 2024-03-24T09:33:08.153 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.001390017, 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000150 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1693536, 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 161472 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:33:08.154 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 9134, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 9526, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 2611412, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 5629810, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 13, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 5, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.823590856, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.342947327, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.184060825, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.246127615, 2024-03-24T09:33:08.155 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 9527, 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.747650862, 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000078477 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2702, 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000484925, 2024-03-24T09:33:08.156 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000179 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 457024 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.157 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 9977, 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 4.091018592, 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000410044 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.158 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 601, 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.043977120, 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000073173 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T09:33:08.159 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 2358, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 1654, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1654, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 4.720098913, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.002853747 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 704, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T09:33:08.160 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 11, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 1445, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 20, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 1425, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 1417, 2024-03-24T09:33:08.161 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 10, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 921, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 921, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 150, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 5546, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 1653, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 703, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 115, 2024-03-24T09:33:08.162 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 42, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 1172, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T09:33:08.163 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 14165, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 3452, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 26526152, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 6, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 14165, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:33:08.164 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 422, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 422, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 13743, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 23, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 484, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:33:08.165 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 1270, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 285, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 30, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 722 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 93, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 889, 2024-03-24T09:33:08.166 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 454, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 1032, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 117, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 235, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 235, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T09:33:08.167 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T09:33:08.168 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 20208, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T09:33:08.169 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 20208, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 24, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 24, 2024-03-24T09:33:08.170 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 3, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 17796162, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 0, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 559, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.019174146, 2024-03-24T09:33:08.171 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001823209 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 0 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 1204, 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 1444, 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 240, 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 223, 2024-03-24T09:33:08.172 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 603, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 380, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 1445, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 1712, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 267, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 921, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 1157, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 236, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 66752, 2024-03-24T09:33:08.173 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 223612 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 3912, 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 7, 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 2358, 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 13, 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 516, 2024-03-24T09:33:08.174 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 248, 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.081719140, 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.004361770 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 10, 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.003550000, 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000355000 2024-03-24T09:33:08.175 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.176 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1142, 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.372842380, 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000326481 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T09:33:08.177 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.178 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.179 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.180 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T09:33:08.181 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.181 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.181 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.181 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.181 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.182 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T09:33:08.183 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3, 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.001240525, 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000413508 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 13, 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.019201118, 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001477009 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.184 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2, 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 2.073279746, 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 1.036639873 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.185 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.186 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T09:33:08.187 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.188 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.189 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.190 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1, 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.001650734, 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.001650734 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 235, 2024-03-24T09:33:08.191 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.166615270, 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.004964320 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.192 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 2, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 1, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 1, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 1262, 2024-03-24T09:33:08.193 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 1262, 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 661, 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:33:08.194 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:33:08.195 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:33:08.196 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:33:08.197 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 300596, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 119, 2024-03-24T09:33:08.198 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9704, 2024-03-24T09:33:08.199 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 186, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 4314566, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 26263, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T09:33:08.200 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 15446, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 13758667, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 33, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 15446, 2024-03-24T09:33:08.201 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 15446, 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 12.528672398, 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000811127 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 15446, 2024-03-24T09:33:08.202 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 15557 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 15446, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 14220, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 1226, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 0, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 10, 2024-03-24T09:33:08.203 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 55, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 614, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 46, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 485, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T09:33:08.204 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 14165, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 20, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T09:33:08.205 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T09:33:08.206 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T09:33:08.207 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 103, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 14, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 12, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 12, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T09:33:08.208 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 116, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 0, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 46 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 1, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T09:33:08.209 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T09:33:08.210 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 5, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 470, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 235, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:08.211 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 41050, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 47398822, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 41050, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 41050, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 47398822, 2024-03-24T09:33:08.212 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:08.213 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 15446, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 13785674, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 15446, 2024-03-24T09:33:08.214 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 13785674, 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T09:33:08.215 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 15446, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 15446, 2024-03-24T09:33:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 15446, 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 15446, 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:33:08.217 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 235, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 27007, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 235, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:08.218 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 55, 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 27007, 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x55f08e698500": { 2024-03-24T09:33:08.219 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 20208, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 13601858, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 20208, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:08.220 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 559, 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 13601858, 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:08.221 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds.imported=42 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=93 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=889 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:33:08.945 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-24T09:33:08.946 DEBUG:tasks.check_counter:Getting stats from l 2024-03-24T09:33:08.946 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de shell --fsid 69c76602-e9be-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-24T09:33:09.238 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T09:33:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:09 smithi143 ceph-mon[35965]: pgmap v342: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:33:09.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:09 smithi143 ceph-mon[35965]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T09:33:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:09 smithi117 ceph-mon[35788]: pgmap v342: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:33:09.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:09 smithi117 ceph-mon[35788]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T09:33:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:09 smithi116 ceph-mon[32207]: pgmap v342: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-24T09:33:09.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:09 smithi116 ceph-mon[32207]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T09:33:11.059 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-03-24T09:33:11.059 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T09:33:11.059 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 761, 2024-03-24T09:33:11.059 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 763, 2024-03-24T09:33:11.059 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 4386160, 2024-03-24T09:33:11.059 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 236735, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 7, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 3, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.050314450, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.013166193, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.014903068, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.017429643, 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:11.060 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 761, 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.017622366, 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000023156 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.061 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 996, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 978, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 8513069, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 370866, 2024-03-24T09:33:11.062 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 4, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 3, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.085458386, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.024639271, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.051031343, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.018125098, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 977, 2024-03-24T09:33:11.063 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.022027487, 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000022546 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.064 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 1648288, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 143520 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_messages": 848, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages": 1018, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_bytes": 4475544, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_bytes": 717254, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_created_connections": 7, 2024-03-24T09:33:11.065 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_active_connections": 5, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_total_time": 0.067291937, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_send_time": 0.022072430, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_recv_time": 0.018704118, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_running_fast_dispatch_time": 0.019704935, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 1016, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.042219963, 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000041555 2024-03-24T09:33:11.066 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_handle_ack_lat": { 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "msgr_send_encrypted_bytes": 455392 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.067 INFO:teuthology.orchestra.run.smithi143.stdout: "cct": { 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "total_workers": 1, 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "unhealthy_workers": 0 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-MDSRank": { 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 3928, 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.081523186, 2024-03-24T09:33:11.068 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000020754 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: "finisher-PurgeQueue": { 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: "queue_len": 0, 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: "complete_latency": { 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.069 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: "mds": { 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: "request": 0, 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: "reply": 0, 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: "reply_latency": { 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.070 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_reply": 0, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "forward": 0, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_complete": 0, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_fetch_keys": 0, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_commit": 0, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_split": 0, 2024-03-24T09:33:11.071 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_merge": 0, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes": 2057, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_top": 1, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_bottom": 2056, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pin_tail": 0, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_pinned": 2056, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_expired": 0, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "inodes_with_caps": 0, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "caps": 0, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "subtrees": 46, 2024-03-24T09:33:11.072 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_hit": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_forward": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_discover": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_dir_fetch": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_remote_ino": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "traverse_lock": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "load_cent": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "q": 0, 2024-03-24T09:33:11.073 INFO:teuthology.orchestra.run.smithi143.stdout: "exported": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "exported_inodes": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "imported": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "imported_inodes": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_dir_fetch": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_backtrace_fetch": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "openino_peer_discover": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rfiles": 0, 2024-03-24T09:33:11.074 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rbytes": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "root_rsnaps": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_set_tag": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_inotable_repaired": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_inodes": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "scrub_file_inodes": 0, 2024-03-24T09:33:11.075 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_inode_file_caps": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_grant": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_caps_dirty": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_cap_release": 0, 2024-03-24T09:33:11.076 INFO:teuthology.orchestra.run.smithi143.stdout: "process_request_cap_release": 0 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_cache": { 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update": 0, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_update_receipt": 0, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_try_discover": 0, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_send_discover": 0, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "dir_handle_discover": 0, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays": 0, 2024-03-24T09:33:11.077 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_delayed": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "num_strays_enqueuing": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_created": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_enqueued": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_reintegrated": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "strays_migrated": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_processing": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_enqueued": 0, 2024-03-24T09:33:11.078 INFO:teuthology.orchestra.run.smithi143.stdout: "num_recovering_prioritized": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_started": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "recovery_completed": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_path": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_quiesce_inode": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_exportdir": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_flush": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragmentdir": 0, 2024-03-24T09:33:11.079 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_fragstats": 0, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "ireq_inodestats": 0 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_log": { 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "evlrg": 0, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "evadd": 0, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "evex": 0, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "evtrm": 0, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "ev": 21450, 2024-03-24T09:33:11.080 INFO:teuthology.orchestra.run.smithi143.stdout: "evexg": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "evexd": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "segadd": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "segex": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "segtrm": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "seg": 25, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "segmjr": 3, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "segexg": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "segexd": 0, 2024-03-24T09:33:11.081 INFO:teuthology.orchestra.run.smithi143.stdout: "expos": 4194304, 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "wrpos": 19258965, 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "rdpos": 19258391, 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "jlat": { 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: "replayed": 21450 2024-03-24T09:33:11.082 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_mem": { 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "ino": 1207, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "ino+": 1708, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "ino-": 501, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "dir": 83, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "dir+": 83, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "dir-": 0, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "dn": 2057, 2024-03-24T09:33:11.083 INFO:teuthology.orchestra.run.smithi143.stdout: "dn+": 2057, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "dn-": 0, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "cap": 0, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "cap+": 0, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "cap-": 0, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "rss": 55700, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "heap": 215420 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_server": { 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_client_request": 0, 2024-03-24T09:33:11.084 INFO:teuthology.orchestra.run.smithi143.stdout: "dispatch_server_request": 0, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_request": 0, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_client_session": 0, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "handle_peer_request": 0, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "req_create_latency": { 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.085 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getattr_latency": { 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getfilelock_latency": { 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.086 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "req_link_latency": { 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookup_latency": { 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.087 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookuphash_latency": { 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupino_latency": { 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.088 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupname_latency": { 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupparent_latency": { 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.089 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lookupsnap_latency": { 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "req_lssnap_latency": { 2024-03-24T09:33:11.090 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.091 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.091 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.091 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.091 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mkdir_latency": { 2024-03-24T09:33:11.091 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mknod_latency": { 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.092 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "req_mksnap_latency": { 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "req_open_latency": { 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.093 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "req_readdir_latency": { 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rename_latency": { 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.094 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "req_renamesnap_latency": { 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "req_snapdiff_latency": { 2024-03-24T09:33:11.095 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmdir_latency": { 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmsnap_latency": { 2024-03-24T09:33:11.096 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "req_rmxattr_latency": { 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.097 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setattr_latency": { 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setdirlayout_latency": { 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.098 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setfilelock_latency": { 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setlayout_latency": { 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.099 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "req_setxattr_latency": { 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "req_symlink_latency": { 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.100 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "req_unlink_latency": { 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_revoke_eviction": 0, 2024-03-24T09:33:11.101 INFO:teuthology.orchestra.run.smithi143.stdout: "cap_acquisition_throttle": 0, 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "req_getvxattr_latency": { 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_sessions": { 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "session_count": 1, 2024-03-24T09:33:11.102 INFO:teuthology.orchestra.run.smithi143.stdout: "session_add": 4, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "session_remove": 3, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_open": 3, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "sessions_stale": 0, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "total_load": 0, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "average_load": 0, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "avg_session_uptime": 3875, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "mdthresh_evicted": 0 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.103 INFO:teuthology.orchestra.run.smithi143.stdout: "mempool": { 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_bytes": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bloom_filter_items": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_alloc_items": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_data_items": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T09:33:11.104 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_other_items": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_bytes": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_extent_items": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_bytes": 0, 2024-03-24T09:33:11.105 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_blob_items": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_fsck_items": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_bytes": 0, 2024-03-24T09:33:11.106 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_txc_items": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_bytes": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluestore_writing_items": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_bytes": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_items": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_reader_items": 0, 2024-03-24T09:33:11.107 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "bluefs_file_writer_items": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_bytes": 206337, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_anon_items": 61, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_bytes": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "buffer_meta_items": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_bytes": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_items": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_bytes": 0, 2024-03-24T09:33:11.108 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_mapbl_items": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_bytes": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_pglog_items": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_bytes": 9944, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_items": 192, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "osdmap_mapping_items": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_bytes": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "pgmap_items": 0, 2024-03-24T09:33:11.109 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_bytes": 4116790, 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "mds_co_items": 15269, 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_bytes": 0, 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_1_items": 0, 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_bytes": 0, 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "unittest_2_items": 0 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "objecter": { 2024-03-24T09:33:11.110 INFO:teuthology.orchestra.run.smithi143.stdout: "op_active": 0, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "op_laggy": 0, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send": 2334, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "op_send_bytes": 21, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "op_resend": 3, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "op_reply": 2334, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "op_latency": { 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2334, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 1.207967928, 2024-03-24T09:33:11.111 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000517552 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "op_inflight": 0, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "oplen_avg": { 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 2334, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 2335 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "op": 2334, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "op_r": 2334, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "op_w": 0, 2024-03-24T09:33:11.112 INFO:teuthology.orchestra.run.smithi143.stdout: "op_rmw": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "op_pg": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_stat": 1477, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_create": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_read": 855, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_write": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writefull": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_writesame": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_append": 0, 2024-03-24T09:33:11.113 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_zero": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_truncate": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_delete": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_mapext": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_sparse_read": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_clonerange": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_getxattr": 1, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_setxattr": 0, 2024-03-24T09:33:11.114 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_cmpxattr": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_rmxattr": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_resetxattrs": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_call": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_watch": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_notify": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_pgls_filter": 0, 2024-03-24T09:33:11.115 INFO:teuthology.orchestra.run.smithi143.stdout: "osdop_other": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_active": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_send": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_resend": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "linger_ping": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_active": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_send": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "poolop_resend": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_active": 0, 2024-03-24T09:33:11.116 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_send": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "poolstat_resend": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_active": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_send": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "statfs_resend": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "command_active": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "command_send": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "command_resend": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "map_epoch": 103, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "map_full": 0, 2024-03-24T09:33:11.117 INFO:teuthology.orchestra.run.smithi143.stdout: "map_inc": 14, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_sessions": 7, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_open": 7, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_session_close": 0, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "osd_laggy": 0, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_wr": 0, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_rd": 2, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_del": 0 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.118 INFO:teuthology.orchestra.run.smithi143.stdout: "oft": { 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_objs": 0, 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_kv_pairs": 0, 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_updates": 0, 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: "omap_total_removes": 0 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: "osd-slow-ops": { 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: "slow_ops_count": 0 2024-03-24T09:33:11.119 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "purge_queue": { 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops": 0, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing": 0, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executing_high_water": 0, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed_ops": 0, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_executed": 0, 2024-03-24T09:33:11.120 INFO:teuthology.orchestra.run.smithi143.stdout: "pq_item_in_journal": 0 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 2601, 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 17124114, 2024-03-24T09:33:11.121 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 2601, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2601, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 17124114, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.122 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_bytes": { 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 104857600, 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:11.123 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2334, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 15065045, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2334, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 15065045, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.124 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-objecter_ops": { 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 1024, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:11.125 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 2334, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 2334, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 2334, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 2334, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.126 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle": { 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:11.127 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.128 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: }, 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "throttle-write_buf_throttle-0x558512c086e0": { 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "val": 0, 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "max": 3758096384, 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "get_started": 0, 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "get": 0, 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "get_sum": 0, 2024-03-24T09:33:11.129 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_fail": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "get_or_fail_success": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "take": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "take_sum": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "put": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "put_sum": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "wait": { 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "avgcount": 0, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "sum": 0.000000000, 2024-03-24T09:33:11.130 INFO:teuthology.orchestra.run.smithi143.stdout: "avgtime": 0.000000000 2024-03-24T09:33:11.131 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.131 INFO:teuthology.orchestra.run.smithi143.stdout: } 2024-03-24T09:33:11.131 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-03-24T09:33:11.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:11 smithi117 ceph-mon[35788]: pgmap v344: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:33:11.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:11 smithi116 ceph-mon[32207]: pgmap v344: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:33:11.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:11 smithi143 ceph-mon[35965]: pgmap v344: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-24T09:33:11.832 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-24T09:33:11.833 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-24T09:33:11.834 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-24T09:33:11.844 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-24T09:33:11.844 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:33:11.844 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:33:11.866 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:33:11.866 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:33:11.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:33:12.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 -- 172.21.15.116:0/1482693427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240105070 msgr2=0x7f724010cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 --2- 172.21.15.116:0/1482693427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240105070 0x7f724010cc40 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f7238009ca0 tx=0x7f723802c0f0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 -- 172.21.15.116:0/1482693427 shutdown_connections 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 --2- 172.21.15.116:0/1482693427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240105070 0x7f724010cc40 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 --2- 172.21.15.116:0/1482693427 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72401009a0 0x7f7240104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 --2- 172.21.15.116:0/1482693427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72400ffff0 0x7f72401003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 -- 172.21.15.116:0/1482693427 >> 172.21.15.116:0/1482693427 conn(0x7f7240076ba0 msgr2=0x7f72400fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 -- 172.21.15.116:0/1482693427 shutdown_connections 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.079+0000 7f7246725640 1 -- 172.21.15.116:0/1482693427 wait complete. 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 Processor -- start 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 -- start start 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400ffff0 0x7f724019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7240105070 0x7f724019ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 0x7f7240196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7240197440 con 0x7f7240196a50 2024-03-24T09:33:12.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f72401975b0 con 0x7f7240105070 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7246725640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f7240197770 con 0x7f72400ffff0 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7244cc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 0x7f7240196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f723f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7240105070 0x7f724019ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f723ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400ffff0 0x7f724019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7244cc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 0x7f7240196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46772/0 (socket says 172.21.15.116:46772) 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.080+0000 7f7244cc2640 1 -- 172.21.15.116:0/720600428 learned_addr learned my addr 172.21.15.116:0/720600428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7244cc2640 1 -- 172.21.15.116:0/720600428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400ffff0 msgr2=0x7f724019c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7244cc2640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400ffff0 0x7f724019c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7244cc2640 1 -- 172.21.15.116:0/720600428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7240105070 msgr2=0x7f724019ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7244cc2640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7240105070 0x7f724019ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7244cc2640 1 -- 172.21.15.116:0/720600428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7238009900 con 0x7f7240196a50 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f723ffff640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400ffff0 0x7f724019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7244cc2640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 0x7f7240196f00 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f723802c700 tx=0x7f7238035d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72380329e0 con 0x7f7240196a50 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72401a3670 con 0x7f7240196a50 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f7238003e20 con 0x7f7240196a50 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238004750 con 0x7f7240196a50 2024-03-24T09:33:12.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.081+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72401a3af0 con 0x7f7240196a50 2024-03-24T09:33:12.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.082+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7240068950 con 0x7f7240196a50 2024-03-24T09:33:12.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.083+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f723803b040 con 0x7f7240196a50 2024-03-24T09:33:12.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.084+0000 7f723d7fa640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72280a2230 0x7f72280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.084+0000 7f723ffff640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72280a2230 0x7f72280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.084+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f72380f0d70 con 0x7f7240196a50 2024-03-24T09:33:12.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.084+0000 7f723ffff640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72280a2230 0x7f72280a46f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7240104b20 tx=0x7f7230005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.088+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f72380b96b0 con 0x7f7240196a50 2024-03-24T09:33:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.264+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f72401009a0 con 0x7f7240196a50 2024-03-24T09:33:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.266+0000 7f723d7fa640 1 -- 172.21.15.116:0/720600428 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) ==== 74+0+25614 (secure 0 0 0) 0x7f72380bcd70 con 0x7f7240196a50 2024-03-24T09:33:12.267 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:33:12.267 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":103,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:33:08.046517+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:33:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72280a2230 msgr2=0x7f72280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72280a2230 0x7f72280a46f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7240104b20 tx=0x7f7230005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 msgr2=0x7f7240196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 0x7f7240196f00 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f723802c700 tx=0x7f7238035d20 comp rx=0 tx=0).stop 2024-03-24T09:33:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 shutdown_connections 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7240196a50 0x7f7240196f00 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7240105070 0x7f724019ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72280a2230 0x7f72280a46f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 --2- 172.21.15.116:0/720600428 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f72400ffff0 0x7f724019c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 >> 172.21.15.116:0/720600428 conn(0x7f7240076ba0 msgr2=0x7f7240103200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 shutdown_connections 2024-03-24T09:33:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.268+0000 7f7246725640 1 -- 172.21.15.116:0/720600428 wait complete. 2024-03-24T09:33:12.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:33:12.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2498860976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc069d80 msgr2=0x7f79dc06a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2498860976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc069d80 0x7f79dc06a1e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f79d0009860 tx=0x7f79d002bb70 comp rx=0 tx=0).stop 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2498860976 shutdown_connections 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2498860976 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79dc06a720 0x7f79dc10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2498860976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc069d80 0x7f79dc06a1e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2498860976 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc107860 0x7f79dc107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2498860976 >> 172.21.15.116:0/2498860976 conn(0x7f79dc0773b0 msgr2=0x7f79dc0777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2498860976 shutdown_connections 2024-03-24T09:33:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.384+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2498860976 wait complete. 2024-03-24T09:33:12.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 Processor -- start 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 -- start start 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 0x7f79dc19d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79dc06a720 0x7f79dc19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc107860 0x7f79dc197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc197fd0 con 0x7f79dc069d80 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc198140 con 0x7f79dc107860 2024-03-24T09:33:12.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79e1ec9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc198300 con 0x7f79dc06a720 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 0x7f79dc19d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 0x7f79dc19d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46796/0 (socket says 172.21.15.116:46796) 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79db7fe640 1 -- 172.21.15.116:0/2710758272 learned_addr learned my addr 172.21.15.116:0/2710758272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79daffd640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79dc06a720 0x7f79dc19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.385+0000 7f79dbfff640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc107860 0x7f79dc197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79db7fe640 1 -- 172.21.15.116:0/2710758272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79dc06a720 msgr2=0x7f79dc19dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79db7fe640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79dc06a720 0x7f79dc19dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79db7fe640 1 -- 172.21.15.116:0/2710758272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc107860 msgr2=0x7f79dc197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79db7fe640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc107860 0x7f79dc197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79db7fe640 1 -- 172.21.15.116:0/2710758272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79cc009560 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79db7fe640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 0x7f79dc19d7d0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00e9c0 tx=0x7f79cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc00cde0 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d0009490 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f79cc0044d0 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc010430 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.386+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc1a4690 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.387+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc102dc0 con 0x7f79dc069d80 2024-03-24T09:33:12.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.388+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f79cc004000 con 0x7f79dc069d80 2024-03-24T09:33:12.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.388+0000 7f79d8ff9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79c40a2230 0x7f79c40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.388+0000 7f79daffd640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79c40a2230 0x7f79c40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.388+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f79cc014070 con 0x7f79dc069d80 2024-03-24T09:33:12.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.389+0000 7f79daffd640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79c40a2230 0x7f79c40a46f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f79d002c180 tx=0x7f79d0032360 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.393+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f79cc097100 con 0x7f79dc069d80 2024-03-24T09:33:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.565+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79dc068950 con 0x7f79dc069d80 2024-03-24T09:33:12.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.566+0000 7f79d8ff9640 1 -- 172.21.15.116:0/2710758272 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) ==== 93+0+10 (secure 0 0 0) 0x7f79dc068950 con 0x7f79dc069d80 2024-03-24T09:33:12.566 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:33:12.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79c40a2230 msgr2=0x7f79c40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79c40a2230 0x7f79c40a46f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f79d002c180 tx=0x7f79d0032360 comp rx=0 tx=0).stop 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 msgr2=0x7f79dc19d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 0x7f79dc19d7d0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f79cc00e9c0 tx=0x7f79cc00ee80 comp rx=0 tx=0).stop 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 shutdown_connections 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79dc107860 0x7f79dc197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f79c40a2230 0x7f79c40a46f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f79dc06a720 0x7f79dc19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 --2- 172.21.15.116:0/2710758272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79dc069d80 0x7f79dc19d7d0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 >> 172.21.15.116:0/2710758272 conn(0x7f79dc0773b0 msgr2=0x7f79dc0ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 shutdown_connections 2024-03-24T09:33:12.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.568+0000 7f79e1ec9640 1 -- 172.21.15.116:0/2710758272 wait complete. 2024-03-24T09:33:12.579 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 -- 172.21.15.116:0/2851591729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 msgr2=0x7fec80100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 --2- 172.21.15.116:0/2851591729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 0x7fec80100aa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fec70009890 tx=0x7fec7002bb70 comp rx=0 tx=0).stop 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 -- 172.21.15.116:0/2851591729 shutdown_connections 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 --2- 172.21.15.116:0/2851591729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80100fe0 0x7fec8010d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 --2- 172.21.15.116:0/2851591729 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 0x7fec80100aa0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 --2- 172.21.15.116:0/2851591729 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80106690 0x7fec80106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 -- 172.21.15.116:0/2851591729 >> 172.21.15.116:0/2851591729 conn(0x7fec80077770 msgr2=0x7fec80077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 -- 172.21.15.116:0/2851591729 shutdown_connections 2024-03-24T09:33:12.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 -- 172.21.15.116:0/2851591729 wait complete. 2024-03-24T09:33:12.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.682+0000 7fec862af640 1 Processor -- start 2024-03-24T09:33:12.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 -- start start 2024-03-24T09:33:12.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 0x7fec80198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 0x7fec80198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80106690 0x7fec801a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec801997c0 con 0x7fec80100fe0 2024-03-24T09:33:12.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fec801a3fe0 con 0x7fec80106690 2024-03-24T09:33:12.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec862af640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fec801a4120 con 0x7fec80100620 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 0x7fec80198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7ffff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 0x7fec80198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 0x7fec80198f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46818/0 (socket says 172.21.15.116:46818) 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 -- 172.21.15.116:0/4016867361 learned_addr learned my addr 172.21.15.116:0/4016867361 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec8484c640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80106690 0x7fec801a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 -- 172.21.15.116:0/4016867361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 msgr2=0x7fec80198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 0x7fec80198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 -- 172.21.15.116:0/4016867361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80106690 msgr2=0x7fec801a3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80106690 0x7fec801a3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 -- 172.21.15.116:0/4016867361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec78009900 con 0x7fec80100fe0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.683+0000 7fec7f7fe640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 0x7fec80198f60 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fec70000c00 tx=0x7fec70004550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.684+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70034850 con 0x7fec80100fe0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.684+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec70009490 con 0x7fec80100fe0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.684+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fec7002ce90 con 0x7fec80100fe0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.684+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70003d70 con 0x7fec80100fe0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.684+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec801a4690 con 0x7fec80100fe0 2024-03-24T09:33:12.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.684+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec80068950 con 0x7fec80100fe0 2024-03-24T09:33:12.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.686+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fec7003b040 con 0x7fec80100fe0 2024-03-24T09:33:12.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.686+0000 7fec7d7fa640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec680a2230 0x7fec680a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.686+0000 7fec7ffff640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec680a2230 0x7fec680a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.686+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7fec700f0150 con 0x7fec80100fe0 2024-03-24T09:33:12.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.687+0000 7fec7ffff640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec680a2230 0x7fec680a46f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fec74009b50 tx=0x7fec74005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.690+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fec700b8b40 con 0x7fec80100fe0 2024-03-24T09:33:12.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.862+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fec801089e0 con 0x7fec80100fe0 2024-03-24T09:33:12.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.863+0000 7fec7d7fa640 1 -- 172.21.15.116:0/4016867361 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) ==== 104+0+11 (secure 0 0 0) 0x7fec70041020 con 0x7fec80100fe0 2024-03-24T09:33:12.863 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:33:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec680a2230 msgr2=0x7fec680a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec680a2230 0x7fec680a46f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fec74009b50 tx=0x7fec74005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 msgr2=0x7fec80198f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 0x7fec80198f60 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fec70000c00 tx=0x7fec70004550 comp rx=0 tx=0).stop 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 shutdown_connections 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec80106690 0x7fec801a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec80100fe0 0x7fec80198f60 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fec680a2230 0x7fec680a46f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 --2- 172.21.15.116:0/4016867361 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fec80100620 0x7fec80198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 >> 172.21.15.116:0/4016867361 conn(0x7fec80077770 msgr2=0x7fec800ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 shutdown_connections 2024-03-24T09:33:12.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.865+0000 7fec862af640 1 -- 172.21.15.116:0/4016867361 wait complete. 2024-03-24T09:33:12.875 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:33:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.979+0000 7f9a78325640 1 -- 172.21.15.116:0/2839910505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a701005d0 msgr2=0x7f9a70100a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.979+0000 7f9a78325640 1 --2- 172.21.15.116:0/2839910505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a701005d0 0x7f9a70100a50 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009930 tx=0x7f9a6002bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- 172.21.15.116:0/2839910505 shutdown_connections 2024-03-24T09:33:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 --2- 172.21.15.116:0/2839910505 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a70100f90 0x7f9a7010d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 --2- 172.21.15.116:0/2839910505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a701005d0 0x7f9a70100a50 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 --2- 172.21.15.116:0/2839910505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a70106670 0x7f9a70106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- 172.21.15.116:0/2839910505 >> 172.21.15.116:0/2839910505 conn(0x7f9a70077720 msgr2=0x7f9a70077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- 172.21.15.116:0/2839910505 shutdown_connections 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- 172.21.15.116:0/2839910505 wait complete. 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 Processor -- start 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- start start 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a701005d0 0x7f9a7019c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a70100f90 0x7f9a7019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 0x7f9a701968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70196f40 con 0x7f9a70106670 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701970b0 con 0x7f9a701005d0 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.980+0000 7f9a78325640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70197270 con 0x7f9a70100f90 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 0x7f9a701968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 0x7f9a701968f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46838/0 (socket says 172.21.15.116:46838) 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a760c1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a701005d0 0x7f9a7019c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 -- 172.21.15.116:0/388939170 learned_addr learned my addr 172.21.15.116:0/388939170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a758c0640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a70100f90 0x7f9a7019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 -- 172.21.15.116:0/388939170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a70100f90 msgr2=0x7f9a7019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a70100f90 0x7f9a7019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 -- 172.21.15.116:0/388939170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a701005d0 msgr2=0x7f9a7019c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a701005d0 0x7f9a7019c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 -- 172.21.15.116:0/388939170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a68009490 con 0x7f9a70106670 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a760c1640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a701005d0 0x7f9a7019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a768c2640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 0x7f9a701968f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00eb10 tx=0x7f9a6c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c004b20 con 0x7f9a70106670 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9a6c004c80 con 0x7f9a70106670 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a60009560 con 0x7f9a70106670 2024-03-24T09:33:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c010690 con 0x7f9a70106670 2024-03-24T09:33:12.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.981+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a70069d40 con 0x7f9a70106670 2024-03-24T09:33:12.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.982+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a70068950 con 0x7f9a70106670 2024-03-24T09:33:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.983+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9a6c005020 con 0x7f9a70106670 2024-03-24T09:33:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.984+0000 7f9a677fe640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9a5c0a2230 0x7f9a5c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.984+0000 7f9a760c1640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9a5c0a2230 0x7f9a5c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.984+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f9a6c014070 con 0x7f9a70106670 2024-03-24T09:33:12.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.984+0000 7f9a760c1640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9a5c0a2230 0x7f9a5c0a46f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9a70197a30 tx=0x7f9a68010040 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:12.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:12.988+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9a6c0972f0 con 0x7f9a70106670 2024-03-24T09:33:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.163+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9a701106d0 con 0x7f9a70106670 2024-03-24T09:33:13.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.164+0000 7f9a677fe640 1 -- 172.21.15.116:0/388939170 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) ==== 100+0+11 (secure 0 0 0) 0x7f9a6c09a9b0 con 0x7f9a70106670 2024-03-24T09:33:13.165 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:33:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9a5c0a2230 msgr2=0x7f9a5c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9a5c0a2230 0x7f9a5c0a46f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9a70197a30 tx=0x7f9a68010040 comp rx=0 tx=0).stop 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 msgr2=0x7f9a701968f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 0x7f9a701968f0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00eb10 tx=0x7f9a6c00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 shutdown_connections 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a70106670 0x7f9a701968f0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9a5c0a2230 0x7f9a5c0a46f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9a70100f90 0x7f9a7019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 --2- 172.21.15.116:0/388939170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9a701005d0 0x7f9a7019c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 >> 172.21.15.116:0/388939170 conn(0x7f9a70077720 msgr2=0x7f9a700ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 shutdown_connections 2024-03-24T09:33:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.166+0000 7f9a78325640 1 -- 172.21.15.116:0/388939170 wait complete. 2024-03-24T09:33:13.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:13 smithi116 ceph-mon[32207]: pgmap v345: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:13.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/720600428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:13.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2710758272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:33:13.176 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:13 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4016867361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:33:13.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:33:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 -- 172.21.15.116:0/4071343523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be40fffd0 msgr2=0x7f1be4100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/4071343523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be40fffd0 0x7f1be4100450 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f1bd4009900 tx=0x7f1bd402bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:13.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 -- 172.21.15.116:0/4071343523 shutdown_connections 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/4071343523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be4100990 0x7f1be410e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/4071343523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be40fffd0 0x7f1be4100450 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/4071343523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be4107910 0x7f1be4107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 -- 172.21.15.116:0/4071343523 >> 172.21.15.116:0/4071343523 conn(0x7f1be4075950 msgr2=0x7f1be4075d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 -- 172.21.15.116:0/4071343523 shutdown_connections 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.281+0000 7f1beb8eb640 1 -- 172.21.15.116:0/4071343523 wait complete. 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 Processor -- start 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 -- start start 2024-03-24T09:33:13.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be40fffd0 0x7f1be419d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be4100990 0x7f1be419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 0x7f1be4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1be41980e0 con 0x7f1be4107910 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1be4198250 con 0x7f1be40fffd0 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f1be4198410 con 0x7f1be4100990 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 0x7f1be4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 0x7f1be4197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46856/0 (socket says 172.21.15.116:46856) 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 -- 172.21.15.116:0/2520800966 learned_addr learned my addr 172.21.15.116:0/2520800966 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9687640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be40fffd0 0x7f1be419d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be8e86640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be4100990 0x7f1be419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 -- 172.21.15.116:0/2520800966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be4100990 msgr2=0x7f1be419de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be4100990 0x7f1be419de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 -- 172.21.15.116:0/2520800966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be40fffd0 msgr2=0x7f1be419d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be40fffd0 0x7f1be419d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 -- 172.21.15.116:0/2520800966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bdc009840 con 0x7f1be4107910 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1be9e88640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 0x7f1be4197a90 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00efd0 tx=0x7f1bdc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc009b40 con 0x7f1be4107910 2024-03-24T09:33:13.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f1bdc003b70 con 0x7f1be4107910 2024-03-24T09:33:13.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.282+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd4009560 con 0x7f1be4107910 2024-03-24T09:33:13.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.283+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bdc0107e0 con 0x7f1be4107910 2024-03-24T09:33:13.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.283+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1be41a47a0 con 0x7f1be4107910 2024-03-24T09:33:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.284+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1be4068950 con 0x7f1be4107910 2024-03-24T09:33:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.284+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f1bdc003ce0 con 0x7f1be4107910 2024-03-24T09:33:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.285+0000 7f1bda7fc640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1bd00a22b0 0x7f1bd00a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.285+0000 7f1be9687640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1bd00a22b0 0x7f1bd00a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.285+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f1bdc014070 con 0x7f1be4107910 2024-03-24T09:33:13.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.285+0000 7f1be9687640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1bd00a22b0 0x7f1bd00a4770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1be4198bd0 tx=0x7f1be0005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:13.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.289+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f1bdc097240 con 0x7f1be4107910 2024-03-24T09:33:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:13 smithi117 ceph-mon[35788]: pgmap v345: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/720600428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2710758272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:33:13.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:13 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4016867361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:33:13.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.463+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1be4109bc0 con 0x7f1be4107910 2024-03-24T09:33:13.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.463+0000 7f1bda7fc640 1 -- 172.21.15.116:0/2520800966 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) ==== 103+0+11 (secure 0 0 0) 0x7f1bdc09a900 con 0x7f1be4107910 2024-03-24T09:33:13.464 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:33:13.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1bd00a22b0 msgr2=0x7f1bd00a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1bd00a22b0 0x7f1bd00a4770 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1be4198bd0 tx=0x7f1be0005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:13.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 msgr2=0x7f1be4197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 0x7f1be4197a90 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc00efd0 tx=0x7f1bdc00c6a0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 shutdown_connections 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1be4107910 0x7f1be4197a90 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f1bd00a22b0 0x7f1bd00a4770 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1be4100990 0x7f1be419de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 --2- 172.21.15.116:0/2520800966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1be40fffd0 0x7f1be419d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.465+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 >> 172.21.15.116:0/2520800966 conn(0x7f1be4075950 msgr2=0x7f1be40fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.466+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 shutdown_connections 2024-03-24T09:33:13.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.466+0000 7f1beb8eb640 1 -- 172.21.15.116:0/2520800966 wait complete. 2024-03-24T09:33:13.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T09:33:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:13 smithi143 ceph-mon[35965]: pgmap v345: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/720600428' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2710758272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:33:13.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:13 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4016867361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:33:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 -- 172.21.15.116:0/343788041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2964100f80 msgr2=0x7f296410d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 --2- 172.21.15.116:0/343788041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2964100f80 0x7f296410d490 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f295c009cd0 tx=0x7f295c02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 -- 172.21.15.116:0/343788041 shutdown_connections 2024-03-24T09:33:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 --2- 172.21.15.116:0/343788041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2964100f80 0x7f296410d490 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 --2- 172.21.15.116:0/343788041 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29641005c0 0x7f2964100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 --2- 172.21.15.116:0/343788041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2964106660 0x7f2964106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 -- 172.21.15.116:0/343788041 >> 172.21.15.116:0/343788041 conn(0x7f2964076990 msgr2=0x7f2964076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 -- 172.21.15.116:0/343788041 shutdown_connections 2024-03-24T09:33:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.579+0000 7f2969e42640 1 -- 172.21.15.116:0/343788041 wait complete. 2024-03-24T09:33:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 Processor -- start 2024-03-24T09:33:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 -- start start 2024-03-24T09:33:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 0x7f296419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2964100f80 0x7f296419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964106660 0x7f29641968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2964196f30 con 0x7f29641005c0 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29641970a0 con 0x7f2964100f80 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2969e42640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f2964197260 con 0x7f2964106660 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 0x7f296419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 0x7f296419c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46874/0 (socket says 172.21.15.116:46874) 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 -- 172.21.15.116:0/1473509648 learned_addr learned my addr 172.21.15.116:0/1473509648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2962ffd640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2964100f80 0x7f296419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2963fff640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964106660 0x7f29641968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 -- 172.21.15.116:0/1473509648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964106660 msgr2=0x7f29641968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964106660 0x7f29641968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 -- 172.21.15.116:0/1473509648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2964100f80 msgr2=0x7f296419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2964100f80 0x7f296419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f29637fe640 1 -- 172.21.15.116:0/1473509648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295c009900 con 0x7f29641005c0 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.580+0000 7f2963fff640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964106660 0x7f29641968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.581+0000 7f29637fe640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 0x7f296419c730 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f295800e9c0 tx=0x7f295800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.581+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958009700 con 0x7f29641005c0 2024-03-24T09:33:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.581+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2964197540 con 0x7f29641005c0 2024-03-24T09:33:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.581+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f29580044d0 con 0x7f29641005c0 2024-03-24T09:33:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.581+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958010430 con 0x7f29641005c0 2024-03-24T09:33:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.581+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29641a37d0 con 0x7f29641005c0 2024-03-24T09:33:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.582+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2964101d00 con 0x7f29641005c0 2024-03-24T09:33:13.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.583+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f2958003f10 con 0x7f29641005c0 2024-03-24T09:33:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.584+0000 7f2960ff9640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f294c0a2230 0x7f294c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.584+0000 7f2962ffd640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f294c0a2230 0x7f294c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.584+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f2958014070 con 0x7f29641005c0 2024-03-24T09:33:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.584+0000 7f2962ffd640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f294c0a2230 0x7f294c0a46f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2964105cd0 tx=0x7f2954005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:13.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.587+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f2958097310 con 0x7f29641005c0 2024-03-24T09:33:13.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.772+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f29641976d0 con 0x7f29641005c0 2024-03-24T09:33:13.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.772+0000 7f2960ff9640 1 -- 172.21.15.116:0/1473509648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) ==== 81+0+957 (secure 0 0 0) 0x7f29641976d0 con 0x7f29641005c0 2024-03-24T09:33:13.773 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/2363601294 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:13.773 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/664840167 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:13.773 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1503975320 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:13.773 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/2363601294 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:13.773 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4025940557 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/227139610 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3305720445 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3598626147 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:13.774 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3855909741 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:13.775 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/448304506 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:13.775 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:13.775 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3108228013 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:13.775 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2688022316 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:listed 16 entries 2024-03-24T09:33:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f294c0a2230 msgr2=0x7f294c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f294c0a2230 0x7f294c0a46f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2964105cd0 tx=0x7f2954005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 msgr2=0x7f296419c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 0x7f296419c730 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f295800e9c0 tx=0x7f295800ee80 comp rx=0 tx=0).stop 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 shutdown_connections 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f294c0a2230 0x7f294c0a46f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2964106660 0x7f29641968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2964100f80 0x7f296419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 --2- 172.21.15.116:0/1473509648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f29641005c0 0x7f296419c730 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 >> 172.21.15.116:0/1473509648 conn(0x7f2964076990 msgr2=0x7f29640feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 shutdown_connections 2024-03-24T09:33:13.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.774+0000 7f2969e42640 1 -- 172.21.15.116:0/1473509648 wait complete. 2024-03-24T09:33:13.785 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-24T09:33:13.785 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T09:33:13.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:33:13.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 -- 172.21.15.116:0/1914034063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 msgr2=0x7f969010d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1914034063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969010d4e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f9688009ca0 tx=0x7f968802c180 comp rx=0 tx=0).stop 2024-03-24T09:33:13.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 -- 172.21.15.116:0/1914034063 shutdown_connections 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1914034063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969010d4e0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1914034063 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690106fd0 0x7f9690100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1914034063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901066b0 0x7f9690106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 -- 172.21.15.116:0/1914034063 >> 172.21.15.116:0/1914034063 conn(0x7f9690077710 msgr2=0x7f9690077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 -- 172.21.15.116:0/1914034063 shutdown_connections 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 -- 172.21.15.116:0/1914034063 wait complete. 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.943+0000 7f9695e0b640 1 Processor -- start 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 -- start start 2024-03-24T09:33:13.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969019c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901066b0 0x7f969019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690106fd0 0x7f9690196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9690196fd0 con 0x7f9690100d40 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9690197140 con 0x7f96901066b0 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9690197300 con 0x7f9690106fd0 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969019c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969019c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46890/0 (socket says 172.21.15.116:46890) 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 -- 172.21.15.116:0/1189115350 learned_addr learned my addr 172.21.15.116:0/1189115350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:13.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9687fff640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901066b0 0x7f969019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968ffff640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690106fd0 0x7f9690196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 -- 172.21.15.116:0/1189115350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690106fd0 msgr2=0x7f9690196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690106fd0 0x7f9690196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 -- 172.21.15.116:0/1189115350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901066b0 msgr2=0x7f969019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901066b0 0x7f969019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 -- 172.21.15.116:0/1189115350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9680009490 con 0x7f9690100d40 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968f7fe640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969019c780 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f968000e9c0 tx=0x7f968000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:13.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968000cde0 con 0x7f9690100d40 2024-03-24T09:33:13.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f96800108e0 con 0x7f9690100d40 2024-03-24T09:33:13.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9688009900 con 0x7f9690100d40 2024-03-24T09:33:13.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96800188e0 con 0x7f9690100d40 2024-03-24T09:33:13.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.944+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9690068870 con 0x7f9690100d40 2024-03-24T09:33:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.945+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9690101d50 con 0x7f9690100d40 2024-03-24T09:33:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.946+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9680010a50 con 0x7f9690100d40 2024-03-24T09:33:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.947+0000 7f968d7fa640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2230 0x7f967c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.947+0000 7f9687fff640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2230 0x7f967c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.947+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9642+0+0 (secure 0 0 0) 0x7f9680014070 con 0x7f9690100d40 2024-03-24T09:33:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.947+0000 7f9687fff640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2230 0x7f967c0a46f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9690105d40 tx=0x7f9678005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:13.951+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f96800973c0 con 0x7f9690100d40 2024-03-24T09:33:14.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.134+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9690068500 con 0x7f9690100d40 2024-03-24T09:33:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.136+0000 7f968d7fa640 1 -- 172.21.15.116:0/1189115350 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25479 (secure 0 0 0) 0x7f968009aa80 con 0x7f9690100d40 2024-03-24T09:33:14.136 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:33:14.137 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":104,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:33:14.052520+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:33:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2230 msgr2=0x7f967c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2230 0x7f967c0a46f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9690105d40 tx=0x7f9678005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 msgr2=0x7f969019c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969019c780 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f968000e9c0 tx=0x7f968000ee80 comp rx=0 tx=0).stop 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 shutdown_connections 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f967c0a2230 0x7f967c0a46f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9690106fd0 0x7f9690196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96901066b0 0x7f969019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 --2- 172.21.15.116:0/1189115350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690100d40 0x7f969019c780 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 >> 172.21.15.116:0/1189115350 conn(0x7f9690077710 msgr2=0x7f96900ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 shutdown_connections 2024-03-24T09:33:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.138+0000 7f9695e0b640 1 -- 172.21.15.116:0/1189115350 wait complete. 2024-03-24T09:33:14.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T09:15:00.179296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T09:19:36.455512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T09:19:37.676709+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-24T09:19:40.951352+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T09:33:14.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T09:33:14.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.257+0000 7fe872b43640 1 -- 172.21.15.116:0/2280319443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c06a740 msgr2=0x7fe86c10e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- 172.21.15.116:0/2280319443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c06a740 0x7fe86c10e550 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fe864009ca0 tx=0x7fe86402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- 172.21.15.116:0/2280319443 shutdown_connections 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- 172.21.15.116:0/2280319443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c06a740 0x7fe86c10e550 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- 172.21.15.116:0/2280319443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c069d80 0x7fe86c06a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- 172.21.15.116:0/2280319443 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c107740 0x7fe86c107b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- 172.21.15.116:0/2280319443 >> 172.21.15.116:0/2280319443 conn(0x7fe86c0773f0 msgr2=0x7fe86c077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- 172.21.15.116:0/2280319443 shutdown_connections 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- 172.21.15.116:0/2280319443 wait complete. 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 Processor -- start 2024-03-24T09:33:14.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- start start 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 0x7fe86c19d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c06a740 0x7fe86c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c107740 0x7fe86c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c197fd0 con 0x7fe86c069d80 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c198140 con 0x7fe86c107740 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe872b43640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c198300 con 0x7fe86c06a740 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe8708df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 0x7fe86c19d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe8708df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 0x7fe86c19d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46914/0 (socket says 172.21.15.116:46914) 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.258+0000 7fe8708df640 1 -- 172.21.15.116:0/4132624116 learned_addr learned my addr 172.21.15.116:0/4132624116 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8710e0640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c107740 0x7fe86c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe86bfff640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c06a740 0x7fe86c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8708df640 1 -- 172.21.15.116:0/4132624116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c06a740 msgr2=0x7fe86c19dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8708df640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c06a740 0x7fe86c19dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8708df640 1 -- 172.21.15.116:0/4132624116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c107740 msgr2=0x7fe86c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8708df640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c107740 0x7fe86c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8708df640 1 -- 172.21.15.116:0/4132624116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe864009900 con 0x7fe86c069d80 2024-03-24T09:33:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe8708df640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 0x7fe86c19d7d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fe86000e9f0 tx=0x7fe86000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860009720 con 0x7fe86c069d80 2024-03-24T09:33:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c1985e0 con 0x7fe86c069d80 2024-03-24T09:33:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fe86000cd80 con 0x7fe86c069d80 2024-03-24T09:33:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860010430 con 0x7fe86c069d80 2024-03-24T09:33:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.259+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c1a4870 con 0x7fe86c069d80 2024-03-24T09:33:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.260+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c068950 con 0x7fe86c069d80 2024-03-24T09:33:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.261+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fe860005020 con 0x7fe86c069d80 2024-03-24T09:33:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.262+0000 7fe869ffb640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe85c0a2230 0x7fe85c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.262+0000 7fe86bfff640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe85c0a2230 0x7fe85c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.262+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fe860014070 con 0x7fe86c069d80 2024-03-24T09:33:14.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.262+0000 7fe86bfff640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe85c0a2230 0x7fe85c0a46f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe86c100410 tx=0x7fe858005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:14.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.266+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fe860097050 con 0x7fe86c069d80 2024-03-24T09:33:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:14 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/388939170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:33:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:14 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2520800966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:33:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:14 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1473509648' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:33:14.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:14 smithi117 ceph-mon[35788]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T09:33:14.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:14 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/388939170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:33:14.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:14 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2520800966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:33:14.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:14 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1473509648' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:33:14.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:14 smithi116 ceph-mon[32207]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T09:33:14.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.437+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe86c109c70 con 0x7fe86c069d80 2024-03-24T09:33:14.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.438+0000 7fe869ffb640 1 -- 172.21.15.116:0/4132624116 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7fe86009a710 con 0x7fe86c069d80 2024-03-24T09:33:14.439 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-24T09:33:14.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe85c0a2230 msgr2=0x7fe85c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe85c0a2230 0x7fe85c0a46f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe86c100410 tx=0x7fe858005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 msgr2=0x7fe86c19d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 0x7fe86c19d7d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fe86000e9f0 tx=0x7fe86000edf0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 shutdown_connections 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe86c107740 0x7fe86c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fe85c0a2230 0x7fe85c0a46f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe86c06a740 0x7fe86c19dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 --2- 172.21.15.116:0/4132624116 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c069d80 0x7fe86c19d7d0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 >> 172.21.15.116:0/4132624116 conn(0x7fe86c0773f0 msgr2=0x7fe86c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 shutdown_connections 2024-03-24T09:33:14.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.440+0000 7fe872b43640 1 -- 172.21.15.116:0/4132624116 wait complete. 2024-03-24T09:33:14.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T09:33:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:14 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/388939170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:33:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:14 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2520800966' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:33:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:14 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1473509648' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:33:14.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:14 smithi143 ceph-mon[35965]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T09:33:14.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 -- 172.21.15.116:0/1624050530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac10bd70 msgr2=0x7fc5ac10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/1624050530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac10bd70 0x7fc5ac10e1a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4009be0 tx=0x7fc5a402bf80 comp rx=0 tx=0).stop 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 -- 172.21.15.116:0/1624050530 shutdown_connections 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/1624050530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac10bd70 0x7fc5ac10e1a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/1624050530 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5ac1032c0 0x7fc5ac10b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/1624050530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 -- 172.21.15.116:0/1624050530 >> 172.21.15.116:0/1624050530 conn(0x7fc5ac077770 msgr2=0x7fc5ac077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 -- 172.21.15.116:0/1624050530 shutdown_connections 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.555+0000 7fc5b3c74640 1 -- 172.21.15.116:0/1624050530 wait complete. 2024-03-24T09:33:14.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 Processor -- start 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 -- start start 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac19d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac1032c0 0x7fc5ac19de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5ac10bd70 0x7fc5ac197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac198140 con 0x7fc5ac1029a0 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac1982b0 con 0x7fc5ac1032c0 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b3c74640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ac198470 con 0x7fc5ac10bd70 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac19d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac19d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46918/0 (socket says 172.21.15.116:46918) 2024-03-24T09:33:14.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 -- 172.21.15.116:0/3109626221 learned_addr learned my addr 172.21.15.116:0/3109626221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b120f640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac1032c0 0x7fc5ac19de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b2211640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5ac10bd70 0x7fc5ac197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 -- 172.21.15.116:0/3109626221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5ac10bd70 msgr2=0x7fc5ac197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5ac10bd70 0x7fc5ac197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 -- 172.21.15.116:0/3109626221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac1032c0 msgr2=0x7fc5ac19de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac1032c0 0x7fc5ac19de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.556+0000 7fc5b1a10640 1 -- 172.21.15.116:0/3109626221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a8009560 con 0x7fc5ac1029a0 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.557+0000 7fc5b1a10640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac19d8f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800e9c0 tx=0x7fc5a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:14.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.557+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a800cde0 con 0x7fc5ac1029a0 2024-03-24T09:33:14.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.557+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a4009840 con 0x7fc5ac1029a0 2024-03-24T09:33:14.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.557+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fc5a80044d0 con 0x7fc5ac1029a0 2024-03-24T09:33:14.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.557+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ac198940 con 0x7fc5ac1029a0 2024-03-24T09:33:14.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.557+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8018430 con 0x7fc5ac1029a0 2024-03-24T09:33:14.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.558+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5ac06a510 con 0x7fc5ac1029a0 2024-03-24T09:33:14.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.559+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fc5a8003fe0 con 0x7fc5ac1029a0 2024-03-24T09:33:14.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.559+0000 7fc5a2ffd640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc5980a2230 0x7fc5980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.559+0000 7fc5b120f640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc5980a2230 0x7fc5980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.560+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fc5a8014070 con 0x7fc5ac1029a0 2024-03-24T09:33:14.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.560+0000 7fc5b120f640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc5980a2230 0x7fc5980a46f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac10dad0 tx=0x7fc59c005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:14.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.563+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fc5a80972d0 con 0x7fc5ac1029a0 2024-03-24T09:33:14.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.735+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc5ac106e60 con 0x7fc5ac1029a0 2024-03-24T09:33:14.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.736+0000 7fc5a2ffd640 1 -- 172.21.15.116:0/3109626221 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7fc5a809a990 con 0x7fc5ac1029a0 2024-03-24T09:33:14.736 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:33:14.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc5980a2230 msgr2=0x7fc5980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc5980a2230 0x7fc5980a46f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac10dad0 tx=0x7fc59c005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 msgr2=0x7fc5ac19d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac19d8f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800e9c0 tx=0x7fc5a800ee80 comp rx=0 tx=0).stop 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 shutdown_connections 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fc5980a2230 0x7fc5980a46f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc5ac10bd70 0x7fc5ac197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5ac1032c0 0x7fc5ac19de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 --2- 172.21.15.116:0/3109626221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5ac1029a0 0x7fc5ac19d8f0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 >> 172.21.15.116:0/3109626221 conn(0x7fc5ac077770 msgr2=0x7fc5ac0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 shutdown_connections 2024-03-24T09:33:14.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.738+0000 7fc5b3c74640 1 -- 172.21.15.116:0/3109626221 wait complete. 2024-03-24T09:33:14.750 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 -- 172.21.15.116:0/636271239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 msgr2=0x7f9b34107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 --2- 172.21.15.116:0/636271239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b34107c40 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009930 tx=0x7f9b2402bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 -- 172.21.15.116:0/636271239 shutdown_connections 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 --2- 172.21.15.116:0/636271239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3406a720 0x7f9b3410e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 --2- 172.21.15.116:0/636271239 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b3406a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 --2- 172.21.15.116:0/636271239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b34107c40 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 -- 172.21.15.116:0/636271239 >> 172.21.15.116:0/636271239 conn(0x7f9b340773b0 msgr2=0x7f9b340777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:14.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 -- 172.21.15.116:0/636271239 shutdown_connections 2024-03-24T09:33:14.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.857+0000 7f9b38b81640 1 -- 172.21.15.116:0/636271239 wait complete. 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 Processor -- start 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 -- start start 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b34199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3406a720 0x7f9b3419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b3419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3419eca0 con 0x7f9b34107860 2024-03-24T09:33:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3419ee10 con 0x7f9b3406a720 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b38b81640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3419efd0 con 0x7f9b34069d80 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b3419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b34199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b3419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46942/0 (socket says 172.21.15.116:46942) 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b34199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.116:38138/0 (socket says 172.21.15.116:38138) 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b32d9d640 1 -- 172.21.15.116:0/4229583895 learned_addr learned my addr 172.21.15.116:0/4229583895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.858+0000 7f9b31d9b640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3406a720 0x7f9b3419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b32d9d640 1 -- 172.21.15.116:0/4229583895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 msgr2=0x7f9b34199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b32d9d640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b34199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b32d9d640 1 -- 172.21.15.116:0/4229583895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3406a720 msgr2=0x7f9b3419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b32d9d640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3406a720 0x7f9b3419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b32d9d640 1 -- 172.21.15.116:0/4229583895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b24009560 con 0x7f9b34107860 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b3259c640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b34199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b32d9d640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b3419e6a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00eb10 tx=0x7f9b2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c004b20 con 0x7f9b34107860 2024-03-24T09:33:14.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3419f2b0 con 0x7f9b34107860 2024-03-24T09:33:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f9b2c004c80 con 0x7f9b34107860 2024-03-24T09:33:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c010690 con 0x7f9b34107860 2024-03-24T09:33:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.859+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b341a6b30 con 0x7f9b34107860 2024-03-24T09:33:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.860+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b34102dc0 con 0x7f9b34107860 2024-03-24T09:33:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.861+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f9b2c0107f0 con 0x7f9b34107860 2024-03-24T09:33:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.862+0000 7f9b237fe640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b1c0a2230 0x7f9b1c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.862+0000 7f9b3259c640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b1c0a2230 0x7f9b1c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.862+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f9b2c014070 con 0x7f9b34107860 2024-03-24T09:33:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.862+0000 7f9b3259c640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b1c0a2230 0x7f9b1c0a46f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9b24005ab0 tx=0x7f9b2402c6d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:14.865+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f9b2c0974b0 con 0x7f9b34107860 2024-03-24T09:33:15.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.037+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9b34068950 con 0x7f9b34107860 2024-03-24T09:33:15.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.038+0000 7f9b237fe640 1 -- 172.21.15.116:0/4229583895 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7f9b2c09ab70 con 0x7f9b34107860 2024-03-24T09:33:15.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-24T09:33:15.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b1c0a2230 msgr2=0x7f9b1c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b1c0a2230 0x7f9b1c0a46f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9b24005ab0 tx=0x7f9b2402c6d0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 msgr2=0x7f9b3419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b3419e6a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00eb10 tx=0x7f9b2c00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 shutdown_connections 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b34107860 0x7f9b3419e6a0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b3406a720 0x7f9b3419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f9b1c0a2230 0x7f9b1c0a46f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 --2- 172.21.15.116:0/4229583895 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9b34069d80 0x7f9b34199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 >> 172.21.15.116:0/4229583895 conn(0x7f9b340773b0 msgr2=0x7f9b3410c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:15.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 shutdown_connections 2024-03-24T09:33:15.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.040+0000 7f9b38b81640 1 -- 172.21.15.116:0/4229583895 wait complete. 2024-03-24T09:33:15.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-24T09:33:15.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 -- 172.21.15.116:0/3142980893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b41005c0 msgr2=0x7f70b4100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/3142980893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b41005c0 0x7f70b4100a40 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f709c009930 tx=0x7f709c02bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 -- 172.21.15.116:0/3142980893 shutdown_connections 2024-03-24T09:33:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/3142980893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70b4100f80 0x7f70b410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/3142980893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b41005c0 0x7f70b4100a40 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/3142980893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b4106660 0x7f70b4106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 -- 172.21.15.116:0/3142980893 >> 172.21.15.116:0/3142980893 conn(0x7f70b4076990 msgr2=0x7f70b4076da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:15.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 -- 172.21.15.116:0/3142980893 shutdown_connections 2024-03-24T09:33:15.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.155+0000 7f70b8c4e640 1 -- 172.21.15.116:0/3142980893 wait complete. 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 Processor -- start 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 -- start start 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b41005c0 0x7f70b419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70b4100f80 0x7f70b419cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 0x7f70b41969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4197010 con 0x7f70b4106660 2024-03-24T09:33:15.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4197180 con 0x7f70b4100f80 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b8c4e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f70b4197340 con 0x7f70b41005c0 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 0x7f70b41969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 0x7f70b41969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46964/0 (socket says 172.21.15.116:46964) 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 -- 172.21.15.116:0/2139837982 learned_addr learned my addr 172.21.15.116:0/2139837982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70abfff640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70b4100f80 0x7f70b419cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b259c640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b41005c0 0x7f70b419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 -- 172.21.15.116:0/2139837982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b41005c0 msgr2=0x7f70b419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b41005c0 0x7f70b419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 -- 172.21.15.116:0/2139837982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70b4100f80 msgr2=0x7f70b419cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70b4100f80 0x7f70b419cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.156+0000 7f70b2d9d640 1 -- 172.21.15.116:0/2139837982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709c009560 con 0x7f70b4106660 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.157+0000 7f70b2d9d640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 0x7f70b41969c0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00eb10 tx=0x7f70ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.157+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac004b20 con 0x7f70b4106660 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.157+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f70ac004c80 con 0x7f70b4106660 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.157+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b4197620 con 0x7f70b4106660 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.157+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac010690 con 0x7f70b4106660 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.157+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b41a38b0 con 0x7f70b4106660 2024-03-24T09:33:15.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.158+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b4101d00 con 0x7f70b4106660 2024-03-24T09:33:15.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.159+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f70ac005020 con 0x7f70b4106660 2024-03-24T09:33:15.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.159+0000 7f70ab7fe640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f70a00a2230 0x7f70a00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.160+0000 7f70b259c640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f70a00a2230 0x7f70a00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.160+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f70ac014070 con 0x7f70b4106660 2024-03-24T09:33:15.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.160+0000 7f70b259c640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f70a00a2230 0x7f70a00a46f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f70b4197b00 tx=0x7f70a4005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:15.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.164+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f70ac0972f0 con 0x7f70b4106660 2024-03-24T09:33:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:15 smithi117 ceph-mon[35788]: pgmap v346: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:15 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1189115350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:15 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4132624116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:15 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3109626221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:15 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/4229583895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:15 smithi116 ceph-mon[32207]: pgmap v346: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:15.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:15 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1189115350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:15.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:15 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4132624116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:15 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3109626221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.318 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:15 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/4229583895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.340+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f70b4108bb0 con 0x7f70b4106660 2024-03-24T09:33:15.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.340+0000 7f70ab7fe640 1 -- 172.21.15.116:0/2139837982 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) ==== 103+0+11 (secure 0 0 0) 0x7f70ac09a9b0 con 0x7f70b4106660 2024-03-24T09:33:15.341 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-24T09:33:15.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f70a00a2230 msgr2=0x7f70a00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f70a00a2230 0x7f70a00a46f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f70b4197b00 tx=0x7f70a4005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:15.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 msgr2=0x7f70b41969c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 0x7f70b41969c0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f70ac00eb10 tx=0x7f70ac00c2d0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 shutdown_connections 2024-03-24T09:33:15.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70b4106660 0x7f70b41969c0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70b4100f80 0x7f70b419cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f70a00a2230 0x7f70a00a46f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 --2- 172.21.15.116:0/2139837982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f70b41005c0 0x7f70b419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 >> 172.21.15.116:0/2139837982 conn(0x7f70b4076990 msgr2=0x7f70b40fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:15.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 shutdown_connections 2024-03-24T09:33:15.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.342+0000 7f70b8c4e640 1 -- 172.21.15.116:0/2139837982 wait complete. 2024-03-24T09:33:15.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T09:33:15.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 -- 172.21.15.116:0/1717073338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a0102980 msgr2=0x7fd2a0102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 --2- 172.21.15.116:0/1717073338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a0102980 0x7fd2a0102d60 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd290009900 tx=0x7fd29002bcd0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 -- 172.21.15.116:0/1717073338 shutdown_connections 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 --2- 172.21.15.116:0/1717073338 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a010bd50 0x7fd2a010e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 --2- 172.21.15.116:0/1717073338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a01032a0 0x7fd2a010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 --2- 172.21.15.116:0/1717073338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a0102980 0x7fd2a0102d60 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 -- 172.21.15.116:0/1717073338 >> 172.21.15.116:0/1717073338 conn(0x7fd2a0077770 msgr2=0x7fd2a0077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 -- 172.21.15.116:0/1717073338 shutdown_connections 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.459+0000 7fd2a596c640 1 -- 172.21.15.116:0/1717073338 wait complete. 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 Processor -- start 2024-03-24T09:33:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 -- start start 2024-03-24T09:33:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a0102980 0x7fd2a019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 0x7fd2a019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a010bd50 0x7fd2a0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0198100 con 0x7fd2a01032a0 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0198270 con 0x7fd2a010bd50 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd2a596c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a0198430 con 0x7fd2a0102980 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd29e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 0x7fd2a019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd29e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 0x7fd2a019de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46982/0 (socket says 172.21.15.116:46982) 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd29e7fc640 1 -- 172.21.15.116:0/2654156582 learned_addr learned my addr 172.21.15.116:0/2654156582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd29effd640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a0102980 0x7fd2a019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.460+0000 7fd29f7fe640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a010bd50 0x7fd2a0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd29e7fc640 1 -- 172.21.15.116:0/2654156582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a0102980 msgr2=0x7fd2a019d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd29e7fc640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a0102980 0x7fd2a019d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd29e7fc640 1 -- 172.21.15.116:0/2654156582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a010bd50 msgr2=0x7fd2a0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd29e7fc640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a010bd50 0x7fd2a0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd29e7fc640 1 -- 172.21.15.116:0/2654156582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd294009490 con 0x7fd2a01032a0 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd29e7fc640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 0x7fd2a019de40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fd29400ea60 tx=0x7fd29400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29400cde0 con 0x7fd2a01032a0 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd2940044d0 con 0x7fd2a01032a0 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd290009560 con 0x7fd2a01032a0 2024-03-24T09:33:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294010690 con 0x7fd2a01032a0 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.461+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a01a47c0 con 0x7fd2a01032a0 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.462+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a0068950 con 0x7fd2a01032a0 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.463+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd294003e40 con 0x7fd2a01032a0 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.464+0000 7fd2a496a640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2880a2230 0x7fd2880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.464+0000 7fd29effd640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2880a2230 0x7fd2880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.464+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fd294014070 con 0x7fd2a01032a0 2024-03-24T09:33:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.464+0000 7fd29effd640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2880a2230 0x7fd2880a46f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd290009230 tx=0x7fd290005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:15.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.468+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd2940971b0 con 0x7fd2a01032a0 2024-03-24T09:33:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:15 smithi143 ceph-mon[35965]: pgmap v346: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:15 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1189115350' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:15 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4132624116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:15 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3109626221' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:15 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/4229583895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T09:33:15.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.654+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd2a0106ea0 con 0x7fd2a01032a0 2024-03-24T09:33:15.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.654+0000 7fd2a496a640 1 -- 172.21.15.116:0/2654156582 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) ==== 81+0+957 (secure 0 0 0) 0x7fd29409a870 con 0x7fd2a01032a0 2024-03-24T09:33:15.655 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/2363601294 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:15.655 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/664840167 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:15.655 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1503975320 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:15.655 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/2363601294 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/4025940557 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/227139610 2024-03-25T09:22:05.255838+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3305720445 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3598626147 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3871815735 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3855909741 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:15.656 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/448304506 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:15.657 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/285563997 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:15.657 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3108228013 2024-03-25T09:12:07.749841+0000 2024-03-24T09:33:15.657 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2688022316 2024-03-25T09:12:33.839806+0000 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:listed 16 entries 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2880a2230 msgr2=0x7fd2880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2880a2230 0x7fd2880a46f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd290009230 tx=0x7fd290005fd0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 msgr2=0x7fd2a019de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 0x7fd2a019de40 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fd29400ea60 tx=0x7fd29400ef20 comp rx=0 tx=0).stop 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 shutdown_connections 2024-03-24T09:33:15.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a010bd50 0x7fd2a0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2a01032a0 0x7fd2a019de40 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd2880a2230 0x7fd2880a46f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 --2- 172.21.15.116:0/2654156582 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2a0102980 0x7fd2a019d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:15.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.656+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 >> 172.21.15.116:0/2654156582 conn(0x7fd2a0077770 msgr2=0x7fd2a00fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:15.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.657+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 shutdown_connections 2024-03-24T09:33:15.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:15.657+0000 7fd2a596c640 1 -- 172.21.15.116:0/2654156582 wait complete. 2024-03-24T09:33:15.668 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-03-24T09:33:15.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:33:15.668 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T09:33:15.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[86622]: fuse finished with error 0 and tester_r 0 2024-03-24T09:33:15.706 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T09:33:16.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:16 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2139837982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:33:16.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:16 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2654156582' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:33:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:16 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2139837982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:33:16.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:16 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2654156582' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:33:16.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:16 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2139837982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-24T09:33:16.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:16 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2654156582' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T09:33:17.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:17 smithi116 ceph-mon[32207]: pgmap v348: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:33:17.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:17 smithi143 ceph-mon[35965]: pgmap v348: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:33:17.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:17 smithi117 ceph-mon[35788]: pgmap v348: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-24T09:33:19.326 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:19 smithi143 ceph-mon[35965]: pgmap v349: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2024-03-24T09:33:19.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:19 smithi117 ceph-mon[35788]: pgmap v349: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2024-03-24T09:33:19.568 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:19 smithi116 ceph-mon[32207]: pgmap v349: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 217 B/s wr, 11 op/s 2024-03-24T09:33:21.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:21 smithi143 ceph-mon[35965]: pgmap v350: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:21.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:21 smithi117 ceph-mon[35788]: pgmap v350: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:21.567 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:21 smithi116 ceph-mon[32207]: pgmap v350: 129 pgs: 129 active+clean; 150 MiB data, 905 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-24T09:33:21.808 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:33:21.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:33:21.809 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T09:33:21.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:33:21.826 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-24T09:33:21.881 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T09:33:21.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:33:21.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T09:33:21.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T09:33:21.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-24T09:33:22.070 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T09:33:22.080 INFO:tasks.cephadm:Teardown begin 2024-03-24T09:33:22.081 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T09:33:22.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:33:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 -- 172.21.15.116:0/3038141277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c10bd50 msgr2=0x7f726c10e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 --2- 172.21.15.116:0/3038141277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c10bd50 0x7f726c10e1f0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f7264009ca0 tx=0x7f726402c0f0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 -- 172.21.15.116:0/3038141277 shutdown_connections 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 --2- 172.21.15.116:0/3038141277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c10bd50 0x7f726c10e1f0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 --2- 172.21.15.116:0/3038141277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f726c1032a0 0x7f726c10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 --2- 172.21.15.116:0/3038141277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c102980 0x7f726c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 -- 172.21.15.116:0/3038141277 >> 172.21.15.116:0/3038141277 conn(0x7f726c077710 msgr2=0x7f726c077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 -- 172.21.15.116:0/3038141277 shutdown_connections 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.188+0000 7f7271607640 1 -- 172.21.15.116:0/3038141277 wait complete. 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 Processor -- start 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 -- start start 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 0x7f726c19d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c1032a0 0x7f726c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f726c10bd50 0x7f726c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f726c1980b0 con 0x7f726c102980 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f726c198220 con 0x7f726c1032a0 2024-03-24T09:33:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7271607640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f726c1983e0 con 0x7f726c10bd50 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 0x7f726c19d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 0x7f726c19d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35342/0 (socket says 172.21.15.116:35342) 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 -- 172.21.15.116:0/2560611057 learned_addr learned my addr 172.21.15.116:0/2560611057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f7263fff640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c1032a0 0x7f726c19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726b7fe640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f726c10bd50 0x7f726c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 -- 172.21.15.116:0/2560611057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f726c10bd50 msgr2=0x7f726c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f726c10bd50 0x7f726c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 -- 172.21.15.116:0/2560611057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c1032a0 msgr2=0x7f726c19ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c1032a0 0x7f726c19ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 -- 172.21.15.116:0/2560611057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7254009490 con 0x7f726c102980 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.189+0000 7f726affd640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 0x7f726c19d8b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f725c00e9f0 tx=0x7f725c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.190+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c009700 con 0x7f726c102980 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.190+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7264009900 con 0x7f726c102980 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.190+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f725c00cd80 con 0x7f726c102980 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.190+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c018840 con 0x7f726c102980 2024-03-24T09:33:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.190+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f726c1a4770 con 0x7f726c102980 2024-03-24T09:33:22.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.191+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f726c068500 con 0x7f726c102980 2024-03-24T09:33:22.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.192+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f725c0189a0 con 0x7f726c102980 2024-03-24T09:33:22.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.192+0000 7f7268ff9640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72580a2230 0x7f72580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.192+0000 7f7263fff640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72580a2230 0x7f72580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.192+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f725c014070 con 0x7f726c102980 2024-03-24T09:33:22.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.193+0000 7f7263fff640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72580a2230 0x7f72580a46f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f726c10db90 tx=0x7f7254005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:22.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.197+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f725c053280 con 0x7f726c102980 2024-03-24T09:33:22.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:22 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:33:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.438+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f726c106e40 con 0x7f726c102980 2024-03-24T09:33:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.438+0000 7f7268ff9640 1 -- 172.21.15.116:0/2560611057 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f725c097120 con 0x7f726c102980 2024-03-24T09:33:22.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:33:22.439 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:33:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:33:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.440+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72580a2230 msgr2=0x7f72580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.440+0000 7f7271607640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72580a2230 0x7f72580a46f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f726c10db90 tx=0x7f7254005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.440+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 msgr2=0x7f726c19d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.440+0000 7f7271607640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 0x7f726c19d8b0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f725c00e9f0 tx=0x7f725c00eeb0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.440+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 shutdown_connections 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f72580a2230 0x7f72580a46f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f726c10bd50 0x7f726c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c1032a0 0x7f726c19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 --2- 172.21.15.116:0/2560611057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f726c102980 0x7f726c19d8b0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 >> 172.21.15.116:0/2560611057 conn(0x7f726c077710 msgr2=0x7f726c0fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 shutdown_connections 2024-03-24T09:33:22.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.441+0000 7f7271607640 1 -- 172.21.15.116:0/2560611057 wait complete. 2024-03-24T09:33:22.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T09:33:22.466 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:22 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:33:22.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:22 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T09:33:22.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 -- 172.21.15.116:0/1973503569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 msgr2=0x7f3e50106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 --2- 172.21.15.116:0/1973503569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50106a90 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009900 tx=0x7f3e4c02bcd0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 -- 172.21.15.116:0/1973503569 shutdown_connections 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 --2- 172.21.15.116:0/1973503569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e50100d40 0x7f3e5010d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 --2- 172.21.15.116:0/1973503569 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e50106fd0 0x7f3e50100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 --2- 172.21.15.116:0/1973503569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50106a90 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 -- 172.21.15.116:0/1973503569 >> 172.21.15.116:0/1973503569 conn(0x7f3e50077710 msgr2=0x7f3e50077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 -- 172.21.15.116:0/1973503569 shutdown_connections 2024-03-24T09:33:22.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.557+0000 7f3e57513640 1 -- 172.21.15.116:0/1973503569 wait complete. 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 Processor -- start 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 -- start start 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e50100d40 0x7f3e50198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e50106fd0 0x7f3e501a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50199870 con 0x7f3e501066b0 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501a4090 con 0x7f3e50106fd0 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e57513640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501a41d0 con 0x7f3e50100d40 2024-03-24T09:33:22.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35364/0 (socket says 172.21.15.116:35364) 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 -- 172.21.15.116:0/1231927749 learned_addr learned my addr 172.21.15.116:0/1231927749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e552af640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e50100d40 0x7f3e50198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e55ab0640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e50106fd0 0x7f3e501a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 -- 172.21.15.116:0/1231927749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e50100d40 msgr2=0x7f3e50198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e50100d40 0x7f3e50198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 -- 172.21.15.116:0/1231927749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e50106fd0 msgr2=0x7f3e501a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e50106fd0 0x7f3e501a3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 -- 172.21.15.116:0/1231927749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4c009560 con 0x7f3e501066b0 2024-03-24T09:33:22.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.558+0000 7f3e54aae640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50199010 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e9c0 tx=0x7f3e4000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:22.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.559+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400096c0 con 0x7f3e501066b0 2024-03-24T09:33:22.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.559+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f3e400054d0 con 0x7f3e501066b0 2024-03-24T09:33:22.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.559+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e501a4430 con 0x7f3e501066b0 2024-03-24T09:33:22.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.559+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40010430 con 0x7f3e501066b0 2024-03-24T09:33:22.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.559+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e501a4940 con 0x7f3e501066b0 2024-03-24T09:33:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.560+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e50068500 con 0x7f3e501066b0 2024-03-24T09:33:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.561+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f3e40004f90 con 0x7f3e501066b0 2024-03-24T09:33:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.561+0000 7f3e467fc640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e3c0a22b0 0x7f3e3c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.561+0000 7f3e552af640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e3c0a22b0 0x7f3e3c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.562+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f3e40014070 con 0x7f3e501066b0 2024-03-24T09:33:22.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.562+0000 7f3e552af640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e3c0a22b0 0x7f3e3c0a4770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009230 tx=0x7f3e4c005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:22.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.566+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f3e40096f90 con 0x7f3e501066b0 2024-03-24T09:33:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.785+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e50068e80 con 0x7f3e501066b0 2024-03-24T09:33:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.786+0000 7f3e467fc640 1 -- 172.21.15.116:0/1231927749 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11953 (secure 0 0 0) 0x7f3e4009a650 con 0x7f3e501066b0 2024-03-24T09:33:22.787 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:33:22.787 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14562,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/203590866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":203590866},{"type":"v1","addr":"172.21.15.116:6835","nonce":203590866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/1967493794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":1967493794},{"type":"v1","addr":"172.21.15.116:6837","nonce":1967493794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T09:19:38.431545+0000","modified":"2024-03-24T09:27:17.973201+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24515,"mds_1":24484,"mds_2":24451,"mds_3":24443,"mds_4":24422},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14631":{"gid":14631,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1877964932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1877964932},{"type":"v1","addr":"172.21.15.116:6839","nonce":1877964932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14649":{"gid":14649,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6841/891659164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":891659164},{"type":"v1","addr":"172.21.15.116:6841","nonce":891659164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24412":{"gid":24412,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6833/2011428167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":2011428167},{"type":"v1","addr":"172.21.15.143:6833","nonce":2011428167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24422":{"gid":24422,"name":"k","rank":4,"incarnation":26,"state":"up:active","state_seq":17,"addr":"172.21.15.117:6833/3849440293","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":3849440293},{"type":"v1","addr":"172.21.15.117:6833","nonce":3849440293}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":15,"addr":"172.21.15.117:6835/1932277434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1932277434},{"type":"v1","addr":"172.21.15.117:6835","nonce":1932277434}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24451":{"gid":24451,"name":"f","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.143:6835/1036797529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6834","nonce":1036797529},{"type":"v1","addr":"172.21.15.143:6835","nonce":1036797529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24469":{"gid":24469,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.143:6837/3300095096","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6836","nonce":3300095096},{"type":"v1","addr":"172.21.15.143:6837","nonce":3300095096}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/309438710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":309438710},{"type":"v1","addr":"172.21.15.117:6837","nonce":309438710}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.143:6839/3139487666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6838","nonce":3139487666},{"type":"v1","addr":"172.21.15.143:6839","nonce":3139487666}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3950675281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3950675281},{"type":"v1","addr":"172.21.15.117:6839","nonce":3950675281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24515,"qdb_cluster":[24443,24422,24451,24515,24484]},"id":1}]} 2024-03-24T09:33:22.787 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-03-24T09:33:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e3c0a22b0 msgr2=0x7f3e3c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e3c0a22b0 0x7f3e3c0a4770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009230 tx=0x7f3e4c005fd0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 msgr2=0x7f3e50199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50199010 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e9c0 tx=0x7f3e4000ee80 comp rx=0 tx=0).stop 2024-03-24T09:33:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 shutdown_connections 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e50106fd0 0x7f3e501a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e501066b0 0x7f3e50199010 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f3e3c0a22b0 0x7f3e3c0a4770 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 --2- 172.21.15.116:0/1231927749 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3e50100d40 0x7f3e50198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.788+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 >> 172.21.15.116:0/1231927749 conn(0x7f3e50077710 msgr2=0x7f3e500ff2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.789+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 shutdown_connections 2024-03-24T09:33:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.789+0000 7f3e57513640 1 -- 172.21.15.116:0/1231927749 wait complete. 2024-03-24T09:33:22.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T09:33:22.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 -- 172.21.15.116:0/1193185455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3241032a0 msgr2=0x7fb32410b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/1193185455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3241032a0 0x7fb32410b780 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fb314009900 tx=0x7fb31402bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 -- 172.21.15.116:0/1193185455 shutdown_connections 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/1193185455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb32410bd50 0x7fb32410e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/1193185455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3241032a0 0x7fb32410b780 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/1193185455 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb324102980 0x7fb324102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 -- 172.21.15.116:0/1193185455 >> 172.21.15.116:0/1193185455 conn(0x7fb324077710 msgr2=0x7fb324077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 -- 172.21.15.116:0/1193185455 shutdown_connections 2024-03-24T09:33:22.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.906+0000 7fb32c1f4640 1 -- 172.21.15.116:0/1193185455 wait complete. 2024-03-24T09:33:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 Processor -- start 2024-03-24T09:33:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 -- start start 2024-03-24T09:33:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb324102980 0x7fb324197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3241032a0 0x7fb3241982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 0x7fb32419c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb32419cc20 con 0x7fb32410bd50 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb32419cd90 con 0x7fb324102980 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32c1f4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fb32419cf50 con 0x7fb3241032a0 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32a791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 0x7fb32419c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32978f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3241032a0 0x7fb3241982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb329f90640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb324102980 0x7fb324197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32a791640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 0x7fb32419c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35374/0 (socket says 172.21.15.116:35374) 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32a791640 1 -- 172.21.15.116:0/3977233730 learned_addr learned my addr 172.21.15.116:0/3977233730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.907+0000 7fb32a791640 1 -- 172.21.15.116:0/3977233730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3241032a0 msgr2=0x7fb3241982a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32a791640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3241032a0 0x7fb3241982a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32a791640 1 -- 172.21.15.116:0/3977233730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb324102980 msgr2=0x7fb324197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32a791640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb324102980 0x7fb324197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32a791640 1 -- 172.21.15.116:0/3977233730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb314009560 con 0x7fb32410bd50 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32a791640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 0x7fb32419c620 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00eaa0 tx=0x7fb31c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c00cdb0 con 0x7fb32410bd50 2024-03-24T09:33:22.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fb31c005020 con 0x7fb32410bd50 2024-03-24T09:33:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32419d230 con 0x7fb32410bd50 2024-03-24T09:33:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb31c0116b0 con 0x7fb32410bd50 2024-03-24T09:33:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.908+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3241a4ab0 con 0x7fb32410bd50 2024-03-24T09:33:22.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.909+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb324068500 con 0x7fb32410bd50 2024-03-24T09:33:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.910+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fb31c005190 con 0x7fb32410bd50 2024-03-24T09:33:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.911+0000 7fb31affd640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3100a22b0 0x7fb3100a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.911+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7fb31c0ce8d0 con 0x7fb32410bd50 2024-03-24T09:33:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.911+0000 7fb329f90640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3100a22b0 0x7fb3100a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:22.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.911+0000 7fb329f90640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3100a22b0 0x7fb3100a4770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb320009bd0 tx=0x7fb320005f90 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:22.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:22.915+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fb31c097280 con 0x7fb32410bd50 2024-03-24T09:33:23.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.094+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb324106e40 con 0x7fb32410bd50 2024-03-24T09:33:23.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.095+0000 7fb31affd640 1 -- 172.21.15.116:0/3977233730 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25479 (secure 0 0 0) 0x7fb31c01f070 con 0x7fb32410bd50 2024-03-24T09:33:23.096 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-24T09:33:23.096 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":104,"fsid":"69c76602-e9be-11ee-95cd-87774f69a715","created":"2024-03-24T09:11:38.627995+0000","modified":"2024-03-24T09:33:14.052520+0000","last_up_change":"2024-03-24T09:18:48.395354+0000","last_in_change":"2024-03-24T09:18:27.725580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T09:15:00.179296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T09:19:36.455512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T09:19:37.676709+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-24T09:19:40.951352+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8b186fcf-17c6-409c-9ca6-7c8373830eea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6803","nonce":2942581569}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6805","nonce":2942581569}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6809","nonce":2942581569}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":2942581569},{"type":"v1","addr":"172.21.15.116:6807","nonce":2942581569}]},"public_addr":"172.21.15.116:6803/2942581569","cluster_addr":"172.21.15.116:6805/2942581569","heartbeat_back_addr":"172.21.15.116:6809/2942581569","heartbeat_front_addr":"172.21.15.116:6807/2942581569","state":["exists","up"]},{"osd":1,"uuid":"62462f7a-4125-4c73-beac-3aee728ab940","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6801","nonce":2635729600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6803","nonce":2635729600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6807","nonce":2635729600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2635729600},{"type":"v1","addr":"172.21.15.117:6805","nonce":2635729600}]},"public_addr":"172.21.15.117:6801/2635729600","cluster_addr":"172.21.15.117:6803/2635729600","heartbeat_back_addr":"172.21.15.117:6807/2635729600","heartbeat_front_addr":"172.21.15.117:6805/2635729600","state":["exists","up"]},{"osd":2,"uuid":"9f4d76a1-5218-4904-8b3e-0d782ada3628","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6801","nonce":541711081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6803","nonce":541711081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6807","nonce":541711081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":541711081},{"type":"v1","addr":"172.21.15.143:6805","nonce":541711081}]},"public_addr":"172.21.15.143:6801/541711081","cluster_addr":"172.21.15.143:6803/541711081","heartbeat_back_addr":"172.21.15.143:6807/541711081","heartbeat_front_addr":"172.21.15.143:6805/541711081","state":["exists","up"]},{"osd":3,"uuid":"3f2f25e8-51f7-4048-a704-c27813a60056","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6811","nonce":2981524673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6813","nonce":2981524673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6817","nonce":2981524673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2981524673},{"type":"v1","addr":"172.21.15.116:6815","nonce":2981524673}]},"public_addr":"172.21.15.116:6811/2981524673","cluster_addr":"172.21.15.116:6813/2981524673","heartbeat_back_addr":"172.21.15.116:6817/2981524673","heartbeat_front_addr":"172.21.15.116:6815/2981524673","state":["exists","up"]},{"osd":4,"uuid":"2a944b4a-1532-4713-a537-792acac8425c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6809","nonce":3606274523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6811","nonce":3606274523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6815","nonce":3606274523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":3606274523},{"type":"v1","addr":"172.21.15.117:6813","nonce":3606274523}]},"public_addr":"172.21.15.117:6809/3606274523","cluster_addr":"172.21.15.117:6811/3606274523","heartbeat_back_addr":"172.21.15.117:6815/3606274523","heartbeat_front_addr":"172.21.15.117:6813/3606274523","state":["exists","up"]},{"osd":5,"uuid":"aaadd9c9-b1bc-4ca0-bb20-760ee737abf9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6809","nonce":1015586525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6811","nonce":1015586525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6815","nonce":1015586525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":1015586525},{"type":"v1","addr":"172.21.15.143:6813","nonce":1015586525}]},"public_addr":"172.21.15.143:6809/1015586525","cluster_addr":"172.21.15.143:6811/1015586525","heartbeat_back_addr":"172.21.15.143:6815/1015586525","heartbeat_front_addr":"172.21.15.143:6813/1015586525","state":["exists","up"]},{"osd":6,"uuid":"09e3ad38-6286-4d42-9ddc-ccfa80ace379","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6819","nonce":254462190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6821","nonce":254462190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6825","nonce":254462190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":254462190},{"type":"v1","addr":"172.21.15.116:6823","nonce":254462190}]},"public_addr":"172.21.15.116:6819/254462190","cluster_addr":"172.21.15.116:6821/254462190","heartbeat_back_addr":"172.21.15.116:6825/254462190","heartbeat_front_addr":"172.21.15.116:6823/254462190","state":["exists","up"]},{"osd":7,"uuid":"fc83d789-61b0-4c26-93ff-c7ddcba3cc21","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6817","nonce":3523258939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6819","nonce":3523258939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6823","nonce":3523258939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3523258939},{"type":"v1","addr":"172.21.15.117:6821","nonce":3523258939}]},"public_addr":"172.21.15.117:6817/3523258939","cluster_addr":"172.21.15.117:6819/3523258939","heartbeat_back_addr":"172.21.15.117:6823/3523258939","heartbeat_front_addr":"172.21.15.117:6821/3523258939","state":["exists","up"]},{"osd":8,"uuid":"b9ce166d-03cd-41ec-a11e-f93e301cfb42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6817","nonce":3224917064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6819","nonce":3224917064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6823","nonce":3224917064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":3224917064},{"type":"v1","addr":"172.21.15.143:6821","nonce":3224917064}]},"public_addr":"172.21.15.143:6817/3224917064","cluster_addr":"172.21.15.143:6819/3224917064","heartbeat_back_addr":"172.21.15.143:6823/3224917064","heartbeat_front_addr":"172.21.15.143:6821/3224917064","state":["exists","up"]},{"osd":9,"uuid":"5c25c107-ff19-45fb-84e3-c994805faa36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6827","nonce":2856079084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6829","nonce":2856079084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6833","nonce":2856079084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":2856079084},{"type":"v1","addr":"172.21.15.116:6831","nonce":2856079084}]},"public_addr":"172.21.15.116:6827/2856079084","cluster_addr":"172.21.15.116:6829/2856079084","heartbeat_back_addr":"172.21.15.116:6833/2856079084","heartbeat_front_addr":"172.21.15.116:6831/2856079084","state":["exists","up"]},{"osd":10,"uuid":"33d2b62d-960d-458f-aa10-364ffb11ee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6825","nonce":3860527713}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6827","nonce":3860527713}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6831","nonce":3860527713}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3860527713},{"type":"v1","addr":"172.21.15.117:6829","nonce":3860527713}]},"public_addr":"172.21.15.117:6825/3860527713","cluster_addr":"172.21.15.117:6827/3860527713","heartbeat_back_addr":"172.21.15.117:6831/3860527713","heartbeat_front_addr":"172.21.15.117:6829/3860527713","state":["exists","up"]},{"osd":11,"uuid":"0397d0cd-1f13-475b-85e6-332bfd32d77d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6825","nonce":2884681125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6827","nonce":2884681125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6831","nonce":2884681125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":2884681125},{"type":"v1","addr":"172.21.15.143:6829","nonce":2884681125}]},"public_addr":"172.21.15.143:6825/2884681125","cluster_addr":"172.21.15.143:6827/2884681125","heartbeat_back_addr":"172.21.15.143:6831/2884681125","heartbeat_front_addr":"172.21.15.143:6829/2884681125","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:09.884569+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:33.934767+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:14:57.562799+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:21.910022+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:15:46.847713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:11.305681+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:16:36.538933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:02.198261+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:27.391960+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:17:53.460829+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:20.173296+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T09:18:46.260780+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":1,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.1b","mappings":[{"from":3,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6801/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/664840167":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/1503975320":"2024-03-25T09:22:05.255838+0000","172.21.15.116:6800/2363601294":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/4025940557":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/227139610":"2024-03-25T09:22:05.255838+0000","172.21.15.116:0/3305720445":"2024-03-25T09:12:33.839806+0000","172.21.15.116:0/3598626147":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6800/3871815735":"2024-03-25T09:12:33.839806+0000","172.21.15.116:6801/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3855909741":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/448304506":"2024-03-25T09:12:07.749841+0000","172.21.15.116:6800/285563997":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/3108228013":"2024-03-25T09:12:07.749841+0000","172.21.15.116:0/2688022316":"2024-03-25T09:12:33.839806+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T09:33:23.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3100a22b0 msgr2=0x7fb3100a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3100a22b0 0x7fb3100a4770 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb320009bd0 tx=0x7fb320005f90 comp rx=0 tx=0).stop 2024-03-24T09:33:23.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 msgr2=0x7fb32419c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 0x7fb32419c620 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fb31c00eaa0 tx=0x7fb31c00ef60 comp rx=0 tx=0).stop 2024-03-24T09:33:23.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 shutdown_connections 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32410bd50 0x7fb32419c620 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fb3100a22b0 0x7fb3100a4770 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3241032a0 0x7fb3241982a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 --2- 172.21.15.116:0/3977233730 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb324102980 0x7fb324197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 >> 172.21.15.116:0/3977233730 conn(0x7fb324077710 msgr2=0x7fb3240fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 shutdown_connections 2024-03-24T09:33:23.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.098+0000 7fb32c1f4640 1 -- 172.21.15.116:0/3977233730 wait complete. 2024-03-24T09:33:23.110 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T09:33:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.215+0000 7f00e273e640 1 -- 172.21.15.116:0/3651935080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 msgr2=0x7f00dc107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 --2- 172.21.15.116:0/3651935080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc107a70 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f00cc009930 tx=0x7f00cc02bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 -- 172.21.15.116:0/3651935080 shutdown_connections 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 --2- 172.21.15.116:0/3651935080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00dc0692b0 0x7f00dc10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 --2- 172.21.15.116:0/3651935080 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00dc0688f0 0x7f00dc068d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 --2- 172.21.15.116:0/3651935080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc107a70 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 -- 172.21.15.116:0/3651935080 >> 172.21.15.116:0/3651935080 conn(0x7f00dc077710 msgr2=0x7f00dc077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 -- 172.21.15.116:0/3651935080 shutdown_connections 2024-03-24T09:33:23.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 -- 172.21.15.116:0/3651935080 wait complete. 2024-03-24T09:33:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 Processor -- start 2024-03-24T09:33:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.216+0000 7f00e273e640 1 -- start start 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e273e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00dc0688f0 0x7f00dc19d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e273e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00dc0692b0 0x7f00dc19de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e273e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e273e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc1980e0 con 0x7f00dc107690 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e273e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc198250 con 0x7f00dc0692b0 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e273e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc198410 con 0x7f00dc0688f0 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35398/0 (socket says 172.21.15.116:35398) 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00dc0688f0 0x7f00dc19d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 -- 172.21.15.116:0/911960321 learned_addr learned my addr 172.21.15.116:0/911960321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00db7fe640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00dc0692b0 0x7f00dc19de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 -- 172.21.15.116:0/911960321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00dc0688f0 msgr2=0x7f00dc19d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00dc0688f0 0x7f00dc19d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 -- 172.21.15.116:0/911960321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00dc0692b0 msgr2=0x7f00dc19de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00dc0692b0 0x7f00dc19de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.217+0000 7f00e0cdb640 1 -- 172.21.15.116:0/911960321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d0009490 con 0x7f00dc107690 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.218+0000 7f00e0cdb640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc197a90 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f00d400eb10 tx=0x7f00d400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.218+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4004b20 con 0x7f00dc107690 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.218+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f00d4004c80 con 0x7f00dc107690 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.218+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00cc009560 con 0x7f00dc107690 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.218+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d4010690 con 0x7f00dc107690 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.218+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc1a47a0 con 0x7f00dc107690 2024-03-24T09:33:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.219+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00dc068500 con 0x7f00dc107690 2024-03-24T09:33:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.220+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f00d4005020 con 0x7f00dc107690 2024-03-24T09:33:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.221+0000 7f00d97fa640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f00c40a22b0 0x7f00c40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.221+0000 7f00dbfff640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f00c40a22b0 0x7f00c40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.221+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9554+0+0 (secure 0 0 0) 0x7f00d4014070 con 0x7f00dc107690 2024-03-24T09:33:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.221+0000 7f00dbfff640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f00c40a22b0 0x7f00c40a4770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f00cc005ab0 tx=0x7f00cc005fd0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:23.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.225+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f00d40971f0 con 0x7f00dc107690 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: pgmap v351: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2560611057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:33:23.318 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1231927749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:33:23.391 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: pgmap v351: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2560611057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:33:23.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1231927749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:33:23.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.458+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f00dc109bf0 con 0x7f00dc107690 2024-03-24T09:33:23.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.462+0000 7f00d97fa640 1 -- 172.21.15.116:0/911960321 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7f00d409a8b0 con 0x7f00dc107690 2024-03-24T09:33:23.462 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T09:33:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f00c40a22b0 msgr2=0x7f00c40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f00c40a22b0 0x7f00c40a4770 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f00cc005ab0 tx=0x7f00cc005fd0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 msgr2=0x7f00dc197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc197a90 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f00d400eb10 tx=0x7f00d400c2d0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 shutdown_connections 2024-03-24T09:33:23.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00dc107690 0x7f00dc197a90 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00dc0692b0 0x7f00dc19de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f00c40a22b0 0x7f00c40a4770 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 --2- 172.21.15.116:0/911960321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f00dc0688f0 0x7f00dc19d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 >> 172.21.15.116:0/911960321 conn(0x7f00dc077710 msgr2=0x7f00dc0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 shutdown_connections 2024-03-24T09:33:23.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.465+0000 7f00e273e640 1 -- 172.21.15.116:0/911960321 wait complete. 2024-03-24T09:33:23.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T09:33:23.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: pgmap v351: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:23.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.509 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.510 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.510 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.510 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:23.510 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2560611057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:33:23.510 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1231927749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T09:33:23.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 -- 172.21.15.116:0/2150126981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100ff0 msgr2=0x7f22fc10d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 --2- 172.21.15.116:0/2150126981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100ff0 0x7f22fc10d500 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009cd0 tx=0x7f22ec02c0f0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 -- 172.21.15.116:0/2150126981 shutdown_connections 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 --2- 172.21.15.116:0/2150126981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100ff0 0x7f22fc10d500 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 --2- 172.21.15.116:0/2150126981 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22fc100630 0x7f22fc100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 --2- 172.21.15.116:0/2150126981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22fc106640 0x7f22fc106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 -- 172.21.15.116:0/2150126981 >> 172.21.15.116:0/2150126981 conn(0x7f22fc077770 msgr2=0x7f22fc077b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 -- 172.21.15.116:0/2150126981 shutdown_connections 2024-03-24T09:33:23.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.600+0000 7f2301239640 1 -- 172.21.15.116:0/2150126981 wait complete. 2024-03-24T09:33:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 Processor -- start 2024-03-24T09:33:23.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 -- start start 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 0x7f22fc19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22fc100ff0 0x7f22fc19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22fc106640 0x7f22fc197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc198070 con 0x7f22fc100630 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1981e0 con 0x7f22fc100ff0 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f2301239640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f22fc1983a0 con 0x7f22fc106640 2024-03-24T09:33:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f22fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 0x7f22fc19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f22fad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 0x7f22fc19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35464/0 (socket says 172.21.15.116:35464) 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f22fad9d640 1 -- 172.21.15.116:0/2479132319 learned_addr learned my addr 172.21.15.116:0/2479132319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.601+0000 7f22fa59c640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22fc100ff0 0x7f22fc19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fb59e640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22fc106640 0x7f22fc197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fad9d640 1 -- 172.21.15.116:0/2479132319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22fc106640 msgr2=0x7f22fc197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fad9d640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22fc106640 0x7f22fc197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fad9d640 1 -- 172.21.15.116:0/2479132319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22fc100ff0 msgr2=0x7f22fc19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fad9d640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22fc100ff0 0x7f22fc19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fad9d640 1 -- 172.21.15.116:0/2479132319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ec009900 con 0x7f22fc100630 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22fad9d640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 0x7f22fc19d820 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f22f400e9f0 tx=0x7f22f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4009700 con 0x7f22fc100630 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f22f400ccc0 con 0x7f22fc100630 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22fc198680 con 0x7f22fc100630 2024-03-24T09:33:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f4018790 con 0x7f22fc100630 2024-03-24T09:33:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.602+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22fc1a48c0 con 0x7f22fc100630 2024-03-24T09:33:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.603+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22fc068950 con 0x7f22fc100630 2024-03-24T09:33:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.605+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f22f4010430 con 0x7f22fc100630 2024-03-24T09:33:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.605+0000 7f22e3fff640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22e40a2230 0x7f22e40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.605+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9896+0+0 (secure 0 0 0) 0x7f22f4014070 con 0x7f22fc100630 2024-03-24T09:33:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.605+0000 7f22fa59c640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22e40a2230 0x7f22e40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.606+0000 7f22fa59c640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22e40a2230 0x7f22e40a46f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f22fc06adc0 tx=0x7f22f000a2d0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.609+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f22f4097600 con 0x7f22fc100630 2024-03-24T09:33:23.774 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c[66551]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.774 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c[66551]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.775 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mds[66555]: -- 172.21.15.143:0/1469185423 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5557938d81a0 con 0x5557938aec00 2024-03-24T09:33:23.775 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c[66551]: starting mds.c at 2024-03-24T09:33:23.775 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f[66237]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.775 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f[66237]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.775 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mds[66241]: -- 172.21.15.143:0/2930886065 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56202ac0e1a0 con 0x56202abe4800 2024-03-24T09:33:23.776 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f[66237]: starting mds.f at 2024-03-24T09:33:23.776 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i[66859]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.776 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i[66859]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.776 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mds[66863]: -- 172.21.15.143:0/2993138470 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df8a6a61a0 con 0x55df8a67cc00 2024-03-24T09:33:23.776 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i[66859]: starting mds.i at 2024-03-24T09:33:23.776 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l[65912]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.777 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l[65912]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.777 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-mds[65916]: -- 172.21.15.143:0/4109071694 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559848a401a0 con 0x559848a17000 2024-03-24T09:33:23.777 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:33:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l[65912]: starting mds.l at 2024-03-24T09:33:23.816 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.816 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.816 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mds[65651]: -- 172.21.15.117:0/1822114531 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558dca4061a0 con 0x558dca3dc800 2024-03-24T09:33:23.817 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: starting mds.b at 2024-03-24T09:33:23.817 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e[65022]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.817 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e[65022]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.817 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mds[65026]: -- 172.21.15.117:0/3314219075 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f28f4be1a0 con 0x55f28f494c00 2024-03-24T09:33:23.817 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e[65022]: starting mds.e at 2024-03-24T09:33:23.818 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k[64704]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.818 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k[64704]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.818 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mds[64708]: -- 172.21.15.117:0/2622309305 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ca31eb01a0 con 0x55ca31e87000 2024-03-24T09:33:23.818 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k[64704]: starting mds.k at 2024-03-24T09:33:23.818 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h[65337]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.819 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h[65337]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.819 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-mds[65341]: -- 172.21.15.117:0/3665266330 <== mon.1 v2:172.21.15.143:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5585f89681a0 con 0x5585f893e800 2024-03-24T09:33:23.819 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:33:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h[65337]: starting mds.h at 2024-03-24T09:33:23.819 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a[32203]: 2024-03-24T09:33:23.459+0000 7f9d10393700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g[73640]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g[73640]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mds[73644]: -- 172.21.15.116:0/1558568014 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e8e686c1a0 con 0x55e8e6842800 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g[73640]: starting mds.g at 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a[74021]: ignoring --setuser ceph since I am not root 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a[74021]: ignoring --setgroup ceph since I am not root 2024-03-24T09:33:23.820 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-mds[74044]: -- 172.21.15.116:0/2680055970 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56203c0681a0 con 0x56203c03e800 2024-03-24T09:33:23.821 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:33:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a[74021]: starting mds.a at 2024-03-24T09:33:23.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:23.829+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f22fc110640 con 0x7f22fc100630 2024-03-24T09:33:24.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3977233730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:24.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:33:24.316 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/911960321' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/911960321' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-24T09:33:24.317 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:24 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2479132319' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:33:24.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3977233730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:24.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:33:24.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:33:24.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:24.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/911960321' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T09:33:24.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T09:33:24.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:33:24.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/911960321' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T09:33:24.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T09:33:24.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-24T09:33:24.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:24 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2479132319' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:33:24.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.467+0000 7f22e3fff640 1 -- 172.21.15.116:0/2479132319 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) ==== 104+0+0 (secure 0 0 0) 0x7f22f409acc0 con 0x7f22fc100630 2024-03-24T09:33:24.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22e40a2230 msgr2=0x7f22e40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22e40a2230 0x7f22e40a46f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f22fc06adc0 tx=0x7f22f000a2d0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 msgr2=0x7f22fc19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 0x7f22fc19d820 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f22f400e9f0 tx=0x7f22f400edf0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 shutdown_connections 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f22e40a2230 0x7f22e40a46f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f22fc106640 0x7f22fc197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22fc100ff0 0x7f22fc19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 --2- 172.21.15.116:0/2479132319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22fc100630 0x7f22fc19d820 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 >> 172.21.15.116:0/2479132319 conn(0x7f22fc077770 msgr2=0x7f22fc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:24.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 shutdown_connections 2024-03-24T09:33:24.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.469+0000 7f2301239640 1 -- 172.21.15.116:0/2479132319 wait complete. 2024-03-24T09:33:24.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T09:33:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3977233730' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T09:33:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T09:33:24.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/911960321' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/911960321' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-24T09:33:24.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:24 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2479132319' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T09:33:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 -- 172.21.15.116:0/48319590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90107700 msgr2=0x7fbb90107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 --2- 172.21.15.116:0/48319590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90107700 0x7fbb90107ae0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009900 tx=0x7fbb8402bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 -- 172.21.15.116:0/48319590 shutdown_connections 2024-03-24T09:33:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 --2- 172.21.15.116:0/48319590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb9006a6e0 0x7fbb9010e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 --2- 172.21.15.116:0/48319590 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90069d40 0x7fbb9006a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 --2- 172.21.15.116:0/48319590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90107700 0x7fbb90107ae0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 -- 172.21.15.116:0/48319590 >> 172.21.15.116:0/48319590 conn(0x7fbb900773e0 msgr2=0x7fbb900777f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:24.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 -- 172.21.15.116:0/48319590 shutdown_connections 2024-03-24T09:33:24.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.590+0000 7fbb95baf640 1 -- 172.21.15.116:0/48319590 wait complete. 2024-03-24T09:33:24.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 Processor -- start 2024-03-24T09:33:24.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 -- start start 2024-03-24T09:33:24.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 0x7fbb9019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb9006a6e0 0x7fbb9019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90107700 0x7fbb90197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90198100 con 0x7fbb90069d40 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90198270 con 0x7fbb9006a6e0 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb95baf640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fbb90198430 con 0x7fbb90107700 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 0x7fbb9019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 0x7fbb9019d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35478/0 (socket says 172.21.15.116:35478) 2024-03-24T09:33:24.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 -- 172.21.15.116:0/3371192871 learned_addr learned my addr 172.21.15.116:0/3371192871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 -- 172.21.15.116:0/3371192871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90107700 msgr2=0x7fbb90197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90107700 0x7fbb90197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 -- 172.21.15.116:0/3371192871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb9006a6e0 msgr2=0x7fbb9019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb9006a6e0 0x7fbb9019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 -- 172.21.15.116:0/3371192871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb84009560 con 0x7fbb90069d40 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.591+0000 7fbb8f7fe640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 0x7fbb9019d900 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fbb84005940 tx=0x7fbb840040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.592+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840044c0 con 0x7fbb90069d40 2024-03-24T09:33:24.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.592+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb901986b0 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.592+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fbb84003aa0 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.592+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84005540 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.592+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb901a49a0 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.593+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb90102dc0 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.594+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fbb84003c10 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.594+0000 7fbb8cff9640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb780a2230 0x7fbb780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.594+0000 7fbb8effd640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb780a2230 0x7fbb780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.594+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 9896+0+0 (secure 0 0 0) 0x7fbb840f0930 con 0x7fbb90069d40 2024-03-24T09:33:24.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.595+0000 7fbb8effd640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb780a2230 0x7fbb780a46f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbb80005fd0 tx=0x7fbb80005850 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:24.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.599+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fbb840b9170 con 0x7fbb90069d40 2024-03-24T09:33:24.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:24.772+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbb90068950 con 0x7fbb90069d40 2024-03-24T09:33:25.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.468+0000 7fbb8cff9640 1 -- 172.21.15.116:0/3371192871 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v106) ==== 181+0+0 (secure 0 0 0) 0x7fbb840bc830 con 0x7fbb90069d40 2024-03-24T09:33:25.469 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2024-03-24T09:33:25.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb780a2230 msgr2=0x7fbb780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:25.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb780a2230 0x7fbb780a46f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbb80005fd0 tx=0x7fbb80005850 comp rx=0 tx=0).stop 2024-03-24T09:33:25.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 msgr2=0x7fbb9019d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:25.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 0x7fbb9019d900 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fbb84005940 tx=0x7fbb840040c0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 shutdown_connections 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fbb780a2230 0x7fbb780a46f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90107700 0x7fbb90197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb9006a6e0 0x7fbb9019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 --2- 172.21.15.116:0/3371192871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb90069d40 0x7fbb9019d900 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.472+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 >> 172.21.15.116:0/3371192871 conn(0x7fbb900773e0 msgr2=0x7fbb900fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.473+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 shutdown_connections 2024-03-24T09:33:25.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.473+0000 7fbb95baf640 1 -- 172.21.15.116:0/3371192871 wait complete. 2024-03-24T09:33:25.487 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: pgmap v352: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:25.487 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T09:33:25.487 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: Cluster is now healthy 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2479132319' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6833/2673487641,v1:172.21.15.143:6835/2673487641] up:boot 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6832/4043435184,v1:172.21.15.117:6833/4043435184] up:boot 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6832/1285274634,v1:172.21.15.143:6834/1285274634] up:boot 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.116:6838/3485123626,v1:172.21.15.116:6839/3485123626] up:boot 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6836/2696563507,v1:172.21.15.143:6838/2696563507] up:boot 2024-03-24T09:33:25.488 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.143:6837/2565044956,v1:172.21.15.143:6839/2565044956] up:boot 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.116:6840/3967929428,v1:172.21.15.116:6841/3967929428] up:boot 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6835/1505236567,v1:172.21.15.117:6838/1505236567] up:boot 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6836/2858598510,v1:172.21.15.117:6839/2858598510] up:boot 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: mds.? [v2:172.21.15.117:6834/4268982991,v1:172.21.15.117:6837/4268982991] up:boot 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: fsmap 12 up:standby 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:33:25.489 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:33:25.490 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:25 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3371192871' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:25.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T09:33:25.524 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: pgmap v352: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: Cluster is now healthy 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2479132319' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6833/2673487641,v1:172.21.15.143:6835/2673487641] up:boot 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6832/4043435184,v1:172.21.15.117:6833/4043435184] up:boot 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6832/1285274634,v1:172.21.15.143:6834/1285274634] up:boot 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.116:6838/3485123626,v1:172.21.15.116:6839/3485123626] up:boot 2024-03-24T09:33:25.525 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6836/2696563507,v1:172.21.15.143:6838/2696563507] up:boot 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.143:6837/2565044956,v1:172.21.15.143:6839/2565044956] up:boot 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.116:6840/3967929428,v1:172.21.15.116:6841/3967929428] up:boot 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6835/1505236567,v1:172.21.15.117:6838/1505236567] up:boot 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6836/2858598510,v1:172.21.15.117:6839/2858598510] up:boot 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: mds.? [v2:172.21.15.117:6834/4268982991,v1:172.21.15.117:6837/4268982991] up:boot 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: fsmap 12 up:standby 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:33:25.526 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:33:25.527 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:25 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3371192871' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:25.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: pgmap v352: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: Cluster is now healthy 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2479132319' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6833/2673487641,v1:172.21.15.143:6835/2673487641] up:boot 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6832/4043435184,v1:172.21.15.117:6833/4043435184] up:boot 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6832/1285274634,v1:172.21.15.143:6834/1285274634] up:boot 2024-03-24T09:33:25.567 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.116:6838/3485123626,v1:172.21.15.116:6839/3485123626] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6836/2696563507,v1:172.21.15.143:6838/2696563507] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.143:6837/2565044956,v1:172.21.15.143:6839/2565044956] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.116:6840/3967929428,v1:172.21.15.116:6841/3967929428] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6835/1505236567,v1:172.21.15.117:6838/1505236567] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6836/2858598510,v1:172.21.15.117:6839/2858598510] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: mds.? [v2:172.21.15.117:6834/4268982991,v1:172.21.15.117:6837/4268982991] up:boot 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: fsmap 12 up:standby 2024-03-24T09:33:25.568 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T09:33:25.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='mgr.15543 172.21.15.116:0/1772115865' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T09:33:25.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:25 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3371192871' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 -- 172.21.15.116:0/1067755291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 msgr2=0x7fd400100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 --2- 172.21.15.116:0/1067755291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd400100a20 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009900 tx=0x7fd3f002bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 -- 172.21.15.116:0/1067755291 shutdown_connections 2024-03-24T09:33:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 --2- 172.21.15.116:0/1067755291 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400100f60 0x7fd40010d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 --2- 172.21.15.116:0/1067755291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd400100a20 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 --2- 172.21.15.116:0/1067755291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400106660 0x7fd400106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 -- 172.21.15.116:0/1067755291 >> 172.21.15.116:0/1067755291 conn(0x7fd400077710 msgr2=0x7fd400077b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 -- 172.21.15.116:0/1067755291 shutdown_connections 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 -- 172.21.15.116:0/1067755291 wait complete. 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 Processor -- start 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.602+0000 7fd4066eb640 1 -- start start 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd40019d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:25.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400100f60 0x7fd40019ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400106660 0x7fd400197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd4001980b0 con 0x7fd4001005a0 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd400198220 con 0x7fd400100f60 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7fd4001983e0 con 0x7fd400106660 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd40019d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd40019d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35506/0 (socket says 172.21.15.116:35506) 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 -- 172.21.15.116:0/2465041786 learned_addr learned my addr 172.21.15.116:0/2465041786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:25.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3ff7fe640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400100f60 0x7fd40019ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd404c88640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400106660 0x7fd400197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 -- 172.21.15.116:0/2465041786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400106660 msgr2=0x7fd400197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400106660 0x7fd400197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 -- 172.21.15.116:0/2465041786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400100f60 msgr2=0x7fd40019ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400100f60 0x7fd40019ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 -- 172.21.15.116:0/2465041786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3f0009560 con 0x7fd4001005a0 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd404c88640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400106660 0x7fd400197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fffff640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd40019d8b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800b4a0 tx=0x7fd3e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:25.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8010610 con 0x7fd4001005a0 2024-03-24T09:33:25.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7fd3e8010c20 con 0x7fd4001005a0 2024-03-24T09:33:25.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4001986c0 con 0x7fd4001005a0 2024-03-24T09:33:25.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8017d00 con 0x7fd4001005a0 2024-03-24T09:33:25.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.603+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4001a4950 con 0x7fd4001005a0 2024-03-24T09:33:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.604+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd400068500 con 0x7fd4001005a0 2024-03-24T09:33:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.605+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7fd3e8010770 con 0x7fd4001005a0 2024-03-24T09:33:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.606+0000 7fd3fd7fa640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3ec0a2230 0x7fd3ec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.606+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9400+0+0 (secure 0 0 0) 0x7fd3e80cd9e0 con 0x7fd4001005a0 2024-03-24T09:33:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.606+0000 7fd3ff7fe640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3ec0a2230 0x7fd3ec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:25.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.606+0000 7fd3ff7fe640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3ec0a2230 0x7fd3ec0a46f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0006010 tx=0x7fd3f002c6c0 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:25.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.610+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7fd3e8096410 con 0x7fd4001005a0 2024-03-24T09:33:25.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:25.785+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd400198f90 con 0x7fd4001005a0 2024-03-24T09:33:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.470+0000 7fd3fd7fa640 1 -- 172.21.15.116:0/2465041786 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v107) ==== 169+0+0 (secure 0 0 0) 0x7fd400198f90 con 0x7fd4001005a0 2024-03-24T09:33:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2024-03-24T09:33:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3ec0a2230 msgr2=0x7fd3ec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3ec0a2230 0x7fd3ec0a46f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0006010 tx=0x7fd3f002c6c0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 msgr2=0x7fd40019d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd40019d8b0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800b4a0 tx=0x7fd3e800b960 comp rx=0 tx=0).stop 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 shutdown_connections 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7fd3ec0a2230 0x7fd3ec0a46f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd400106660 0x7fd400197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd400100f60 0x7fd40019ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 --2- 172.21.15.116:0/2465041786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4001005a0 0x7fd40019d8b0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 >> 172.21.15.116:0/2465041786 conn(0x7fd400077710 msgr2=0x7fd4000ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 shutdown_connections 2024-03-24T09:33:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.473+0000 7fd4066eb640 1 -- 172.21.15.116:0/2465041786 wait complete. 2024-03-24T09:33:26.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-24T09:33:26.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 -- 172.21.15.116:0/2537227630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8069d80 msgr2=0x7f86a806a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 --2- 172.21.15.116:0/2537227630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8069d80 0x7f86a806a1e0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f8690009900 tx=0x7f869002bc30 comp rx=0 tx=0).stop 2024-03-24T09:33:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 -- 172.21.15.116:0/2537227630 shutdown_connections 2024-03-24T09:33:26.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 --2- 172.21.15.116:0/2537227630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86a806a720 0x7f86a810e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 --2- 172.21.15.116:0/2537227630 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8069d80 0x7f86a806a1e0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 --2- 172.21.15.116:0/2537227630 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a81076a0 0x7f86a8107a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 -- 172.21.15.116:0/2537227630 >> 172.21.15.116:0/2537227630 conn(0x7f86a80773b0 msgr2=0x7f86a80777c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 -- 172.21.15.116:0/2537227630 shutdown_connections 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 -- 172.21.15.116:0/2537227630 wait complete. 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 Processor -- start 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.601+0000 7f86ace74640 1 -- start start 2024-03-24T09:33:26.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a8069d80 0x7f86a819d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86a806a720 0x7f86a819de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 0x7f86a8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86a81980d0 con 0x7f86a81076a0 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f86a8198240 con 0x7f86a806a720 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 -- 0x7f86a8198400 con 0x7f86a8069d80 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 0x7f86a8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:26.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 0x7f86a8197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59572/0 (socket says 172.21.15.116:59572) 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 -- 172.21.15.116:0/1773540748 learned_addr learned my addr 172.21.15.116:0/1773540748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f869ffff640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86a806a720 0x7f86a819de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a659c640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a8069d80 0x7f86a819d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 -- 172.21.15.116:0/1773540748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a8069d80 msgr2=0x7f86a819d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a8069d80 0x7f86a819d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 -- 172.21.15.116:0/1773540748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86a806a720 msgr2=0x7f86a819de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86a806a720 0x7f86a819de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:26.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 -- 172.21.15.116:0/1773540748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8690009560 con 0x7f86a81076a0 2024-03-24T09:33:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86a6d9d640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 0x7f86a8197a80 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f86a000cad0 tx=0x7f86a000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a000ae20 con 0x7f86a81076a0 2024-03-24T09:33:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(26 keys) ==== 1102+0+0 (secure 0 0 0) 0x7f86a0005020 con 0x7f86a81076a0 2024-03-24T09:33:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a81986e0 con 0x7f86a81076a0 2024-03-24T09:33:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a000f690 con 0x7f86a81076a0 2024-03-24T09:33:26.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.602+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a81a4970 con 0x7f86a81076a0 2024-03-24T09:33:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.604+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a8102dc0 con 0x7f86a81076a0 2024-03-24T09:33:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.605+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 26) ==== 136888+0+0 (secure 0 0 0) 0x7f86a0005190 con 0x7f86a81076a0 2024-03-24T09:33:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.605+0000 7f869f7fe640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86940a2230 0x7f86940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T09:33:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.605+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 8754+0+0 (secure 0 0 0) 0x7f86a00ce170 con 0x7f86a81076a0 2024-03-24T09:33:26.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.605+0000 7f86a659c640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86940a2230 0x7f86940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T09:33:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.606+0000 7f86a659c640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86940a2230 0x7f86940a46f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f86a8198bc0 tx=0x7f869800b390 comp rx=0 tx=0).ready entity=mgr.15543 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T09:33:26.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.609+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+184932 (secure 0 0 0) 0x7f86a0096dc0 con 0x7f86a81076a0 2024-03-24T09:33:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:26 smithi143 ceph-mon[35965]: pgmap v354: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:26 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/3371192871' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:33:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:26 smithi143 ceph-mon[35965]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T09:33:26.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:26 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2465041786' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:26.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:26.798+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f86a8109bd0 con 0x7f86a81076a0 2024-03-24T09:33:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:26 smithi117 ceph-mon[35788]: pgmap v354: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:26 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/3371192871' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:33:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:26 smithi117 ceph-mon[35788]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T09:33:26.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:26 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2465041786' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:26.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:26 smithi116 ceph-mon[32207]: pgmap v354: 129 pgs: 129 active+clean; 150 MiB data, 909 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-03-24T09:33:26.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:26 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/3371192871' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:33:26.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:26 smithi116 ceph-mon[32207]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T09:33:26.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:26 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2465041786' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:27.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.486+0000 7f869f7fe640 1 -- 172.21.15.116:0/1773540748 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v108) ==== 178+0+0 (secure 0 0 0) 0x7f86a8109bd0 con 0x7f86a81076a0 2024-03-24T09:33:27.487 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data_ec' removed 2024-03-24T09:33:27.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86940a2230 msgr2=0x7f86940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86940a2230 0x7f86940a46f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f86a8198bc0 tx=0x7f869800b390 comp rx=0 tx=0).stop 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 msgr2=0x7f86a8197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 0x7f86a8197a80 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f86a000cad0 tx=0x7f86a000cf90 comp rx=0 tx=0).stop 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 shutdown_connections 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a81076a0 0x7f86a8197a80 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86a806a720 0x7f86a819de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:27.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.116:6800/2615204634,v1:172.21.15.116:6801/2615204634] conn(0x7f86940a2230 0x7f86940a46f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 --2- 172.21.15.116:0/1773540748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f86a8069d80 0x7f86a819d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T09:33:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 >> 172.21.15.116:0/1773540748 conn(0x7f86a80773b0 msgr2=0x7f86a80ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T09:33:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 shutdown_connections 2024-03-24T09:33:27.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-24T09:33:27.489+0000 7f86ace74640 1 -- 172.21.15.116:0/1773540748 wait complete. 2024-03-24T09:33:27.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:33:27.533 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:33:27.568 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:33:27.603 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T09:33:27.604 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T09:33:27.619 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T09:33:27.639 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T09:33:27.666 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T09:33:27.666 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T09:33:27.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a 2024-03-24T09:33:27.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:27 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/2465041786' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:33:27.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:27 smithi143 ceph-mon[35965]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T09:33:27.774 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:27 smithi143 ceph-mon[35965]: from='client.? 172.21.15.116:0/1773540748' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:27 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/2465041786' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:33:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:27 smithi117 ceph-mon[35788]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T09:33:27.816 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:27 smithi117 ceph-mon[35788]: from='client.? 172.21.15.116:0/1773540748' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:27.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:27 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/2465041786' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T09:33:27.817 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:27 smithi116 ceph-mon[32207]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T09:33:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:27 smithi116 ceph-mon[32207]: from='client.? 172.21.15.116:0/1773540748' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T09:33:27.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:27 smithi116 systemd[1]: Stopping Ceph mon.a for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:28.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a[32203]: 2024-03-24T09:33:28.201+0000 7f9d15b9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:33:28.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a[32203]: 2024-03-24T09:33:28.201+0000 7f9d15b9e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T09:33:28.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:28 smithi116 podman[110786]: 2024-03-24 09:33:28.252483296 +0000 UTC m=+0.393320627 container died b745d26f56b3945da7cc8d897d939d3c6b40d94bfcd6ca7e3777f6fd6f9e1a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T09:33:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:28 smithi116 podman[110786]: 2024-03-24 09:33:28.651131355 +0000 UTC m=+0.791968682 container cleanup b745d26f56b3945da7cc8d897d939d3c6b40d94bfcd6ca7e3777f6fd6f9e1a3d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T09:33:28.818 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 24 09:33:28 smithi116 bash[110786]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-a 2024-03-24T09:33:29.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T09:33:29.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:29.120 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T09:33:29.121 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T09:33:29.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.b 2024-03-24T09:33:29.566 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:29 smithi117 systemd[1]: Stopping Ceph mon.b for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:30.066 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-b[35783]: 2024-03-24T09:33:29.664+0000 7f83c79af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:33:30.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-b[35783]: 2024-03-24T09:33:29.664+0000 7f83c79af700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T09:33:30.067 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:29 smithi117 podman[71318]: 2024-03-24 09:33:29.828333443 +0000 UTC m=+0.514347153 container died b766b95c86402f475168b28066cf4f29217bb8802af015ac2425d7c87f28be20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-b, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T09:33:30.482 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:30 smithi117 podman[71318]: 2024-03-24 09:33:30.231362164 +0000 UTC m=+0.917375871 container cleanup b766b95c86402f475168b28066cf4f29217bb8802af015ac2425d7c87f28be20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T09:33:30.482 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 24 09:33:30 smithi117 bash[71318]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-b 2024-03-24T09:33:30.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T09:33:30.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:30.688 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T09:33:30.689 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T09:33:30.689 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.c 2024-03-24T09:33:31.023 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:30 smithi143 systemd[1]: Stopping Ceph mon.c for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-c[35961]: 2024-03-24T09:33:31.022+0000 7f90891e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:33:31.274 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-c[35961]: 2024-03-24T09:33:31.022+0000 7f90891e0700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T09:33:31.275 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:31 smithi143 podman[72548]: 2024-03-24 09:33:31.064025939 +0000 UTC m=+0.174793005 container died f7132b5183bb06a2264f6978045d30e7551202a3272a9a9c7806ff3d057180f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T09:33:31.724 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:31 smithi143 podman[72548]: 2024-03-24 09:33:31.423529316 +0000 UTC m=+0.534296373 container cleanup f7132b5183bb06a2264f6978045d30e7551202a3272a9a9c7806ff3d057180f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T09:33:31.724 INFO:journalctl@ceph.mon.c.smithi143.stdout:Mar 24 09:33:31 smithi143 bash[72548]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mon-c 2024-03-24T09:33:31.892 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T09:33:31.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:31.937 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T09:33:31.938 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T09:33:31.938 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x 2024-03-24T09:33:32.274 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:33:31 smithi116 systemd[1]: Stopping Ceph mgr.x for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:32.567 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:33:32 smithi116 podman[110907]: 2024-03-24 09:33:32.279139306 +0000 UTC m=+0.149392552 container died 00e781c18f7d81f63ed6eb2638a231e626d4fc12f3cc5786d348bc8282f89d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:33:32.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:33:32 smithi116 podman[110907]: 2024-03-24 09:33:32.697673407 +0000 UTC m=+0.567926665 container cleanup 00e781c18f7d81f63ed6eb2638a231e626d4fc12f3cc5786d348bc8282f89d51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:21b3bd672163092b4e542f772797fcba02d3d4de, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T09:33:32.915 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 24 09:33:32 smithi116 bash[110907]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-x 2024-03-24T09:33:33.114 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T09:33:33.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:33.152 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T09:33:33.153 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T09:33:33.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.y 2024-03-24T09:33:33.498 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:33:33 smithi117 systemd[1]: Stopping Ceph mgr.y for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:33.794 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:33:33 smithi117 podman[71440]: 2024-03-24 09:33:33.498071484 +0000 UTC m=+0.152915069 container died fca04d5a098737f4946b62270201607171e324112bad0e505f821ffbe4d902f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T09:33:34.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:33:33 smithi117 podman[71440]: 2024-03-24 09:33:33.89459734 +0000 UTC m=+0.549440923 container cleanup fca04d5a098737f4946b62270201607171e324112bad0e505f821ffbe4d902f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:33:34.066 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 24 09:33:33 smithi117 bash[71440]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-y 2024-03-24T09:33:34.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T09:33:34.361 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:34.361 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T09:33:34.362 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T09:33:34.362 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.z 2024-03-24T09:33:34.690 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:33:34 smithi143 systemd[1]: Stopping Ceph mgr.z for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:34.978 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:33:34 smithi143 podman[72667]: 2024-03-24 09:33:34.690462607 +0000 UTC m=+0.136706359 container died 83938f718d47590133583e1ed28e3c1a78d22b26c0adb874e8f6d73d758dbdfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:33:35.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:33:35 smithi143 podman[72667]: 2024-03-24 09:33:35.078115897 +0000 UTC m=+0.524359652 container cleanup 83938f718d47590133583e1ed28e3c1a78d22b26c0adb874e8f6d73d758dbdfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2) 2024-03-24T09:33:35.274 INFO:journalctl@ceph.mgr.z.smithi143.stdout:Mar 24 09:33:35 smithi143 bash[72667]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mgr-z 2024-03-24T09:33:35.469 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T09:33:35.506 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:35.506 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T09:33:35.506 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T09:33:35.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.0 2024-03-24T09:33:35.817 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:35 smithi116 systemd[1]: Stopping Ceph osd.0 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:36.318 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0[47373]: 2024-03-24T09:33:35.817+0000 7fd711898700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:33:36.318 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0[47373]: 2024-03-24T09:33:35.817+0000 7fd711898700 -1 osd.0 108 *** Got signal Terminated *** 2024-03-24T09:33:36.318 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0[47373]: 2024-03-24T09:33:35.817+0000 7fd711898700 -1 osd.0 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:33:41.488 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:41 smithi116 podman[111043]: 2024-03-24 09:33:41.196583195 +0000 UTC m=+5.508441649 container died 13e96d2a244243a5522afb7741a46f6a936de3b5c9c933b8121cafdf1351e986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:33:41.818 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:41 smithi116 podman[111043]: 2024-03-24 09:33:41.58628655 +0000 UTC m=+5.898145002 container cleanup 13e96d2a244243a5522afb7741a46f6a936de3b5c9c933b8121cafdf1351e986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:33:41.818 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:41 smithi116 bash[111043]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0 2024-03-24T09:33:42.192 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:41 smithi116 podman[111062]: 2024-03-24 09:33:41.862805243 +0000 UTC m=+0.669522013 container remove 13e96d2a244243a5522afb7741a46f6a936de3b5c9c933b8121cafdf1351e986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T09:33:42.449 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:42 smithi116 podman[111153]: 2024-03-24 09:33:42.094974173 +0000 UTC m=+0.017379793 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:33:42.449 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:42 smithi116 podman[111153]: 2024-03-24 09:33:42.19711625 +0000 UTC m=+0.119521883 container create 48dd0140961d865dd626251e691dcf02ec094d2c0a2061b03fb52eb3383e2590 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T09:33:42.741 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:42 smithi116 podman[111153]: 2024-03-24 09:33:42.449791301 +0000 UTC m=+0.372196905 container init 48dd0140961d865dd626251e691dcf02ec094d2c0a2061b03fb52eb3383e2590 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T09:33:42.741 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:42 smithi116 podman[111153]: 2024-03-24 09:33:42.45321533 +0000 UTC m=+0.375620922 container start 48dd0140961d865dd626251e691dcf02ec094d2c0a2061b03fb52eb3383e2590 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:33:42.741 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:42 smithi116 podman[111153]: 2024-03-24 09:33:42.558332269 +0000 UTC m=+0.480737876 container attach 48dd0140961d865dd626251e691dcf02ec094d2c0a2061b03fb52eb3383e2590 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T09:33:43.044 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 24 09:33:42 smithi116 podman[111153]: 2024-03-24 09:33:42.741731741 +0000 UTC m=+0.664137341 container died 48dd0140961d865dd626251e691dcf02ec094d2c0a2061b03fb52eb3383e2590 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:33:43.373 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T09:33:43.415 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:43.415 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T09:33:43.415 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T09:33:43.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.1 2024-03-24T09:33:43.731 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:43 smithi117 systemd[1]: Stopping Ceph osd.1 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:44.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1[40401]: 2024-03-24T09:33:43.731+0000 7feccb574700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:33:44.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1[40401]: 2024-03-24T09:33:43.731+0000 7feccb574700 -1 osd.1 108 *** Got signal Terminated *** 2024-03-24T09:33:44.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1[40401]: 2024-03-24T09:33:43.731+0000 7feccb574700 -1 osd.1 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:33:49.423 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:49 smithi117 podman[71565]: 2024-03-24 09:33:49.132796766 +0000 UTC m=+5.526316112 container died b54f015d43e1d4af5223bdda8459df9306314fc782f637acd41d028346f3489a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T09:33:49.766 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:49 smithi117 podman[71565]: 2024-03-24 09:33:49.531654125 +0000 UTC m=+5.925173515 container cleanup b54f015d43e1d4af5223bdda8459df9306314fc782f637acd41d028346f3489a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T09:33:49.766 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:49 smithi117 bash[71565]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1 2024-03-24T09:33:50.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:49 smithi117 podman[71583]: 2024-03-24 09:33:49.766266243 +0000 UTC m=+0.635984541 container remove b54f015d43e1d4af5223bdda8459df9306314fc782f637acd41d028346f3489a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T09:33:50.488 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:50 smithi117 podman[71677]: 2024-03-24 09:33:50.091004198 +0000 UTC m=+0.017702494 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:33:50.488 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:50 smithi117 podman[71677]: 2024-03-24 09:33:50.242242656 +0000 UTC m=+0.168940950 container create dd85461a43f1e33098c4c745635f59658417baabb480f687bb54a8bbd556c0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:33:50.488 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:50 smithi117 podman[71677]: 2024-03-24 09:33:50.487812314 +0000 UTC m=+0.414510606 container init dd85461a43f1e33098c4c745635f59658417baabb480f687bb54a8bbd556c0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:33:50.784 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:50 smithi117 podman[71677]: 2024-03-24 09:33:50.491654984 +0000 UTC m=+0.418353286 container start dd85461a43f1e33098c4c745635f59658417baabb480f687bb54a8bbd556c0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T09:33:50.784 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:50 smithi117 podman[71677]: 2024-03-24 09:33:50.620148945 +0000 UTC m=+0.546847239 container attach dd85461a43f1e33098c4c745635f59658417baabb480f687bb54a8bbd556c0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:33:51.066 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 24 09:33:50 smithi117 podman[71677]: 2024-03-24 09:33:50.78464292 +0000 UTC m=+0.711341214 container died dd85461a43f1e33098c4c745635f59658417baabb480f687bb54a8bbd556c0c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T09:33:51.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T09:33:51.459 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:51.459 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T09:33:51.459 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T09:33:51.459 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.2 2024-03-24T09:33:51.765 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:51 smithi143 systemd[1]: Stopping Ceph osd.2 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:33:52.024 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2[41138]: 2024-03-24T09:33:51.764+0000 7f52223fc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:33:52.024 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2[41138]: 2024-03-24T09:33:51.764+0000 7f52223fc700 -1 osd.2 108 *** Got signal Terminated *** 2024-03-24T09:33:52.025 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2[41138]: 2024-03-24T09:33:51.764+0000 7f52223fc700 -1 osd.2 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:33:57.424 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:57 smithi143 podman[72791]: 2024-03-24 09:33:57.117551523 +0000 UTC m=+5.465198995 container died bb325f71c02099d63a62e3b07b823cb324bfdd1435579193fe3f36db7d949cbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T09:33:57.774 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:57 smithi143 podman[72791]: 2024-03-24 09:33:57.531984968 +0000 UTC m=+5.879632448 container cleanup bb325f71c02099d63a62e3b07b823cb324bfdd1435579193fe3f36db7d949cbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:33:57.774 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:57 smithi143 bash[72791]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2 2024-03-24T09:33:58.141 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:57 smithi143 podman[72820]: 2024-03-24 09:33:57.783280933 +0000 UTC m=+0.665796255 container remove bb325f71c02099d63a62e3b07b823cb324bfdd1435579193fe3f36db7d949cbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T09:33:58.428 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:58 smithi143 podman[72914]: 2024-03-24 09:33:58.043587226 +0000 UTC m=+0.017967000 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:33:58.429 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:58 smithi143 podman[72914]: 2024-03-24 09:33:58.159108529 +0000 UTC m=+0.133488287 container create cfe50091f4242b83c3313ca335db109b7b2ddafe2aa0a2fa712277afb936a2f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T09:33:58.429 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:58 smithi143 podman[72914]: 2024-03-24 09:33:58.428272222 +0000 UTC m=+0.402651980 container init cfe50091f4242b83c3313ca335db109b7b2ddafe2aa0a2fa712277afb936a2f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:33:58.724 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:58 smithi143 podman[72914]: 2024-03-24 09:33:58.432181788 +0000 UTC m=+0.406561546 container start cfe50091f4242b83c3313ca335db109b7b2ddafe2aa0a2fa712277afb936a2f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T09:33:58.724 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:58 smithi143 podman[72914]: 2024-03-24 09:33:58.52001848 +0000 UTC m=+0.494398252 container attach cfe50091f4242b83c3313ca335db109b7b2ddafe2aa0a2fa712277afb936a2f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:33:59.024 INFO:journalctl@ceph.osd.2.smithi143.stdout:Mar 24 09:33:58 smithi143 podman[72914]: 2024-03-24 09:33:58.725053724 +0000 UTC m=+0.699433486 container died cfe50091f4242b83c3313ca335db109b7b2ddafe2aa0a2fa712277afb936a2f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T09:33:59.411 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T09:33:59.453 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:33:59.453 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T09:33:59.453 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T09:33:59.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.3 2024-03-24T09:33:59.756 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:33:59 smithi116 systemd[1]: Stopping Ceph osd.3 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:00.067 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:33:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:33:59.755+0000 7fe30c684700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:00.068 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:33:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:33:59.755+0000 7fe30c684700 -1 osd.3 108 *** Got signal Terminated *** 2024-03-24T09:34:00.068 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:33:59 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:33:59.755+0000 7fe30c684700 -1 osd.3 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:02.817 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:02 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:34:02.397+0000 7fe304c70700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.468995+0000 front 2024-03-24T09:33:37.469028+0000 (oldest deadline 2024-03-24T09:34:02.169191+0000) 2024-03-24T09:34:03.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:02 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:02.737+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:03.719 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:34:03.353+0000 7fe304c70700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.468995+0000 front 2024-03-24T09:33:37.469028+0000 (oldest deadline 2024-03-24T09:34:02.169191+0000) 2024-03-24T09:34:03.719 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:03 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:03.561+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:04.024 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:03.546+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:04.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:03 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:03.766+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:04.734 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3[53887]: 2024-03-24T09:34:04.334+0000 7fe304c70700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.468995+0000 front 2024-03-24T09:33:37.469028+0000 (oldest deadline 2024-03-24T09:34:02.169191+0000) 2024-03-24T09:34:04.735 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:04 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:04.580+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:04.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:04.504+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:05.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:04 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:04.726+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:05.548 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:05 smithi116 podman[111260]: 2024-03-24 09:34:05.257644495 +0000 UTC m=+5.609714143 container died 83fbf11889e8d4c899075ee95f86c6abc15e99270714f0580e51d3c3c39d81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:34:05.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:05.481+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:05.818 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:05 smithi116 podman[111260]: 2024-03-24 09:34:05.639408761 +0000 UTC m=+5.991478418 container cleanup 83fbf11889e8d4c899075ee95f86c6abc15e99270714f0580e51d3c3c39d81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T09:34:05.818 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:05 smithi116 bash[111260]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3 2024-03-24T09:34:05.818 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:05 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:05.592+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:06.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:05 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:05.766+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:06.227 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:05 smithi116 podman[111286]: 2024-03-24 09:34:05.907986021 +0000 UTC m=+0.646305857 container remove 83fbf11889e8d4c899075ee95f86c6abc15e99270714f0580e51d3c3c39d81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T09:34:06.561 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:06 smithi116 podman[111383]: 2024-03-24 09:34:06.130421688 +0000 UTC m=+0.019006207 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:06.561 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:06 smithi116 podman[111383]: 2024-03-24 09:34:06.233974198 +0000 UTC m=+0.122558694 container create e7814048a19d2070ba46214bf274ade1bbacbd07fc5a6620ed5ccf3e969d8325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T09:34:06.561 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:06 smithi116 podman[111383]: 2024-03-24 09:34:06.462071996 +0000 UTC m=+0.350656511 container init e7814048a19d2070ba46214bf274ade1bbacbd07fc5a6620ed5ccf3e969d8325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:34:06.562 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:06 smithi116 podman[111383]: 2024-03-24 09:34:06.465442911 +0000 UTC m=+0.354027407 container start e7814048a19d2070ba46214bf274ade1bbacbd07fc5a6620ed5ccf3e969d8325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T09:34:06.562 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:06 smithi116 podman[111383]: 2024-03-24 09:34:06.561735342 +0000 UTC m=+0.450319839 container attach e7814048a19d2070ba46214bf274ade1bbacbd07fc5a6620ed5ccf3e969d8325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T09:34:06.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:06.513+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:06.818 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 24 09:34:06 smithi116 podman[111383]: 2024-03-24 09:34:06.757499089 +0000 UTC m=+0.646083589 container died e7814048a19d2070ba46214bf274ade1bbacbd07fc5a6620ed5ccf3e969d8325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-24T09:34:06.818 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:06 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:06.576+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:07.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:06 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:06.778+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:07.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:06.791+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:07.335 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T09:34:07.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:07.376 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T09:34:07.376 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T09:34:07.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.4 2024-03-24T09:34:07.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:07 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:07.563+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:07.700 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:07 smithi117 systemd[1]: Stopping Ceph osd.4 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:07.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:07.482+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:08.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:07 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:07.748+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:08.066 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:07 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4[47090]: 2024-03-24T09:34:07.699+0000 7f27f08dd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:08.067 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:07 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4[47090]: 2024-03-24T09:34:07.699+0000 7f27f08dd700 -1 osd.4 108 *** Got signal Terminated *** 2024-03-24T09:34:08.067 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:07 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4[47090]: 2024-03-24T09:34:07.699+0000 7f27f08dd700 -1 osd.4 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:08.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:07.780+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:08.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:08 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:08.504+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:08.775 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:08 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:08.750+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:08.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:08 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:08.548+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:09.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:08 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:08.779+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:09.771 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:09 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:09.475+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:09.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:09 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:09.505+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:10.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:09 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:09.770+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:10.316 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:09 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:09.824+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:10.518 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:10 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:10.035+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:10.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:10 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:10.517+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:10.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:10 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:10.551+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:10.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:10 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:10.782+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:10.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:10 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:10.782+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:10.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:10 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:10.540+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:10.818 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:10 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:10.540+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:11.037 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:10 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:10.778+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:11.510 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:11 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:11.037+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:11.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:11 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:11.508+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:11.816 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:11 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4[47090]: 2024-03-24T09:34:11.766+0000 7f27e8ec9700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:44.995044+0000 front 2024-03-24T09:33:44.995022+0000 (oldest deadline 2024-03-24T09:34:10.895189+0000) 2024-03-24T09:34:11.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:11 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:11.530+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:11.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:11 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:11.745+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:11.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:11 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:11.745+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:11.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:11 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:11.537+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:11.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:11 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:11.537+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:12.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:12 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:12.052+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:12.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:11 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:11.825+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:12.771 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:12 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:12.518+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:12.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:12 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:12.479+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:12.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:12 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:12.479+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:12.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:12.565+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:12.818 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:12 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:12.565+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:13.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:12 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:12.770+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:13.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:12 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:12.770+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:13.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:12 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:12.796+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:13.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:13 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:13.033+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:13.451 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:13 smithi117 podman[71795]: 2024-03-24 09:34:13.175665893 +0000 UTC m=+5.602960683 container died b5ba8fe69a046218c1ed8e623a40b7f4d3f3be1878c37380804a017e0b9127bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T09:34:13.758 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:13 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:13.450+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:13.759 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:13 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:13.450+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:13.798 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:13 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:13.529+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:13.799 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:13 smithi117 podman[71795]: 2024-03-24 09:34:13.55833216 +0000 UTC m=+5.985626958 container cleanup b5ba8fe69a046218c1ed8e623a40b7f4d3f3be1878c37380804a017e0b9127bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:34:13.799 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:13 smithi117 bash[71795]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4 2024-03-24T09:34:14.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:13 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:13.757+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:14.066 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:13 smithi117 podman[71813]: 2024-03-24 09:34:13.826452891 +0000 UTC m=+0.643509478 container remove b5ba8fe69a046218c1ed8e623a40b7f4d3f3be1878c37380804a017e0b9127bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-03-24T09:34:14.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:13 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:13.797+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:14.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:13 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:13.797+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:14.067 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:13 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:13.591+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:14.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:13 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:13.591+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:14.501 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:14 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:14.074+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:14.529 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:14 smithi117 podman[71905]: 2024-03-24 09:34:14.070745636 +0000 UTC m=+0.017269258 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:14.529 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:14 smithi117 podman[71905]: 2024-03-24 09:34:14.177095771 +0000 UTC m=+0.123619388 container create 7ebead22e88bafe68153b6e6fd009fd11c990150a1b49cc5d1cd64a207e94866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=) 2024-03-24T09:34:14.529 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:14 smithi117 podman[71905]: 2024-03-24 09:34:14.412937401 +0000 UTC m=+0.359461021 container init 7ebead22e88bafe68153b6e6fd009fd11c990150a1b49cc5d1cd64a207e94866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T09:34:14.529 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:14 smithi117 podman[71905]: 2024-03-24 09:34:14.415886266 +0000 UTC m=+0.362409884 container start 7ebead22e88bafe68153b6e6fd009fd11c990150a1b49cc5d1cd64a207e94866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T09:34:14.530 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:14 smithi117 podman[71905]: 2024-03-24 09:34:14.529739007 +0000 UTC m=+0.476262625 container attach 7ebead22e88bafe68153b6e6fd009fd11c990150a1b49cc5d1cd64a207e94866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T09:34:14.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:14 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:14.500+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:14.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:14 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:14.500+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:14.806 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:14 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:14.577+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:14.806 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 24 09:34:14 smithi117 podman[71905]: 2024-03-24 09:34:14.705459894 +0000 UTC m=+0.651983526 container died 7ebead22e88bafe68153b6e6fd009fd11c990150a1b49cc5d1cd64a207e94866 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:34:14.907 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:14 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:14.633+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:14.907 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:14 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:14.633+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:15.057 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:14 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:14.804+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:15.058 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:14 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:14.804+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:15.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:15.042+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:15.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:14 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:14.805+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:15.317 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:14 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:14.907+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:15.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T09:34:15.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:15.444 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T09:34:15.444 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T09:34:15.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.5 2024-03-24T09:34:15.751 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:15 smithi143 systemd[1]: Stopping Ceph osd.5 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:15.751 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:15.508+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:15.751 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:15.509+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:15.752 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:15.750+0000 7f526d9d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:15.752 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:15.750+0000 7f526d9d5700 -1 osd.5 108 *** Got signal Terminated *** 2024-03-24T09:34:15.768 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:15 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:15.565+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:15.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:15 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:15.664+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:15.818 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:15 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:15.664+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:16.024 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:15.750+0000 7f526d9d5700 -1 osd.5 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:16.025 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:15 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:15.855+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:16.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:15 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:15.767+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:16.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:15 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:15.767+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:16.317 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:15 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:15.917+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:16.473 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:16 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:16.067+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:16.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:16 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:16.473+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:16.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:16 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:16.473+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:17.066 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:16 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:16.578+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:17.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:16 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:16.739+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:17.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:16 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:16.739+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:17.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:16 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:16.921+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:17.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:16 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:16.711+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:17.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:16 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:16.711+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:17.101 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:16 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:16.817+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:17.506 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:17 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:17.101+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:17.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:17 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:17.505+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:17.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:17 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:17.505+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:17.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:17 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:17.562+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:17.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:17 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:17.700+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:17.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:17 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:17.700+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:18.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:17 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:17.911+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:18.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:17 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:17.705+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:18.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:17 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:17.705+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:18.118 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:17 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:17.841+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:18.461 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:18 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:18.118+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:18.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:18 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:18.458+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:18.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:18 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:18.458+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:18.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:18 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:18.458+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.522199+0000 front 2024-03-24T09:33:56.522300+0000 (oldest deadline 2024-03-24T09:34:17.622258+0000) 2024-03-24T09:34:18.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:18 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:18.526+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:18.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:18 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:18.675+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:18.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:18 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:18.675+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:19.068 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:18 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:18.880+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:19.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:18 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:18.722+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:19.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:18 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:18.722+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:19.098 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:18 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:18.823+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:19.450 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:19 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:19.098+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:19.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:19 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:19.449+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:19.774 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:19 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:19.449+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:19.775 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:19 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:19.449+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.522199+0000 front 2024-03-24T09:33:56.522300+0000 (oldest deadline 2024-03-24T09:34:17.622258+0000) 2024-03-24T09:34:19.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:19 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:19.509+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:19.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:19 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:19.509+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:19.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:19 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:19.649+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:19.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:19 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:19.649+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:20.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:19 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:19.926+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:20.068 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:19 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:19.926+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:20.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:19 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:19.744+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:20.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:19 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:19.744+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:20.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:19 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:19.744+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:20.148 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:19 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:19.774+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:20.148 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:19 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:19.774+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:20.416 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:20 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:20.148+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:20.770 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:20 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:20.416+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:38.519115+0000 front 2024-03-24T09:33:38.518942+0000 (oldest deadline 2024-03-24T09:34:02.619108+0000) 2024-03-24T09:34:20.770 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:20 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:20.416+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.820870+0000 front 2024-03-24T09:33:47.820920+0000 (oldest deadline 2024-03-24T09:34:11.920865+0000) 2024-03-24T09:34:20.770 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:20 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5[47633]: 2024-03-24T09:34:20.416+0000 7f5265fc1700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.522199+0000 front 2024-03-24T09:33:56.522300+0000 (oldest deadline 2024-03-24T09:34:17.622258+0000) 2024-03-24T09:34:20.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:20 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:20.536+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:20.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:20 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:20.536+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:20.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:20 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:20.608+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:20.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:20 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:20.608+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:21.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:20 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:20.770+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:21.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:20 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:20.770+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:21.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:20 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:20.914+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:21.068 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:20 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:20.914+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:21.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:20 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:20.739+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:21.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:20 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:20.739+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:21.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:20 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:20.739+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:21.300 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:21 smithi143 podman[73019]: 2024-03-24 09:34:21.074271385 +0000 UTC m=+5.432650310 container died b4b6ea48e2fe898598db12b7d224ee32b8de2b314b88baafcb0c41dd82928251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T09:34:21.300 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:21 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:21.132+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:21.560 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:21 smithi143 podman[73019]: 2024-03-24 09:34:21.484371078 +0000 UTC m=+5.842750008 container cleanup b4b6ea48e2fe898598db12b7d224ee32b8de2b314b88baafcb0c41dd82928251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:34:21.560 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:21 smithi143 bash[73019]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5 2024-03-24T09:34:21.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:21.490+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:21.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:21.490+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:21.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:21.564+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:21.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:21.564+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:21.818 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:21 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:21.564+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:22.024 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:21 smithi143 podman[73037]: 2024-03-24 09:34:21.761180737 +0000 UTC m=+0.681355370 container remove b4b6ea48e2fe898598db12b7d224ee32b8de2b314b88baafcb0c41dd82928251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T09:34:22.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:21 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:21.731+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:22.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:21 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:21.731+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:22.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:21 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:21.867+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:22.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:21 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:21.867+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:22.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:21 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:21.780+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:22.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:21 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:21.780+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:22.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:21 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:21.780+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:22.381 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:22 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:22.151+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:22.381 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:22 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:22.151+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:22.381 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:22 smithi143 podman[73137]: 2024-03-24 09:34:22.024518182 +0000 UTC m=+0.025091604 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:22.382 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:22 smithi143 podman[73137]: 2024-03-24 09:34:22.155767585 +0000 UTC m=+0.156340986 container create 99c110abb18cd47a41d41f83ce29e8ad3191198b146e4541d776024959bf1712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:34:22.676 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:22 smithi143 podman[73137]: 2024-03-24 09:34:22.381346288 +0000 UTC m=+0.381919709 container init 99c110abb18cd47a41d41f83ce29e8ad3191198b146e4541d776024959bf1712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:34:22.677 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:22 smithi143 podman[73137]: 2024-03-24 09:34:22.385764613 +0000 UTC m=+0.386338018 container start 99c110abb18cd47a41d41f83ce29e8ad3191198b146e4541d776024959bf1712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T09:34:22.677 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:22 smithi143 podman[73137]: 2024-03-24 09:34:22.473238776 +0000 UTC m=+0.473812202 container attach 99c110abb18cd47a41d41f83ce29e8ad3191198b146e4541d776024959bf1712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD) 2024-03-24T09:34:22.717 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:22 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:22.455+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:22.717 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:22 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:22.455+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:22.718 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:22 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:22.558+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:22.718 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:22 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:22.558+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:22.718 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:22 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:22.558+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:22.985 INFO:journalctl@ceph.osd.5.smithi143.stdout:Mar 24 09:34:22 smithi143 podman[73137]: 2024-03-24 09:34:22.677352608 +0000 UTC m=+0.677926017 container died 99c110abb18cd47a41d41f83ce29e8ad3191198b146e4541d776024959bf1712 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T09:34:22.985 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:22 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:22.695+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:22.985 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:22 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:22.695+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:23.067 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:22 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:22.907+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:23.068 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:22 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:22.907+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:23.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:22 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:22.811+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:23.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:22 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:22.811+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:23.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:22 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:22.811+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:23.251 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:23.106+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:23.251 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:23.106+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:23.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T09:34:23.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:23.314 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T09:34:23.314 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T09:34:23.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.6 2024-03-24T09:34:23.635 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:23 smithi116 systemd[1]: Stopping Ceph osd.6 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:23.635 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:23.634+0000 7f36e7f30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:23.741 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:23.427+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:23.741 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:23.427+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:23.742 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:23.546+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:23.742 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:23.546+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:23.742 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:23 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:23.546+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:23.910 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:23.634+0000 7f36e7f30700 -1 osd.6 108 *** Got signal Terminated *** 2024-03-24T09:34:23.910 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:23.634+0000 7f36e7f30700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:23.910 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:23.845+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:23.910 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:23.845+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:23.910 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:23.845+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:24.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:23.690+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:24.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:23 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:23.690+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:24.317 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:23.910+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:24.317 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:23 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:23.910+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:24.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:24 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:24.092+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:24.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:24 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:24.092+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:24.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:24 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:24.399+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:24.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:24 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:24.399+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:24.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:24 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:24.559+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:24.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:24 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:24.559+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:24.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:24 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:24.559+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:25.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:24 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:24.675+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:25.025 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:24 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:24.675+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:25.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:24.940+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:25.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:24.940+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:25.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:24.864+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:25.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:24.864+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:25.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:24.864+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:25.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:25 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:25.141+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:25.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:25 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:25.141+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:25.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:25 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:25.358+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:25.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:25 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:25.358+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:25.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:25 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:25.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:25.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:25 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:25.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:25.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:25 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:25.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:26.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:25 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:25.707+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:26.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:25 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:25.707+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:26.317 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:25 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:25.931+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:26.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:25 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:25.931+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:26.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:25 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:25.868+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:26.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:25 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:25.868+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:26.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:25 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:25.868+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:26.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:26 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:26.093+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:26.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:26 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:26.093+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:26.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:26 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:26.372+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:26.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:26 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:26.372+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:26.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:26 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:26.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:26.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:26 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:26.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:26.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:26 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:26.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:26.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:26 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:26.568+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:27.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:26 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:26.747+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:27.025 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:26 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:26.747+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:27.317 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:26.886+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:27.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:26.886+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:27.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:26.896+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:27.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:26.896+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:27.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:26 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:26.896+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:27.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:27.096+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:27.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:27.096+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:27.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:27 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:27.394+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:27.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:27 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:27.394+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:27.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:27 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:27.569+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:27.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:27 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:27.569+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:27.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:27 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:27.569+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:27.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:27 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:27.569+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:28.105 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:27.786+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:28.105 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:27.786+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:28.105 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:27 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:27.786+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:28.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:27.871+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:48.628244+0000 front 2024-03-24T09:33:48.628468+0000 (oldest deadline 2024-03-24T09:34:13.928218+0000) 2024-03-24T09:34:28.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6[60579]: 2024-03-24T09:34:27.871+0000 7f36e4524700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:55.629188+0000 front 2024-03-24T09:33:55.629227+0000 (oldest deadline 2024-03-24T09:34:19.729313+0000) 2024-03-24T09:34:28.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:27.935+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:28.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:27.935+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:28.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:27 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:27.935+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:28.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:28 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:28.104+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:28.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:28 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:28.104+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:28.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:28.360+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:28.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:28.360+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:28.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:28.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:28.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:28.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:28.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:28.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:28.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:28.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:29.098 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:28 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:28.830+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:29.098 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:28 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:28.830+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:29.098 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:28 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:28.830+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:29.318 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:29 smithi116 podman[111488]: 2024-03-24 09:34:29.17065699 +0000 UTC m=+5.661618917 container died 59ee1ddbc4af268ac49e5fd032824fe2ef6f9601d1e9bbc7c17d0d97ee907d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T09:34:29.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:28.951+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:29.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:28.951+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:29.319 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:28.951+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:29.319 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:28 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:28.951+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:29.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:29 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:29.098+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:29.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:29 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:29.098+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:29.802 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:29 smithi116 podman[111488]: 2024-03-24 09:34:29.526574188 +0000 UTC m=+6.017536125 container cleanup 59ee1ddbc4af268ac49e5fd032824fe2ef6f9601d1e9bbc7c17d0d97ee907d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:34:29.803 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:29 smithi116 bash[111488]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6 2024-03-24T09:34:29.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:29.369+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:29.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:29.369+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:29.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:29.369+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:29.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:29.499+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:29.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:29.499+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:29.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:29.499+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:29.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:29 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:29.499+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:30.068 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:29 smithi116 podman[111506]: 2024-03-24 09:34:29.802766373 +0000 UTC m=+0.627695093 container remove 59ee1ddbc4af268ac49e5fd032824fe2ef6f9601d1e9bbc7c17d0d97ee907d07 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T09:34:30.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:29.960+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:30.068 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:29.960+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:30.069 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:29.960+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:30.069 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:29 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:29.960+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:30.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:29 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:29.853+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:30.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:29 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:29.853+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:30.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:29 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:29.853+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:30.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:30 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:30.089+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:30.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:30 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:30.089+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:30.276 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:30 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:30.089+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:30.389 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:30 smithi116 podman[111599]: 2024-03-24 09:34:30.031984642 +0000 UTC m=+0.018482095 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:30.390 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:30 smithi116 podman[111599]: 2024-03-24 09:34:30.154212642 +0000 UTC m=+0.140710086 container create 3c8859b626a247714798fb389771d196c7cc4762fe3757aa0ec2d309cdbdc4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T09:34:30.390 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:30 smithi116 podman[111599]: 2024-03-24 09:34:30.390114823 +0000 UTC m=+0.376612270 container init 3c8859b626a247714798fb389771d196c7cc4762fe3757aa0ec2d309cdbdc4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:34:30.676 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:30 smithi116 podman[111599]: 2024-03-24 09:34:30.393562145 +0000 UTC m=+0.380059593 container start 3c8859b626a247714798fb389771d196c7cc4762fe3757aa0ec2d309cdbdc4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T09:34:30.676 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:30 smithi116 podman[111599]: 2024-03-24 09:34:30.498499607 +0000 UTC m=+0.484997053 container attach 3c8859b626a247714798fb389771d196c7cc4762fe3757aa0ec2d309cdbdc4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T09:34:30.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:30.408+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:30.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:30.408+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:30.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:30.408+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:30.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:30.504+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:30.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:30.504+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:30.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:30.504+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:30.818 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:30 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:30.504+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:30.951 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 24 09:34:30 smithi116 podman[111599]: 2024-03-24 09:34:30.67719726 +0000 UTC m=+0.663694706 container died 3c8859b626a247714798fb389771d196c7cc4762fe3757aa0ec2d309cdbdc4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T09:34:31.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:31.062+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:31.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:31.062+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:31.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:31.062+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:31.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:30 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:30.883+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:31.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:30 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:30.883+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:31.276 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:30 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:30.883+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:31.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:30.950+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:31.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:30.950+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:31.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:30.950+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:31.321 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:30 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:30.950+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:31.369 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T09:34:31.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:31.411 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T09:34:31.411 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T09:34:31.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.7 2024-03-24T09:34:31.710 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:31.525+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:31.710 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:31.525+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:31.710 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:31.525+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:31.710 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:31.525+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:31.422+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:31.422+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:31.422+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 systemd[1]: Stopping Ceph osd.7 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:31.709+0000 7fd6d3790700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:31.709+0000 7fd6d3790700 -1 osd.7 108 *** Got signal Terminated *** 2024-03-24T09:34:31.711 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:31 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:31.709+0000 7fd6d3790700 -1 osd.7 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:32.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:32 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:32.018+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:32.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:32 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:32.018+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:32.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:32 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:32.018+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:32.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:31.897+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:32.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:31.897+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:32.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:31 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:31.897+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:32.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:31.956+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:32.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:31.956+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:32.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:31.956+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:32.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:31 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:31.956+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:32.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:32.429+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:32.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:32.429+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:32.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:32.429+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:32.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:32.511+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:32.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:32.511+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:32.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:32.511+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:32.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:32.511+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:33.219 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:32 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:32.908+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:33.220 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:32 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:32.908+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:33.220 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:32 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:32.908+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:33.220 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:32 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:32.908+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:33.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:32 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:32.922+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:33.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:32 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:32.922+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:33.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:32 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:32.922+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:33.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:33 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:33.037+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:33.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:33 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:33.037+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:33.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:33 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:33.037+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:33.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:33.412+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:33.412+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:33.412+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:33.485+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:33.485+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:33.485+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:33.485+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:33.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:33 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:33.485+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:34.233 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:33.917+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:34.233 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:33.917+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:34.233 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:33.917+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:34.233 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:33 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:33.917+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:34.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:34 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:34.075+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:34.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:34 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:34.075+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:34.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:34 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:34.075+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:34.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:33 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:33.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:34.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:33 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:33.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:34.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:33 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:33.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:34.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:34.399+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:34.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:34.399+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:34.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:34.399+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:34.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:34.399+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.750548+0000 front 2024-03-24T09:34:08.750619+0000 (oldest deadline 2024-03-24T09:34:34.050656+0000) 2024-03-24T09:34:34.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:34.471+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:34.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:34.471+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:34.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:34.471+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:34.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:34.471+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:34.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:34 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:34.471+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:35.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:35.029+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:35.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:35.029+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:35.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:35.029+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:35.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:34 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:34.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:35.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:34 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:34.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:35.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:34 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:34.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:35.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:34 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:34.939+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:35.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:34 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:34.939+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:35.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:34 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:34.939+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:35.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:34 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:34.939+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:35.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:34 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:34.939+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:35.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:35.372+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:35.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:35.372+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:35.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:35.372+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:35.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:35.372+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.750548+0000 front 2024-03-24T09:34:08.750619+0000 (oldest deadline 2024-03-24T09:34:34.050656+0000) 2024-03-24T09:34:35.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:35.423+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:35.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:35.423+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:35.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:35.423+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:35.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:35.423+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:35.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:35 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:35.423+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:36.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:36.077+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:36.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:36.077+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:36.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:36.077+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:36.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:36.077+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:36.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:35.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:36.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:35.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:36.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:35.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:36.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:35 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:35.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:36.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:35.961+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:36.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:35.961+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:36.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:35.961+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:36.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:35.961+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:36.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:35 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:35.961+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:36.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:36.387+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.246719+0000 front 2024-03-24T09:33:47.246691+0000 (oldest deadline 2024-03-24T09:34:10.146858+0000) 2024-03-24T09:34:36.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:36.387+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:53.548149+0000 front 2024-03-24T09:33:53.548252+0000 (oldest deadline 2024-03-24T09:34:19.448257+0000) 2024-03-24T09:34:36.816 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:36.387+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.850169+0000 front 2024-03-24T09:34:02.850086+0000 (oldest deadline 2024-03-24T09:34:28.750165+0000) 2024-03-24T09:34:36.817 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7[53655]: 2024-03-24T09:34:36.387+0000 7fd6cbd7c700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.750548+0000 front 2024-03-24T09:34:08.750619+0000 (oldest deadline 2024-03-24T09:34:34.050656+0000) 2024-03-24T09:34:36.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:36.381+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:36.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:36.381+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:36.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:36.381+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:36.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:36.381+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:36.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:36.381+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:37.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:37.059+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:37.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:37.059+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:37.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:37.059+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:37.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:37.059+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:37.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:36.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:37.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:36.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:37.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:36.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:37.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:36 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:36.962+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:37.310 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:37 smithi117 podman[72024]: 2024-03-24 09:34:37.045014467 +0000 UTC m=+5.437234101 container died d33530d726f98335b4f079d7c3752c1969cf2a551c7bcc63568b671ad2f8dc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:34:37.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:36.914+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:37.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:36.914+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:37.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:36.914+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:37.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:36.914+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:37.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:36 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:36.914+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:37.566 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:37 smithi117 podman[72024]: 2024-03-24 09:34:37.401105666 +0000 UTC m=+5.793325306 container cleanup d33530d726f98335b4f079d7c3752c1969cf2a551c7bcc63568b671ad2f8dc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T09:34:37.566 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:37 smithi117 bash[72024]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7 2024-03-24T09:34:37.567 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:37 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:37.358+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:37.567 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:37 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:37.358+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:37.567 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:37 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:37.358+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:37.567 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:37 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:37.358+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:37.567 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:37 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:37.358+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:38.058 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:37 smithi117 podman[72041]: 2024-03-24 09:34:37.702508833 +0000 UTC m=+0.657101519 container remove d33530d726f98335b4f079d7c3752c1969cf2a551c7bcc63568b671ad2f8dc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T09:34:38.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:38.106+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:38.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:38.106+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:38.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:38.106+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:38.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:38.106+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:38.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:37.990+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:38.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:37.990+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:38.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:37.990+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:38.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:37 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:37.990+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:38.316 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:38 smithi117 podman[72132]: 2024-03-24 09:34:37.961841536 +0000 UTC m=+0.017317064 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:38.316 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:38 smithi117 podman[72132]: 2024-03-24 09:34:38.086549494 +0000 UTC m=+0.142025008 container create 903e212f1f2be4d83930373641d58c05bb65c0777a45aaa89b7c42c8540e8c9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T09:34:38.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:37 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:37.937+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:38.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:37 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:37.937+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:38.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:37 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:37.937+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:38.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:37 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:37.937+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:38.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:37 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:37.937+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:38.625 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:38 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:38.316+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:38.625 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:38 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:38.316+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:38.625 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:38 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:38.316+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:38.626 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:38 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:38.316+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:38.626 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:38 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:38.316+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:38.626 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:38 smithi117 podman[72132]: 2024-03-24 09:34:38.339251089 +0000 UTC m=+0.394726611 container init 903e212f1f2be4d83930373641d58c05bb65c0777a45aaa89b7c42c8540e8c9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:34:38.626 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:38 smithi117 podman[72132]: 2024-03-24 09:34:38.342351247 +0000 UTC m=+0.397826768 container start 903e212f1f2be4d83930373641d58c05bb65c0777a45aaa89b7c42c8540e8c9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:34:38.626 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:38 smithi117 podman[72132]: 2024-03-24 09:34:38.42239094 +0000 UTC m=+0.477866490 container attach 903e212f1f2be4d83930373641d58c05bb65c0777a45aaa89b7c42c8540e8c9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T09:34:38.975 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 24 09:34:38 smithi117 podman[72132]: 2024-03-24 09:34:38.625963123 +0000 UTC m=+0.681438657 container died 903e212f1f2be4d83930373641d58c05bb65c0777a45aaa89b7c42c8540e8c9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5) 2024-03-24T09:34:39.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.116+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:39.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.116+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:39.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.116+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:39.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.116+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:39.277 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:38.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:39.277 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:38.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:39.277 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:38.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:39.277 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:38 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:38.961+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:39.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T09:34:39.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:38 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:38.926+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:39.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:38 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:38.926+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:39.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:38 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:38.926+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:39.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:38 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:38.926+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:39.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:38 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:38.926+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:39.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:39.348 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T09:34:39.348 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-24T09:34:39.348 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.8 2024-03-24T09:34:39.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:39 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:39.336+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:39.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:39 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:39.336+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:39.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:39 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:39.336+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:39.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:39 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:39.336+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:39.567 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:39 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:39.336+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:39.638 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 systemd[1]: Stopping Ceph osd.8 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:39.914 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.638+0000 7fafad8f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:39.914 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.638+0000 7fafad8f2700 -1 osd.8 108 *** Got signal Terminated *** 2024-03-24T09:34:39.914 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:39.638+0000 7fafad8f2700 -1 osd.8 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:40.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:40.155+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:40.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:40.155+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:40.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:40.155+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:40.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:40.155+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:40.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:39.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:40.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:39.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:40.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:39.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:40.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:39 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:39.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:40.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:39 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:39.930+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:40.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:39 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:39.930+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:40.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:39 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:39.930+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:40.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:39 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:39.930+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:40.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:39 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:39.930+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:40.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:40 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:40.366+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:40.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:40 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:40.366+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:40.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:40 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:40.366+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:40.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:40 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:40.366+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:40.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:40 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:40.366+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:41.180 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:40.870+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:41.180 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:40.870+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:41.180 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:40.870+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:41.180 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:40 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:40.870+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:41.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:40 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:40.927+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:41.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:40 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:40.927+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:41.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:40 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:40.927+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:41.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:40 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:40.927+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:41.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:40 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:40.927+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:41.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:41.179+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:41.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:41.179+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:41.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:41.179+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:41.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:41.179+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:41.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:41.179+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:15.775522+0000 front 2024-03-24T09:34:15.775476+0000 (oldest deadline 2024-03-24T09:34:41.075532+0000) 2024-03-24T09:34:41.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:41.399+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:41.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:41.399+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:41.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:41.399+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:41.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:41.399+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:41.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:41.399+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:42.163 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:41.906+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:42.163 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:41.906+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:42.163 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:41.906+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:42.163 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:41 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:41.906+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:42.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:41 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:41.976+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:42.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:41 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:41.976+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:42.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:41 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:41.976+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:42.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:41 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:41.976+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:42.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:41 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:41.976+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:42.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:42.163+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:42.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:42.163+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:42.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:42.163+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:42.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:42.163+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:42.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:42.163+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:15.775522+0000 front 2024-03-24T09:34:15.775476+0000 (oldest deadline 2024-03-24T09:34:41.075532+0000) 2024-03-24T09:34:42.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:42.397+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:42.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:42.397+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:42.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:42.397+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:42.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:42.397+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:42.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:42.397+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:42.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:42 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:42.397+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:43.160 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:42.901+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:43.160 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:42.901+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:43.160 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:42.901+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:43.160 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:42 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:42.901+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:43.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:43 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:43.005+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:43.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:43 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:43.005+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:43.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:43 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:43.005+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:43.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:43 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:43.005+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:43.319 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:43 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:43.005+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:43.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:43.160+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:43.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:43.160+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:43.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:43.160+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:43.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:43.160+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:43.525 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:43.160+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:15.775522+0000 front 2024-03-24T09:34:15.775476+0000 (oldest deadline 2024-03-24T09:34:41.075532+0000) 2024-03-24T09:34:43.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:43.442+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:43.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:43.442+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:43.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:43.442+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:43.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:43.442+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:43.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:43.442+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:43.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:43 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:43.442+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:44.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:43.919+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:43.919+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:43.919+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:43 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:43.919+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:44.146+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:45.470883+0000 front 2024-03-24T09:33:45.470916+0000 (oldest deadline 2024-03-24T09:34:09.570968+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:44.146+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.572684+0000 front 2024-03-24T09:33:56.572711+0000 (oldest deadline 2024-03-24T09:34:21.173521+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:44.146+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.474528+0000 front 2024-03-24T09:34:03.474545+0000 (oldest deadline 2024-03-24T09:34:29.374475+0000) 2024-03-24T09:34:44.275 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:44.146+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:09.875279+0000 front 2024-03-24T09:34:09.875416+0000 (oldest deadline 2024-03-24T09:34:35.775173+0000) 2024-03-24T09:34:44.276 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8[54493]: 2024-03-24T09:34:44.146+0000 7fafa5ede700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:15.775522+0000 front 2024-03-24T09:34:15.775476+0000 (oldest deadline 2024-03-24T09:34:41.075532+0000) 2024-03-24T09:34:44.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:44 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:44.044+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:44.317 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:44 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:44.044+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:44.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:44 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:44.044+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:44.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:44 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:44.044+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:44.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:44 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:44.044+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:44.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:44 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:44.480+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:44.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:44 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:44.480+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:44.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:44 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:44.480+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:44.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:44 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:44.480+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:44.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:44 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:44.480+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:44.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:44 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:44.480+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:45.274 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:45 smithi143 podman[73246]: 2024-03-24 09:34:45.062104775 +0000 UTC m=+5.522677719 container died 645493ef080f6fba0ba2bb830c723ac71b0c08444ae02b2a7fb420079b0edc24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T09:34:45.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:44.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:45.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:44.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:45.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:44.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:45.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:44.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:45.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:45 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:45.034+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:45.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:45 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:45.034+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:45.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:45 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:45.034+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:45.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:45 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:45.034+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:45.318 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:45 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:45.034+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:45.319 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:45 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:45.034+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:45.623 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:45 smithi143 podman[73246]: 2024-03-24 09:34:45.388813348 +0000 UTC m=+5.849386289 container cleanup 645493ef080f6fba0ba2bb830c723ac71b0c08444ae02b2a7fb420079b0edc24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:34:45.623 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:45 smithi143 bash[73246]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8 2024-03-24T09:34:45.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:45 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:45.528+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:45.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:45 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:45.528+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:45.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:45 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:45.528+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:45.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:45 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:45.528+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:45.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:45 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:45.528+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:45.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:45 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:45.528+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:45.935 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:45 smithi143 podman[73263]: 2024-03-24 09:34:45.623312649 +0000 UTC m=+0.560774684 container remove 645493ef080f6fba0ba2bb830c723ac71b0c08444ae02b2a7fb420079b0edc24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T09:34:46.251 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:45 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:45.951+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:46.251 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:45 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:45.951+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:46.251 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:45 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:45.952+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:46.251 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:45 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:45.952+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:46.251 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:45 smithi143 podman[73353]: 2024-03-24 09:34:45.839440567 +0000 UTC m=+0.017833450 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:46.252 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:45 smithi143 podman[73353]: 2024-03-24 09:34:45.967829495 +0000 UTC m=+0.146222377 container create ddc07358271f3698db59b1987c3cd460a53300c38918a68dff6aa64ae5554d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T09:34:46.252 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:46 smithi143 podman[73353]: 2024-03-24 09:34:46.176542154 +0000 UTC m=+0.354935037 container init ddc07358271f3698db59b1987c3cd460a53300c38918a68dff6aa64ae5554d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T09:34:46.252 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:46 smithi143 podman[73353]: 2024-03-24 09:34:46.179512498 +0000 UTC m=+0.357905382 container start ddc07358271f3698db59b1987c3cd460a53300c38918a68dff6aa64ae5554d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:34:46.252 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:46 smithi143 podman[73353]: 2024-03-24 09:34:46.251456954 +0000 UTC m=+0.429849848 container attach ddc07358271f3698db59b1987c3cd460a53300c38918a68dff6aa64ae5554d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T09:34:46.524 INFO:journalctl@ceph.osd.8.smithi143.stdout:Mar 24 09:34:46 smithi143 podman[73353]: 2024-03-24 09:34:46.463355771 +0000 UTC m=+0.641748658 container died ddc07358271f3698db59b1987c3cd460a53300c38918a68dff6aa64ae5554d75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T09:34:46.567 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:46.069+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:46.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:46.069+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:46.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:46.069+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:46.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:46.069+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:46.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:46.069+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:46.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:46 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:46.069+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:46.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:46 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:46.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:46.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:46 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:46.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:46.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:46 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:46.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:46.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:46 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:46.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:46.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:46 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:46.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:46.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:46 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:46.548+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:47.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:46 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:46.942+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:47.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:46 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:46.942+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:47.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:46 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:46.942+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:47.024 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:46 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:46.942+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:47.025 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:46 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:46.942+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:47.107 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.8.service' 2024-03-24T09:34:47.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:47.149 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-24T09:34:47.149 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-24T09:34:47.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.9 2024-03-24T09:34:47.480 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.095+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:47.480 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.095+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:47.480 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.095+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:47.480 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.095+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:47.481 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.095+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:47.481 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.095+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:47.481 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 systemd[1]: Stopping Ceph osd.9 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:47.481 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.479+0000 7fa9fca9a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:47.481 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.479+0000 7fa9fca9a700 -1 osd.9 108 *** Got signal Terminated *** 2024-03-24T09:34:47.481 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:47 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:47.479+0000 7fa9fca9a700 -1 osd.9 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:47.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:47 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:47.544+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:47.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:47 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:47.544+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:47.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:47 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:47.544+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:47.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:47 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:47.544+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:47.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:47 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:47.544+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:47.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:47 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:47.544+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:48.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:47.920+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:48.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:47.920+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:48.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:47.920+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:48.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:47.920+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:48.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:47 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:47.920+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:48.567 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:48.144+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:48.567 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:48.144+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:48.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:48.144+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:48.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:48.144+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:48.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:48.144+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:48.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:48 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:48.144+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:49.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:48.583+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:49.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:48.583+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:49.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:48.583+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:49.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:48.583+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:49.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:48.583+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:49.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:48 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:48.583+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:49.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:48.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:49.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:48.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:49.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:48.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:49.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:48.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:49.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:48.914+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:49.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:49.141+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:49.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:49.141+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:49.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:49.141+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:49.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:49.141+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:49.569 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:49.141+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:49.569 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:49 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:49.141+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:49.816 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:49.536+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:49.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:49.536+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:49.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:49.536+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:49.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:49.536+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:49.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:49.536+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:49.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:49 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:49.536+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:50.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:49.933+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:50.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:49.933+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:50.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:49.933+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:50.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:49.933+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:50.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:49.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:50.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:49 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:49.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:50.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:50.157+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:50.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:50.157+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:50.569 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:50.157+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:50.569 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:50.157+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:50.569 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:50.157+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:50.569 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:50 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:50.157+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:50.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:50.565+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:50.817 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:50.565+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:50.818 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:50.565+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:50.818 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:50.565+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:50.818 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:50.565+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:50.818 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:50 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:50.565+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:51.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:50.927+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:51.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:50.927+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:51.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:50.927+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:51.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:50.927+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:51.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:50.927+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:51.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:50 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:50.927+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:51.567 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:51.153+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:51.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:51.153+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:51.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:51.153+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:51.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:51.153+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:51.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:51.153+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:51.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:51 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:51.153+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:52.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:51.592+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:52.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:51.592+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:52.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:51.592+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:52.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:51.592+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:52.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:51.592+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:52.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:51 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:51.592+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:52.187 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:51.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:52.187 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:51.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:52.187 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:51.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:52.187 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:51.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:52.187 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:51.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:52.188 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:51 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:51.934+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:52.567 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:52.195+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:37.318439+0000 front 2024-03-24T09:33:37.318470+0000 (oldest deadline 2024-03-24T09:34:03.218636+0000) 2024-03-24T09:34:52.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:52.195+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.320058+0000 front 2024-03-24T09:33:47.319879+0000 (oldest deadline 2024-03-24T09:34:10.120517+0000) 2024-03-24T09:34:52.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:52.195+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:54.722111+0000 front 2024-03-24T09:33:54.722042+0000 (oldest deadline 2024-03-24T09:34:18.822094+0000) 2024-03-24T09:34:52.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:52.195+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:33:59.823450+0000 front 2024-03-24T09:33:59.823464+0000 (oldest deadline 2024-03-24T09:34:28.624044+0000) 2024-03-24T09:34:52.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:52.195+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:08.624711+0000 front 2024-03-24T09:34:08.624730+0000 (oldest deadline 2024-03-24T09:34:33.924491+0000) 2024-03-24T09:34:52.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9[67607]: 2024-03-24T09:34:52.195+0000 7fa9f5086700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:20.326073+0000 front 2024-03-24T09:34:20.326157+0000 (oldest deadline 2024-03-24T09:34:45.025887+0000) 2024-03-24T09:34:52.967 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:52.586+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:52.968 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:52.586+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:52.968 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:52.586+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:52.968 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:52.586+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:52.968 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:52.586+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:52.968 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:52 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:52.586+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:53.163 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:52 smithi116 podman[111715]: 2024-03-24 09:34:52.904994595 +0000 UTC m=+5.554979564 container died ba1b894136d2a3ecd0e7a61b30514bcefcf63ab20145f885222340fb9826c7b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T09:34:53.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:52.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:53.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:52.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:53.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:52.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:53.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:52.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:53.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:52.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:53.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:52.913+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:53.530 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:53 smithi116 podman[111715]: 2024-03-24 09:34:53.254614091 +0000 UTC m=+5.904599069 container cleanup ba1b894136d2a3ecd0e7a61b30514bcefcf63ab20145f885222340fb9826c7b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:34:53.530 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:53 smithi116 bash[111715]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9 2024-03-24T09:34:53.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:53 smithi116 podman[111732]: 2024-03-24 09:34:53.530867247 +0000 UTC m=+0.623504943 container remove ba1b894136d2a3ecd0e7a61b30514bcefcf63ab20145f885222340fb9826c7b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T09:34:54.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:54.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:54.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:54.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:54.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:54.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:54.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:53 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:53.570+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:34:54.183 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:53 smithi116 podman[111822]: 2024-03-24 09:34:53.773435957 +0000 UTC m=+0.018594991 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:34:54.184 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:53 smithi116 podman[111822]: 2024-03-24 09:34:53.881785828 +0000 UTC m=+0.126944889 container create 1a8ffe5186550efe442f6f2c63970009f0d03dc4e597dc703f7e1b2038e556fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T09:34:54.184 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:54 smithi116 podman[111822]: 2024-03-24 09:34:54.100994104 +0000 UTC m=+0.346153133 container init 1a8ffe5186550efe442f6f2c63970009f0d03dc4e597dc703f7e1b2038e556fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T09:34:54.184 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:54 smithi116 podman[111822]: 2024-03-24 09:34:54.104341282 +0000 UTC m=+0.349500304 container start 1a8ffe5186550efe442f6f2c63970009f0d03dc4e597dc703f7e1b2038e556fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:34:54.184 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:54 smithi116 podman[111822]: 2024-03-24 09:34:54.184153447 +0000 UTC m=+0.429312469 container attach 1a8ffe5186550efe442f6f2c63970009f0d03dc4e597dc703f7e1b2038e556fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T09:34:54.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:53.936+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:54.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:53.936+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:54.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:53.936+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:54.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:53.936+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:54.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:53.936+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:54.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:53 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:53.936+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:54.568 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:54 smithi116 podman[111822]: 2024-03-24 09:34:54.391495801 +0000 UTC m=+0.636654812 container died 1a8ffe5186550efe442f6f2c63970009f0d03dc4e597dc703f7e1b2038e556fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:34:54.908 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 24 09:34:54 smithi116 podman[111848]: 2024-03-24 09:34:54.887748102 +0000 UTC m=+0.490991426 container remove 1a8ffe5186550efe442f6f2c63970009f0d03dc4e597dc703f7e1b2038e556fe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T09:34:54.913 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.9.service' 2024-03-24T09:34:54.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:34:54.954 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-24T09:34:54.954 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-24T09:34:54.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.10 2024-03-24T09:34:54.984 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:54.984 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:54.984 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:54.985 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:54.985 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:54.985 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:54.985 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:54 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:54.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:34:55.252 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 systemd[1]: Stopping Ceph osd.10 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:34:55.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:54 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:54.898+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:55.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:54 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:54.898+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:55.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:54 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:54.898+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:55.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:54 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:54.898+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:55.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:54 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:54.898+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:55.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:54 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:54.898+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:55.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.251+0000 7f903e718700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:34:55.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.251+0000 7f903e718700 -1 osd.10 108 *** Got signal Terminated *** 2024-03-24T09:34:55.566 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.251+0000 7f903e718700 -1 osd.10 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:34:56.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:56.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:56.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:56.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:56.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:56.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:56.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:55 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:55.589+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:34:56.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:55 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:55.947+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:56.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:55 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:55.947+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:56.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:55 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:55.947+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:56.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:55 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:55.947+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:56.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:55 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:55.947+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:56.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:55 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:55.947+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:57.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.601+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:57.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.602+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:57.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.602+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:57.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.602+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:57.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.602+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:57.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.602+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:57.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:56 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:56.602+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:34:57.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:56.917+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:57.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:56.917+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:57.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:56.917+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:57.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:56.917+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:57.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:56.917+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:57.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:56.917+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:58.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:58.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:58.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:58.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:58.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:58.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:58.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:57 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:57.627+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:34:58.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:57.889+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:58.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:57.889+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:58.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:57.889+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:58.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:57.889+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:58.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:57.889+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:58.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:57 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:57.889+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:34:59.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:34:59.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:34:59.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:34:59.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:34:59.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:34:59.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:34:59.068 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:58 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:58.616+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:34:59.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:58.912+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:34:59.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:58.912+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:34:59.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:58.912+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:34:59.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:58.912+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:34:59.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:58.912+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:34:59.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:58 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:58.912+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:00.066 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.211970+0000 front 2024-03-24T09:33:40.211766+0000 (oldest deadline 2024-03-24T09:34:02.511825+0000) 2024-03-24T09:35:00.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-24T09:33:47.113709+0000 front 2024-03-24T09:33:47.113665+0000 (oldest deadline 2024-03-24T09:34:10.613794+0000) 2024-03-24T09:35:00.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:56.415652+0000 front 2024-03-24T09:33:56.415485+0000 (oldest deadline 2024-03-24T09:34:21.115430+0000) 2024-03-24T09:35:00.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:02.816412+0000 front 2024-03-24T09:34:02.816284+0000 (oldest deadline 2024-03-24T09:34:26.316328+0000) 2024-03-24T09:35:00.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.317517+0000 front 2024-03-24T09:34:10.317566+0000 (oldest deadline 2024-03-24T09:34:33.217729+0000) 2024-03-24T09:35:00.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:17.918939+0000 front 2024-03-24T09:34:17.919018+0000 (oldest deadline 2024-03-24T09:34:41.418692+0000) 2024-03-24T09:35:00.067 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:34:59 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10[60239]: 2024-03-24T09:34:59.598+0000 7f9036d04700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.720328+0000 front 2024-03-24T09:34:27.720375+0000 (oldest deadline 2024-03-24T09:34:53.020275+0000) 2024-03-24T09:35:00.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:00.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:00.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:00.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:00.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:00.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:00.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:34:59 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:34:59.950+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:01.035 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:00 smithi117 podman[72249]: 2024-03-24 09:35:00.756479088 +0000 UTC m=+5.610074153 container died 854b23418c5b7a5969f28857d58de7c8546df13ae5eb236441b0dfc7ccdcb488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:35:01.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:01.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:01.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:01.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:01.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:01.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:01.276 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:00 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:00.925+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:01.316 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 podman[72249]: 2024-03-24 09:35:01.117648773 +0000 UTC m=+5.971243837 container cleanup 854b23418c5b7a5969f28857d58de7c8546df13ae5eb236441b0dfc7ccdcb488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:35:01.316 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 bash[72249]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10 2024-03-24T09:35:01.728 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 podman[72266]: 2024-03-24 09:35:01.41045333 +0000 UTC m=+0.653064943 container remove 854b23418c5b7a5969f28857d58de7c8546df13ae5eb236441b0dfc7ccdcb488 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T09:35:02.030 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 podman[72359]: 2024-03-24 09:35:01.630762253 +0000 UTC m=+0.016768757 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:35:02.030 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 podman[72359]: 2024-03-24 09:35:01.744483142 +0000 UTC m=+0.130489667 container create 44668644928b1aa8f2f9f7fb2d91aeabae20f382b8ec1c612f245b0823417753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-03-24T09:35:02.030 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 podman[72359]: 2024-03-24 09:35:01.954960277 +0000 UTC m=+0.340966771 container init 44668644928b1aa8f2f9f7fb2d91aeabae20f382b8ec1c612f245b0823417753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:35:02.030 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:01 smithi117 podman[72359]: 2024-03-24 09:35:01.958050241 +0000 UTC m=+0.344056725 container start 44668644928b1aa8f2f9f7fb2d91aeabae20f382b8ec1c612f245b0823417753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True) 2024-03-24T09:35:02.031 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:02 smithi117 podman[72359]: 2024-03-24 09:35:02.029811007 +0000 UTC m=+0.415817527 container attach 44668644928b1aa8f2f9f7fb2d91aeabae20f382b8ec1c612f245b0823417753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T09:35:02.231 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:02.231 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:02.231 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:02.232 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:02.232 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:02.232 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:02.232 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:01 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:01.946+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:02.316 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 24 09:35:02 smithi117 podman[72359]: 2024-03-24 09:35:02.248451567 +0000 UTC m=+0.634458044 container died 44668644928b1aa8f2f9f7fb2d91aeabae20f382b8ec1c612f245b0823417753 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-24T09:35:02.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.10.service' 2024-03-24T09:35:02.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:02.944 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-24T09:35:02.944 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-24T09:35:02.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.11 2024-03-24T09:35:03.273 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:03.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:03.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:03.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:03.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:03.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:03.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:02.966+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:03.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:02 smithi143 systemd[1]: Stopping Ceph osd.11 for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:03.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.273+0000 7f191615c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:03.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.273+0000 7f191615c700 -1 osd.11 108 *** Got signal Terminated *** 2024-03-24T09:35:03.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.273+0000 7f191615c700 -1 osd.11 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T09:35:04.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:04.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:04.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:04.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:04.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:04.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:04.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:03 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:03.932+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:05.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:05.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:05.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:05.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:05.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:05.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:05.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:04 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:04.938+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:06.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:06.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:06.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:06.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:06.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:06.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:06.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:05 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:05.959+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:07.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:07.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:07.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:07.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:07.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:07.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:07.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:06 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:06.941+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:08.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-24T09:33:40.279908+0000 front 2024-03-24T09:33:40.279976+0000 (oldest deadline 2024-03-24T09:34:06.180054+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6804 osd.2 since back 2024-03-24T09:33:52.982652+0000 front 2024-03-24T09:33:52.982704+0000 (oldest deadline 2024-03-24T09:34:19.383283+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-24T09:34:03.985190+0000 front 2024-03-24T09:34:03.985290+0000 (oldest deadline 2024-03-24T09:34:27.485136+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-24T09:34:10.286631+0000 front 2024-03-24T09:34:10.286667+0000 (oldest deadline 2024-03-24T09:34:34.986667+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6812 osd.5 since back 2024-03-24T09:34:19.087665+0000 front 2024-03-24T09:34:19.087683+0000 (oldest deadline 2024-03-24T09:34:46.088230+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-24T09:34:27.789277+0000 front 2024-03-24T09:34:27.789199+0000 (oldest deadline 2024-03-24T09:34:49.489010+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-24T09:34:33.590313+0000 front 2024-03-24T09:34:33.590169+0000 (oldest deadline 2024-03-24T09:34:59.489862+0000) 2024-03-24T09:35:08.275 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:07 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11[61534]: 2024-03-24T09:35:07.935+0000 7f190e748700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.143:6820 osd.8 since back 2024-03-24T09:34:42.990762+0000 front 2024-03-24T09:34:42.990736+0000 (oldest deadline 2024-03-24T09:35:07.090685+0000) 2024-03-24T09:35:08.916 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:08 smithi143 podman[73470]: 2024-03-24 09:35:08.654897752 +0000 UTC m=+5.509291198 container died 99a4803132a927513f561cb253080c3aba2efa4fbb0e4aceca5c50441c1177c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:35:09.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73470]: 2024-03-24 09:35:09.007382625 +0000 UTC m=+5.861776073 container cleanup 99a4803132a927513f561cb253080c3aba2efa4fbb0e4aceca5c50441c1177c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:35:09.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 bash[73470]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11 2024-03-24T09:35:09.582 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73488]: 2024-03-24 09:35:09.291755036 +0000 UTC m=+0.629534515 container remove 99a4803132a927513f561cb253080c3aba2efa4fbb0e4aceca5c50441c1177c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:35:09.886 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73584]: 2024-03-24 09:35:09.48503969 +0000 UTC m=+0.017699121 image pull 553463a90fd5dcbd654ed9debc69e27bdd9d55bf450e60a53eb3e730689e5e12 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919 2024-03-24T09:35:09.886 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73584]: 2024-03-24 09:35:09.584081989 +0000 UTC m=+0.116741408 container create c3d2a3baed2d5f19f305f221065eed7759397304d6e2d162eb7b027e518386c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T09:35:09.887 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73584]: 2024-03-24 09:35:09.811643852 +0000 UTC m=+0.344303279 container init c3d2a3baed2d5f19f305f221065eed7759397304d6e2d162eb7b027e518386c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:35:09.887 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73584]: 2024-03-24 09:35:09.814641356 +0000 UTC m=+0.347300781 container start c3d2a3baed2d5f19f305f221065eed7759397304d6e2d162eb7b027e518386c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:35:09.887 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:09 smithi143 podman[73584]: 2024-03-24 09:35:09.886376821 +0000 UTC m=+0.419036243 container attach c3d2a3baed2d5f19f305f221065eed7759397304d6e2d162eb7b027e518386c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T09:35:10.274 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:10 smithi143 podman[73584]: 2024-03-24 09:35:10.107265896 +0000 UTC m=+0.639925313 container died c3d2a3baed2d5f19f305f221065eed7759397304d6e2d162eb7b027e518386c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T09:35:10.804 INFO:journalctl@ceph.osd.11.smithi143.stdout:Mar 24 09:35:10 smithi143 podman[73610]: 2024-03-24 09:35:10.789470417 +0000 UTC m=+0.675810408 container remove c3d2a3baed2d5f19f305f221065eed7759397304d6e2d162eb7b027e518386c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux ) 2024-03-24T09:35:10.810 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@osd.11.service' 2024-03-24T09:35:10.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:10.857 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-24T09:35:10.857 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-24T09:35:10.857 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.a 2024-03-24T09:35:11.215 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:35:10 smithi116 systemd[1]: Stopping Ceph mds.a for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:11.568 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:35:11 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a[74021]: 2024-03-24T09:35:11.214+0000 7f515e409700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:11.568 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:35:11 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a[74021]: 2024-03-24T09:35:11.214+0000 7f515e409700 -1 mds.a *** got signal Terminated *** 2024-03-24T09:35:15.499 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:35:15 smithi116 podman[111939]: 2024-03-24 09:35:15.221290475 +0000 UTC m=+4.171261766 container died 5c1a4d77530075d640807738cc48e4e35a4ec4bd47889e8724a3f1d644d0c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T09:35:15.817 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:35:15 smithi116 podman[111939]: 2024-03-24 09:35:15.606116753 +0000 UTC m=+4.556088049 container cleanup 5c1a4d77530075d640807738cc48e4e35a4ec4bd47889e8724a3f1d644d0c892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-24T09:35:15.818 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 24 09:35:15 smithi116 bash[111939]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-a 2024-03-24T09:35:16.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.a.service' 2024-03-24T09:35:16.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:16.058 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-24T09:35:16.059 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-24T09:35:16.059 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.d 2024-03-24T09:35:16.317 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:35:16 smithi116 systemd[1]: Stopping Ceph mds.d for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:16.817 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:35:16 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d[73042]: 2024-03-24T09:35:16.376+0000 7f6c37f7d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:16.818 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:35:16 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d[73042]: 2024-03-24T09:35:16.376+0000 7f6c37f7d700 -1 mds.d *** got signal Terminated *** 2024-03-24T09:35:18.067 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:35:17 smithi116 podman[112059]: 2024-03-24 09:35:17.799743323 +0000 UTC m=+1.529221904 container died e02a1bbc976816117595a9b871ae6191dbf780f505306a6edc8ad1fa7f45b949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T09:35:18.410 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:35:18 smithi116 podman[112059]: 2024-03-24 09:35:18.168261892 +0000 UTC m=+1.897740470 container cleanup e02a1bbc976816117595a9b871ae6191dbf780f505306a6edc8ad1fa7f45b949 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:35:18.411 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 24 09:35:18 smithi116 bash[112059]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-d 2024-03-24T09:35:18.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.d.service' 2024-03-24T09:35:18.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:18.570 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-24T09:35:18.570 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-24T09:35:18.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.g 2024-03-24T09:35:19.067 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:35:18 smithi116 systemd[1]: Stopping Ceph mds.g for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:19.068 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:35:18 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g[73640]: 2024-03-24T09:35:18.888+0000 7f6cba78e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:19.068 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:35:18 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g[73640]: 2024-03-24T09:35:18.888+0000 7f6cba78e700 -1 mds.g *** got signal Terminated *** 2024-03-24T09:35:23.197 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:35:22 smithi116 podman[112183]: 2024-03-24 09:35:22.900391314 +0000 UTC m=+4.103381955 container died fc2441da97ec17ab58abb893e5f322d533b42d7458d68d60b0d878200e2b33c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T09:35:23.523 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:35:23 smithi116 podman[112183]: 2024-03-24 09:35:23.305691203 +0000 UTC m=+4.508681852 container cleanup fc2441da97ec17ab58abb893e5f322d533b42d7458d68d60b0d878200e2b33c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:35:23.523 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 24 09:35:23 smithi116 bash[112183]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-g 2024-03-24T09:35:23.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.g.service' 2024-03-24T09:35:23.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:23.695 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-24T09:35:23.695 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-24T09:35:23.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.j 2024-03-24T09:35:24.067 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:35:23 smithi116 systemd[1]: Stopping Ceph mds.j for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:24.068 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:35:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j[72689]: 2024-03-24T09:35:24.033+0000 7f2343a42700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:24.068 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:35:24 smithi116 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j[72689]: 2024-03-24T09:35:24.033+0000 7f2343a42700 -1 mds.j *** got signal Terminated *** 2024-03-24T09:35:27.184 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:35:26 smithi116 podman[112303]: 2024-03-24 09:35:26.913475464 +0000 UTC m=+2.992442923 container died e47f948a299cf2a39f0358b43a58ca438d7ad689773d50b4bb3508d498bd8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T09:35:27.534 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:35:27 smithi116 podman[112303]: 2024-03-24 09:35:27.27593388 +0000 UTC m=+3.354901543 container cleanup e47f948a299cf2a39f0358b43a58ca438d7ad689773d50b4bb3508d498bd8f7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:35:27.536 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 24 09:35:27 smithi116 bash[112303]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-j 2024-03-24T09:35:27.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.j.service' 2024-03-24T09:35:27.722 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:27.722 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-24T09:35:27.723 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-24T09:35:27.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.b 2024-03-24T09:35:28.032 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:35:27 smithi117 systemd[1]: Stopping Ceph mds.b for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:28.316 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:35:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: 2024-03-24T09:35:28.032+0000 7fd9f6302700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:28.316 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:35:28 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b[65647]: 2024-03-24T09:35:28.032+0000 7fd9f6302700 -1 mds.b *** got signal Terminated *** 2024-03-24T09:35:31.816 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:35:31 smithi117 podman[72465]: 2024-03-24 09:35:31.551710782 +0000 UTC m=+3.642366698 container died 551dc8fe39be56da4b15f6ccdc4b410a0da7744a00c94cf826e8ce7f8b629a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T09:35:32.189 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:35:31 smithi117 podman[72465]: 2024-03-24 09:35:31.972382075 +0000 UTC m=+4.063038036 container cleanup 551dc8fe39be56da4b15f6ccdc4b410a0da7744a00c94cf826e8ce7f8b629a2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T09:35:32.190 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 24 09:35:31 smithi117 bash[72465]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-b 2024-03-24T09:35:32.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.b.service' 2024-03-24T09:35:32.429 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:32.429 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-24T09:35:32.429 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-24T09:35:32.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.e 2024-03-24T09:35:32.759 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:35:32 smithi117 systemd[1]: Stopping Ceph mds.e for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:33.066 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:35:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e[65022]: 2024-03-24T09:35:32.759+0000 7f83583d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:33.066 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:35:32 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e[65022]: 2024-03-24T09:35:32.759+0000 7f83583d7700 -1 mds.e *** got signal Terminated *** 2024-03-24T09:35:35.809 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:35:35 smithi117 podman[72596]: 2024-03-24 09:35:35.546512437 +0000 UTC m=+2.890747675 container died b6d6ed44dcd8b9b732d2e41ccd9aba686338ceb095ea3346f07090340d52826c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T09:35:36.066 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:35:35 smithi117 podman[72596]: 2024-03-24 09:35:35.892208645 +0000 UTC m=+3.236443892 container cleanup b6d6ed44dcd8b9b732d2e41ccd9aba686338ceb095ea3346f07090340d52826c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T09:35:36.066 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 24 09:35:35 smithi117 bash[72596]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-e 2024-03-24T09:35:36.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.e.service' 2024-03-24T09:35:36.346 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:36.347 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-24T09:35:36.347 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-24T09:35:36.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.h 2024-03-24T09:35:36.662 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:36 smithi117 systemd[1]: Stopping Ceph mds.h for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:37.066 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h[65337]: 2024-03-24T09:35:36.662+0000 7efefdd78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:37.066 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:36 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h[65337]: 2024-03-24T09:35:36.662+0000 7efefdd78700 -1 mds.h *** got signal Terminated *** 2024-03-24T09:35:39.816 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:39 smithi117 podman[72718]: 2024-03-24 09:35:39.540689436 +0000 UTC m=+2.981124572 container died e0c634a61ae50f5299a79bc934182e76388bcb27db2149d787d8f868883bc166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux ) 2024-03-24T09:35:40.387 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:40 smithi117 podman[72718]: 2024-03-24 09:35:40.17020642 +0000 UTC m=+3.610641557 container cleanup e0c634a61ae50f5299a79bc934182e76388bcb27db2149d787d8f868883bc166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T09:35:40.387 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:40 smithi117 bash[72718]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h 2024-03-24T09:35:40.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.h.service' 2024-03-24T09:35:40.652 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:40 smithi117 podman[72732]: 2024-03-24 09:35:40.387917436 +0000 UTC m=+0.845811833 container remove e0c634a61ae50f5299a79bc934182e76388bcb27db2149d787d8f868883bc166 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:35:40.653 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:40 smithi117 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.h.service: Deactivated successfully. 2024-03-24T09:35:40.653 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:40 smithi117 systemd[1]: Stopped Ceph mds.h for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:35:40.653 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 24 09:35:40 smithi117 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.h.service: Consumed 7.604s CPU time. 2024-03-24T09:35:40.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:40.663 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-24T09:35:40.663 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-24T09:35:40.663 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.k 2024-03-24T09:35:41.066 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:35:40 smithi117 systemd[1]: Stopping Ceph mds.k for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:41.566 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:35:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k[64704]: 2024-03-24T09:35:41.098+0000 7f1b3a6c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:41.566 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:35:41 smithi117 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k[64704]: 2024-03-24T09:35:41.098+0000 7f1b3a6c0700 -1 mds.k *** got signal Terminated *** 2024-03-24T09:35:43.807 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:35:43 smithi117 podman[72839]: 2024-03-24 09:35:43.548089574 +0000 UTC m=+2.671472140 container died 6b2a3e1261f6ec019d36398b926c9d0e0f4afd6a656e4c61b84db8fdad0ee019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:35:44.066 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:35:43 smithi117 podman[72839]: 2024-03-24 09:35:43.90678482 +0000 UTC m=+3.030167387 container cleanup 6b2a3e1261f6ec019d36398b926c9d0e0f4afd6a656e4c61b84db8fdad0ee019 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T09:35:44.066 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 24 09:35:43 smithi117 bash[72839]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-k 2024-03-24T09:35:44.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.k.service' 2024-03-24T09:35:44.355 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:44.356 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-24T09:35:44.356 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-24T09:35:44.356 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.c 2024-03-24T09:35:44.664 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:35:44 smithi143 systemd[1]: Stopping Ceph mds.c for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:45.024 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:35:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c[66551]: 2024-03-24T09:35:44.664+0000 7f76459bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:45.024 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:35:44 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c[66551]: 2024-03-24T09:35:44.664+0000 7f76459bc700 -1 mds.c *** got signal Terminated *** 2024-03-24T09:35:47.806 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:35:47 smithi143 podman[73692]: 2024-03-24 09:35:47.528934501 +0000 UTC m=+2.982516692 container died 8b385a11dbd3fca8708a3e6a1b2747e55b2377f1c3dfc38effd6bbfc448921d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:35:48.148 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:35:47 smithi143 podman[73692]: 2024-03-24 09:35:47.897004537 +0000 UTC m=+3.350586727 container cleanup 8b385a11dbd3fca8708a3e6a1b2747e55b2377f1c3dfc38effd6bbfc448921d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T09:35:48.148 INFO:journalctl@ceph.mds.c.smithi143.stdout:Mar 24 09:35:47 smithi143 bash[73692]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-c 2024-03-24T09:35:48.275 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.c.service' 2024-03-24T09:35:48.313 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:48.313 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-24T09:35:48.313 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-24T09:35:48.313 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.f 2024-03-24T09:35:48.524 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:48 smithi143 systemd[1]: Stopping Ceph mds.f for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:49.025 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f[66237]: 2024-03-24T09:35:48.626+0000 7fd20c659700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:49.025 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:48 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f[66237]: 2024-03-24T09:35:48.626+0000 7fd20c659700 -1 mds.f *** got signal Terminated *** 2024-03-24T09:35:51.909 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:51 smithi143 podman[73826]: 2024-03-24 09:35:51.547020905 +0000 UTC m=+3.012280368 container died f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6) 2024-03-24T09:35:52.176 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:51 smithi143 podman[73826]: 2024-03-24 09:35:51.98389097 +0000 UTC m=+3.449150445 container cleanup f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T09:35:52.176 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:51 smithi143 bash[73826]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f 2024-03-24T09:35:52.401 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.f.service' 2024-03-24T09:35:52.426 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:52 smithi143 podman[73840]: 2024-03-24 09:35:52.17661226 +0000 UTC m=+0.626672394 container remove f3d09d2ba8ced093971b7a169dea2bacfe5c62b0bd9e9ae012ad66784ff38d15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T09:35:52.427 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:52 smithi143 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.f.service: Deactivated successfully. 2024-03-24T09:35:52.427 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:52 smithi143 systemd[1]: Stopped Ceph mds.f for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:35:52.427 INFO:journalctl@ceph.mds.f.smithi143.stdout:Mar 24 09:35:52 smithi143 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.f.service: Consumed 4min 31.732s CPU time. 2024-03-24T09:35:52.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:52.436 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-24T09:35:52.437 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-24T09:35:52.437 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.i 2024-03-24T09:35:52.745 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:52 smithi143 systemd[1]: Stopping Ceph mds.i for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:53.024 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i[66859]: 2024-03-24T09:35:52.745+0000 7f56d6b11700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:53.024 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:52 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i[66859]: 2024-03-24T09:35:52.745+0000 7f56d6b11700 -1 mds.i *** got signal Terminated *** 2024-03-24T09:35:55.795 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:55 smithi143 podman[73947]: 2024-03-24 09:35:55.54414463 +0000 UTC m=+2.891309363 container died 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T09:35:56.071 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:55 smithi143 podman[73947]: 2024-03-24 09:35:55.878719473 +0000 UTC m=+3.225884249 container cleanup 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T09:35:56.071 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:55 smithi143 bash[73947]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i 2024-03-24T09:35:56.305 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.i.service' 2024-03-24T09:35:56.330 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:56 smithi143 podman[73961]: 2024-03-24 09:35:56.071254448 +0000 UTC m=+0.524860118 container remove 317ad529fd26fb0e1a0f446f6623aa003aa1a21d7cd3bfced1f776db2e341dc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T09:35:56.330 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:56 smithi143 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.i.service: Deactivated successfully. 2024-03-24T09:35:56.330 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:56 smithi143 systemd[1]: Stopped Ceph mds.i for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:35:56.330 INFO:journalctl@ceph.mds.i.smithi143.stdout:Mar 24 09:35:56 smithi143 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.i.service: Consumed 4min 28.438s CPU time. 2024-03-24T09:35:56.342 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:35:56.342 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-24T09:35:56.342 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-24T09:35:56.343 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.l 2024-03-24T09:35:56.657 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:35:56 smithi143 systemd[1]: Stopping Ceph mds.l for 69c76602-e9be-11ee-95cd-87774f69a715... 2024-03-24T09:35:57.024 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:35:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l[65912]: 2024-03-24T09:35:56.656+0000 7f45920b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T09:35:57.025 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:35:56 smithi143 ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l[65912]: 2024-03-24T09:35:56.656+0000 7f45920b9700 -1 mds.l *** got signal Terminated *** 2024-03-24T09:35:59.798 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:35:59 smithi143 podman[74069]: 2024-03-24 09:35:59.536532903 +0000 UTC m=+2.981805797 container died 8034a12199b3f1a44cb55d2e079d4c4084083a833d571c1ff58de5f3f7280ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T09:36:00.065 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:35:59 smithi143 podman[74069]: 2024-03-24 09:35:59.873479523 +0000 UTC m=+3.318752465 container cleanup 8034a12199b3f1a44cb55d2e079d4c4084083a833d571c1ff58de5f3f7280ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T09:36:00.065 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:35:59 smithi143 bash[74069]: ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l 2024-03-24T09:36:00.297 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.l.service' 2024-03-24T09:36:00.321 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:36:00 smithi143 podman[74083]: 2024-03-24 09:36:00.065902578 +0000 UTC m=+0.528745273 container remove 8034a12199b3f1a44cb55d2e079d4c4084083a833d571c1ff58de5f3f7280ff1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a32537e3857c015cff859e834d65089d866c82026571595cd97228205b5b8919, name=ceph-69c76602-e9be-11ee-95cd-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240323.003144-squid-21b3bd6, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T09:36:00.321 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:36:00 smithi143 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.l.service: Deactivated successfully. 2024-03-24T09:36:00.321 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:36:00 smithi143 systemd[1]: Stopped Ceph mds.l for 69c76602-e9be-11ee-95cd-87774f69a715. 2024-03-24T09:36:00.321 INFO:journalctl@ceph.mds.l.smithi143.stdout:Mar 24 09:36:00 smithi143 systemd[1]: ceph-69c76602-e9be-11ee-95cd-87774f69a715@mds.l.service: Consumed 3.610s CPU time. 2024-03-24T09:36:00.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T09:36:00.332 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-24T09:36:00.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T09:36:00.521 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:36:02.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T09:36:03.078 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:36:05.493 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T09:36:05.683 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:36:07.996 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:36:08.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:36:08.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T09:36:08.095 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T09:36:08.098 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi116/crash 2024-03-24T09:36:08.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash -- . 2024-03-24T09:36:08.145 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T09:36:08.146 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-24T09:36:08.148 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi117/crash 2024-03-24T09:36:08.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash -- . 2024-03-24T09:36:08.187 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T09:36:08.187 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-03-24T09:36:08.189 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi143/crash 2024-03-24T09:36:08.190 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash -- . 2024-03-24T09:36:08.223 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/69c76602-e9be-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T09:36:08.223 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now 2024-03-24T09:36:08.225 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T09:36:08.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T09:36:08.268 INFO:tasks.cephadm:Compressing logs... 2024-03-24T09:36:08.268 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T09:36:08.310 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T09:36:08.313 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T09:36:08.337 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T09:36:08.337 INFO:teuthology.orchestra.run.smithi116.stderr:: No such file or directory 2024-03-24T09:36:08.337 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-03-24T09:36:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T09:36:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.0.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T09:36:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T09:36:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-24T09:36:08.338 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-24T09:36:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T09:36:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:: No such file or directory 2024-03-24T09:36:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-24T09:36:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T09:36:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-24T09:36:08.344 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.11.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T09:36:08.345 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.f.log: 90.3% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T09:36:08.346 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log 2024-03-24T09:36:08.353 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log: 84.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T09:36:08.353 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T09:36:08.353 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log: 87.5% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T09:36:08.353 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-24T09:36:08.353 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.j.log: /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.3.log: 90.1% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.x.log: 94.6% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T09:36:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T09:36:08.358 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log 2024-03-24T09:36:08.358 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log: 84.4% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T09:36:08.358 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T09:36:08.358 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log: 87.5% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T09:36:08.358 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-24T09:36:08.359 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T09:36:08.359 INFO:teuthology.orchestra.run.smithi117.stderr:: No such file or directory 2024-03-24T09:36:08.359 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-24T09:36:08.359 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.b.log: /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T09:36:08.360 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-24T09:36:08.361 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T09:36:08.361 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T09:36:08.361 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T09:36:08.362 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T09:36:08.363 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-24T09:36:08.364 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.9.log: 91.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T09:36:08.364 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-24T09:36:08.365 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log 2024-03-24T09:36:08.370 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-24T09:36:08.370 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-24T09:36:08.374 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T09:36:08.378 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T09:36:08.379 INFO:teuthology.orchestra.run.smithi117.stderr: 84.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T09:36:08.381 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log: 91.9% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T09:36:08.382 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T09:36:08.385 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T09:36:08.390 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-24T09:36:08.402 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T09:36:08.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T09:36:08.715 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T09:36:08.968 INFO:teuthology.orchestra.run.smithi143.stderr: 92.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T09:36:08.997 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T09:36:09.119 INFO:teuthology.orchestra.run.smithi143.stderr: 95.7% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-24T09:36:09.144 INFO:teuthology.orchestra.run.smithi143.stderr: 95.3% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-24T09:36:09.433 INFO:teuthology.orchestra.run.smithi116.stderr: 95.8% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-24T09:36:09.664 INFO:teuthology.orchestra.run.smithi116.stderr: 90.6% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T09:36:09.707 INFO:teuthology.orchestra.run.smithi116.stderr: 95.4% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-24T09:36:10.470 INFO:teuthology.orchestra.run.smithi117.stderr: 95.4% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-24T09:36:12.000 INFO:teuthology.orchestra.run.smithi143.stderr: 92.7% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-24T09:36:12.478 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T09:36:12.480 INFO:teuthology.orchestra.run.smithi116.stderr: 92.5% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T09:36:12.898 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T09:36:13.167 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T09:36:13.574 INFO:teuthology.orchestra.run.smithi143.stderr: 92.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T09:36:13.632 INFO:teuthology.orchestra.run.smithi117.stderr: 92.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-24T09:36:14.186 INFO:teuthology.orchestra.run.smithi116.stderr: 92.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-24T09:36:14.511 INFO:teuthology.orchestra.run.smithi143.stderr: 92.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-24T09:36:14.590 INFO:teuthology.orchestra.run.smithi143.stderr: 94.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-24T09:36:15.243 INFO:teuthology.orchestra.run.smithi143.stderr: 92.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T09:36:15.601 INFO:teuthology.orchestra.run.smithi117.stderr: 93.8% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-24T09:36:15.743 INFO:teuthology.orchestra.run.smithi143.stderr: 94.0% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-24T09:36:15.745 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-03-24T09:36:15.745 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m7.417s 2024-03-24T09:36:15.745 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m36.025s 2024-03-24T09:36:15.745 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m1.109s 2024-03-24T09:36:15.882 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T09:36:15.932 INFO:teuthology.orchestra.run.smithi117.stderr: 92.7% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T09:36:17.319 INFO:teuthology.orchestra.run.smithi117.stderr: 93.6% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-24T09:36:17.494 INFO:teuthology.orchestra.run.smithi116.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-03-24T09:36:17.496 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-24T09:36:17.496 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m9.173s 2024-03-24T09:36:17.497 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m31.245s 2024-03-24T09:36:17.497 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m1.053s 2024-03-24T09:36:33.000 INFO:teuthology.orchestra.run.smithi117.stderr: 93.2% -- replaced with /var/log/ceph/69c76602-e9be-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-24T09:36:33.001 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-24T09:36:33.002 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m24.674s 2024-03-24T09:36:33.002 INFO:teuthology.orchestra.run.smithi117.stderr:user 1m5.440s 2024-03-24T09:36:33.002 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m2.025s 2024-03-24T09:36:33.002 INFO:tasks.cephadm:Archiving logs... 2024-03-24T09:36:33.003 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi116/log 2024-03-24T09:36:33.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T09:36:36.948 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi117/log 2024-03-24T09:36:36.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T09:36:44.958 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi143/log 2024-03-24T09:36:44.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T09:36:49.437 INFO:tasks.cephadm:Removing cluster... 2024-03-24T09:36:49.438 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --force 2024-03-24T09:36:49.629 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:36:49.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --force 2024-03-24T09:36:50.096 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:36:50.423 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 69c76602-e9be-11ee-95cd-87774f69a715 --force 2024-03-24T09:36:50.615 INFO:teuthology.orchestra.run.smithi143.stdout:Deleting cluster with fsid: 69c76602-e9be-11ee-95cd-87774f69a715 2024-03-24T09:36:50.912 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T09:36:50.912 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T09:36:50.931 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T09:36:50.948 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T09:36:50.963 INFO:tasks.cephadm:Teardown complete 2024-03-24T09:36:50.963 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T09:36:50.975 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T09:36:50.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T09:36:50.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T09:36:50.990 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T09:36:51.103 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T09:36:51.115 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T09:36:51.115 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:36:51.118 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:36:51.120 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T09:36:51.132 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:36:51.132 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:36:51.134 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-03-24T09:36:51.136 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:36:51.137 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-24T09:36:51.137 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 +35us[ +38us] +/- 69ms 2024-03-24T09:36:51.137 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -537us[ -534us] +/- 61ms 2024-03-24T09:36:51.137 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 +828us[ +828us] +/- 85ms 2024-03-24T09:36:51.137 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +17us[ +33us] +/- 69ms 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -560us[ -544us] +/- 61ms 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +808us[ +824us] +/- 84ms 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:51.139 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T09:36:51.140 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-24T09:36:51.140 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 110 +38us[ +46us] +/- 86ms 2024-03-24T09:36:51.140 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -550us[ -541us] +/- 61ms 2024-03-24T09:36:51.140 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +818us[ +826us] +/- 84ms 2024-03-24T09:36:51.140 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T09:36:51.140 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T09:36:51.151 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T09:36:51.152 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T09:36:51.169 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T09:36:51.270 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T09:36:51.343 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T09:36:51.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:36:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:36:51.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:36:51.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:36:51.475 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T09:36:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:36:51.506 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T09:36:51.516 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T09:36:51.536 INFO:teuthology.task.internal:Duration was 1926.871494 seconds 2024-03-24T09:36:51.537 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T09:36:51.553 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T09:36:51.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T09:36:51.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T09:36:51.558 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T09:36:51.597 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:36:51.598 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:36:51.601 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T09:36:52.105 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T09:36:52.105 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-24T09:36:52.105 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T09:36:52.125 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-24T09:36:52.125 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T09:36:52.173 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-03-24T09:36:52.173 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T09:36:52.199 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T09:36:52.199 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T09:36:52.202 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T09:36:52.216 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T09:36:52.269 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T09:36:52.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T09:36:52.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T09:36:52.283 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T09:36:52.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T09:36:52.688 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T09:36:52.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T09:36:52.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T09:36:52.742 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T09:36:52.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T09:36:52.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T09:36:52.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T09:36:52.784 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T09:36:52.805 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-24T09:36:52.807 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-24T09:36:52.833 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-03-24T09:36:52.858 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T09:36:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:36:52.875 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T09:36:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:36:52.889 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T09:36:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T09:36:52.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T09:36:52.928 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T09:36:52.929 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi116 2024-03-24T09:36:52.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T09:36:52.972 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi117 2024-03-24T09:36:52.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T09:36:53.010 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041/remote/smithi143 2024-03-24T09:36:53.010 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T09:36:53.049 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T09:36:53.049 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T09:36:53.051 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T09:36:53.053 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T09:36:53.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T09:36:53.111 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T09:36:53.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T09:36:53.129 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T09:36:53.129 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T09:36:53.132 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T09:36:53.134 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T09:36:53.144 INFO:teuthology.orchestra.run.smithi116.stdout: 265309 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 09:36 /home/ubuntu/cephtest 2024-03-24T09:36:53.146 INFO:teuthology.orchestra.run.smithi117.stdout: 265366 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 09:36 /home/ubuntu/cephtest 2024-03-24T09:36:53.152 INFO:teuthology.orchestra.run.smithi143.stdout: 265848 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 09:36 /home/ubuntu/cephtest 2024-03-24T09:36:53.153 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T09:36:53.218 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T09:36:53.254 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T09:36:53.282 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-24T09:36:53.282 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T09:36:53.308 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-24T09:36:53.308 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T09:36:53.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 08:56:43.707467', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFYTgwwOu+qTUs1gsSOAj/87LIqr9KvLYwkWMvOtbWNrIEn9HddFLJ3lhalWnfLbi2fNxScUOEgQ9dHvquEaMDg='} 2024-03-24T09:36:53.332 DEBUG:teuthology.nuke:shortname: smithi143 2024-03-24T09:36:53.333 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T09:36:53.341 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 08:56:43.703876', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIdp7nXb8gXZr4iEpfroPoTTDcs4HToxi81HhJcAuMTP4kHdGtHX7tb18XCJAiWD04HaKCbfS44q0CZi6tlEvbw='} 2024-03-24T09:36:53.359 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619041', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 08:56:43.705931', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI+NgiQJlqPmpDDDfH0nKgQEVRF7PjFKCsnhiFi0ocLjsAmtiy4I9SWepamQiP71ZN73t1ataxNAOtjyjOEH42c='} 2024-03-24T09:36:53.372 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-24T09:36:53.373 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T09:36:53.399 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-24T09:36:53.400 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:36:53.415 INFO:teuthology.orchestra.console.smithi143:Power off 2024-03-24T09:36:53.415 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T09:36:53.431 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-24T09:36:53.431 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T09:36:53.440 DEBUG:teuthology.orchestra.console.smithi143:power off output: Chassis Power Control: Down/Off 2024-03-24T09:36:53.441 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:36:53.448 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T09:36:53.455 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-24T09:36:53.455 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:36:53.465 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T09:36:53.479 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-24T09:36:57.449 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:36:57.466 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:36:57.480 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:36:57.560 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T09:36:57.577 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is on 2024-03-24T09:36:57.589 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-24T09:37:01.565 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:37:01.578 DEBUG:teuthology.orchestra.console.smithi143:pexpect command: ipmitool -H smithi143.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:37:01.590 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:37:01.675 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-24T09:37:01.687 DEBUG:teuthology.orchestra.console.smithi143:check power output: Chassis Power is off 2024-03-24T09:37:01.699 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-24T09:37:01.788 INFO:teuthology.orchestra.console.smithi143:Power off completed 2024-03-24T09:37:01.800 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-24T09:37:05.680 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T09:37:05.790 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-24T09:37:05.891 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-24T09:37:05.997 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 1926.871494293213 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi116 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3dbc45f72e5443bdb48dee0d180b8ca2 status: fail success: false 2024-03-24T09:37:05.997 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T09:37:06.089 INFO:teuthology.run:FAIL